RuntimeError: redex-all crashed with exit code -6!

[xnfreedom]

use code version: 45d0f92

Error info:
WARNING: Unable to find RequiresApi annotation. It's either unused (okay) or been deleted (not okay)
WARNING: No inliner config
Failed to allocate Lgnu/crypto/hash/Haval;.transform:([BI)V

config file looks like :

{
"redex" : {
"keep_annotations": [
"Lcom/path/to/your/DoNotStrip;"
],
"passes" : [
"RegAllocPass",
"InterDexPass"
]
},
"RenameClassesPassV2" : {
"dont_rename_annotated": [
"Lcom/path/to/your/DoNotStrip;"
]
},
"InterDexPass" : {
"coldstart_classes": "~/redex/classtest/filemanager_class_list.txt"
},
"RegAllocPass" : {
"live_range_splitting": false
}
}

full output log as the attachments.
filemanager.txt.zip

[justinjhendrick]

This error is happening because RegAlloc believes its input code is invalid. I looked at the error message and I think I agree with it.

Here is the short version of the error message RegAlloc threw:
Type violation of v6

and the relevant code section is:

Failed to allocate Lgnu/crypto/hash/Haval;.transform:([BI)V
CFG:
 Block B0: entry
   preds:
   [0x7f725a158090] OPCODE: IOPCODE_LOAD_PARAM_OBJECT v0
   [0x7f7240104a00] OPCODE: IOPCODE_LOAD_PARAM_OBJECT v1
   [0x7f7240104a60] OPCODE: IOPCODE_LOAD_PARAM v2
   [0x7f724017c0a0] DEBUG: DBG_SET_PROLOGUE_END
   [0x7f724017c0d0] POSITION: SourceFile:202
   [0x7f7240104a90] OPCODE: MOVE_OBJECT v3, v0
   [0x7f7240104ac0] OPCODE: MONITOR_ENTER v3
   [0x7f7240104af0] OPCODE: MOVE v4, v2
   [0x7f7240104b20] OPCODE: ADD_INT_LIT8 v5, v4, 1
   succs: (goto B1)
 Block B1:
   preds: (goto B0)
   [0x7f7254645950] TRY: TRY_START 0x7f7254ac4510
   [0x7f7240104b50] OPCODE: MOVE_OBJECT v6, v1
   [0x7f7240104b80] OPCODE: MOVE v7, v2
   [0x7f7240104bb0] OPCODE: AGET_BYTE v6, v6                  // ERROR HERE
   succs: (goto B2) (throw B323)
...

The last line of code in this snippet looks incorrect. v6 holds an array, but it's being used as the index into itself, which is a type error. An array is not an int.

I think the correct line would be:

OPCODE: AGET_BYTE v6, v7

[justinjhendrick]

Could you run dexdump on your input dex(es), find Lgnu/crypto/hash/Haval;.transform:([BI)V, and send that in an attachment, please? I want to see if the input is valid before it gets passed into Redex or if a Redex transformation (before Register Allocation) made a mistake.

[AlanXuGuangZ]

Like this:
#5 : (in Lgnu/crypto/hash/Whirlpool;)
name : 'transform'
type : '([BI)V'
access : 0x0014 (PROTECTED FINAL)
code -
registers : 16
ins : 3
outs : 0
insns size : 2235 16-bit code units
catches : (none)
positions :
0x000a line=311
0x0050 line=315
0x0096 line=319
0x00dc line=323
0x0122 line=327
0x0168 line=331
0x01ae line=335
0x01f4 line=339
0x023a line=345
0x023e line=346
0x0242 line=347
0x0246 line=348
0x024a line=349
0x024e line=350
0x0252 line=351
0x0256 line=352
0x025a line=354
0x0261 line=355
0x0268 line=356
0x026f line=357
0x0276 line=358
0x027d line=359
0x0284 line=360
0x028b line=361
0x0292 line=364
0x02a4 line=366
0x02a9 line=472
0x02b3 line=473
0x02bd line=474
0x02c7 line=475
0x02d1 line=476
0x02db line=477
0x02e5 line=478
0x02ef line=479
0x02f9 line=480
0x02fa line=369
0x0355 line=375
0x03ab line=380
0x0401 line=385
0x0457 line=390
0x04ad line=395
0x0503 line=400
0x0559 line=405
0x05af line=410
0x05b3 line=411
0x05b7 line=412
0x05bb line=413
0x05bf line=414
0x05c3 line=415
0x05c7 line=416
0x05cb line=417
0x05cf line=420
0x0628 line=425
0x0681 line=430
0x06da line=435
0x0733 line=440
0x078c line=445
0x07e5 line=450
0x083e line=455
0x0897 line=461
0x089b line=462
0x089f line=463
0x08a3 line=464
0x08a7 line=465
0x08ab line=466
0x08af line=467
0x08b3 line=468
0x08b7 line=366
locals :
0x0000 - 0x08bb reg=13 this Lgnu/crypto/hash/Whirlpool;
source_file_idx : 20446 (SourceFile)

Please see attachment for details
filemanager_dexdump.zip

[agampe]

Please create the dexdump with the -d option, otherwise it will not contain actual code. It is also OK to not include the complete dump, just the Haval stuff is enough.

[agampe]

Alternatively, can you add this to your redex config and run it again, posting any failures:

"ir_type_checker": {
  "run_after_each_pass": true
}

You will have to add a pass before RegAllocPass, though. Something like SimplifyCFGPass should work.

@justinjhendrick : note that the config is actually super-simple and indicates the input is broken.

[AlanXuGuangZ]

I have created the dexdump with the -d.
Like this：

    #5              : (in Lgnu/crypto/hash/Haval;)
      name          : 'transform'
      type          : '([BI)V'
      access        : 0x20004 (PROTECTED DECLARED_SYNCHRONIZED)
      code          -
      registers     : 395
      ins           : 3
      outs          : 11
      insns size    : 5238 16-bit code units

Please see attachment for details
filemanager_dexdump-d.zip

[AlanXuGuangZ]

In addition, i have added ir_type_checkerto my redex config and run it again.
Config file looks like :

{
  "redex" : {
    "keep_annotations": [
      "Lcom/path/to/your/DoNotStrip;"
    ],
    "passes" : [
      "ir_type_checker",
      "RegAllocPass",
      "InterDexPass"
    ]
  },
  "ir_type_checker": {
    "run_after_each_pass": true
  },
  "RenameClassesPassV2" : {
  "dont_rename_annotated": [
    "Lcom/path/to/your/DoNotStrip;"
   ]
  },
  "InterDexPass" : {
    "coldstart_classes": "~/redex/classtest/filemanager_class_list.txt" 
  },
  "RegAllocPass" : {
    "live_range_splitting": false
  }
}

Error info:

terminate called after throwing an instance of 'RedexException'
  what():  libredex/PassManager.cpp:319: void PassManager::activate_pass(const char*, const Json::Value&): assertion `false' failed.
No pass named ir_type_checker!

Full output log as the attachments.
filemanager.log

[agampe]

From the dexdump:

5559b8:                                        |[5559b8] gnu.crypto.hash.Haval.transform:([BI)V
5559c8: 0800 8801                              |0000: move-object/from16 v0, v392
5559cc: 1d00                                   |0002: monitor-enter v0
5559ce: 0200 8a01                              |0003: move/from16 v0, v394
5559d2: d802 0001                              |0005: add-int/lit8 v2, v0, #int 1 // #01
5559d6: 0800 8901                              |0007: move-object/from16 v0, v393
5559da: 0201 8a01                              |0009: move/from16 v1, v394
5559de: 4803 0000                              |000b: aget-byte v3, v0, v0

Which confirms that the input bytecode is broken.

@AlanXuGuangZ What tool produced the dex file?

If you are not using Haval (which seems likely, as this should get you a VerifyError on device), you may try to prepend the remove-unreachable-classes pass (RemoveUnreachablePass) to the passes list and hope that it gets removed (or do it manually). If you are using it, you will need to get a version with valid bytecode.

Side note: The IRTypeChecker is not a pass. You do not need to add ir_type_checker to the passes list. However, it won't complain right now, as there is no pre-pass, only post-passes.

[AlanXuGuangZ]

What tool produced the dex file?
The APK compiled with Android gradle.

I have prepend the remove-unreachable-classes pass to the passes list.
The config file :

{
  "redex" : {
    "keep_annotations": [
      "Lcom/path/to/your/DoNotStrip;"
    ],
    "passes" : [
      "RemoveUnreachablePass",
      "RegAllocPass",
      "InterDexPass"
    ]
  },
  "ir_type_checker": {
    "run_after_each_pass": true
  },
  "RenameClassesPassV2" : {
  "dont_rename_annotated": [
    "Lcom/path/to/your/DoNotStrip;"
   ]
  },
  "InterDexPass" : {
    "coldstart_classes": "~/redex/classtest/filemanager_class_list.txt" 
  },
  "RegAllocPass" : {
    "live_range_splitting": false
  }
}

But it'll still report a mistake.just look likes this:

WARNING: Unable to find RequiresApi annotation. It's either unused (okay) or been deleted (not okay)
WARNING: No inliner config
Failed to allocate Lgnu/crypto/hash/Haval;.transform:([BI)V
...
terminate called after throwing an instance of 'Traceback (most recent call last):
  File "/tmp/redex.mB0gkK/redex.py", line 1096, in <module>
    run_redex(args)
  File "/tmp/redex.mB0gkK/redex.py", line 1074, in run_redex
    run_redex_binary(state)
  File "/tmp/redex.mB0gkK/redex.py", line 411, in run_redex_binary
    if run():
  File "/tmp/redex.mB0gkK/redex.py", line 396, in run
    % script_filenames
RuntimeError: redex-all crashed with exit code 1! You can re-run it under gdb by running /tmp/redex.mB0gkK/redex-gdb-9dp27fjl.sh or under lldb by running /tmp/redex.mB0gkK/redex-lldb-br6xzsh0.sh

Full output log as the attachments.
filemanager.log

[xnfreedom]

There 's some apps do redex successfully. Some other apps failed with error code -6.
All these apps compiled with android gradle tool.

[agampe]

"Successful" apps may not have broken bytecode included - you do not give enough details to understand the issue.

"android gradle tool" is not a compiler, it's a build system. Is Gradle configured to use dx, Jack or d8/r8? What version do you use? Are you using it with Android Studio or standalone? If the former, what Studio version do you use?

Where does the Haval code come from? Are you using Java sources for GNU crypto? A jar file? If the latter, where did you get it from?

If RemoveUnreachablePass cannot remove Haval, it is wired up in a way that it seems "reachable" for the simple static analysis that the pass does. You may need to add more aggressive and time-consuming passes to remove the dependency, or attempt to do it manually.

[justinjhendrick]

Closing due to inactivity. Feel free to re-open if you have more info.