mobile-install: App on architecture arm64 with arm64-v8a abi could not find native library

[nxty2011]

Description of the problem

I want to build an android binary which rely on a native library, and expect it to run on an android 5.1 system whose architecture is arm64, A53 CPU.

So, I build this apk with commands:
bazel mobile-install --fat_apk_cpu=arm64-v8a --compilation_mode=opt <build file location>:<target name> --start_app -s

With -s option, every build command and its flags seem to be reasonable. But after building, the android device run the target apk with failure. From adb tool, I got the error log, which is:

E/AndroidRuntime( 5399): FATAL EXCEPTION: main
E/AndroidRuntime( 5399): Process: , PID: 5399
E/AndroidRuntime( 5399): java.lang.UnsatisfiedLinkError: com.google.devtools.build.android.incrementaldeployment.IncrementalClassLoader$DelegateClassLoader[DexPathList[[dex file "/data/local/tmp/incrementaldeployment//dex/incremental_classes1.dex"],nativeLibraryDirectories=[/vendor/lib64, /system/lib64]]] couldn't find ".so"

As far as I have known, the built apk file would contains native libs in separate directories which would be named according to the abis used by the native library. So I checked location <apk name>/lib/arm64/ of installed android device for native library I built with bazel. In fact, I found the desired native dynamic library at this position, but I don't know why the intalled apk on android device could not found it.

After I got the error log information, I knew it is because the apk did not find the native dynamic library in the directory where it should be. I manually cp the dynamic library into /system/lib64, which is system library directory, and as the error log indicated. Then the apk could run and work on android device!

So this is quite wired? is it because bazel pack the apk in a way that the android system could not recognize?

Minimal example

I built examples/android/java/bazel:hello_world in bazel source code directory with command:

bazel mobile-install --fat_apk_cpu=arm64-v8a --compilation_mode=opt examples/android/java/bazel:hello_world --start_app -s

It reproduced the same situation as I have encountered. The App failed to run with log information:

E/AndroidRuntime( 5609): FATAL EXCEPTION: main
E/AndroidRuntime( 5609): Process: bazel.android, PID: 5609
E/AndroidRuntime( 5609): java.lang.UnsatisfiedLinkError: com.google.devtools.build.android.incrementaldeployment.IncrementalClassLoader$DelegateClassLoader[DexPathList[[dex file "/data/local/tmp/incrementaldeployment/bazel.android/dex/incremental_classes1.dex"],nativeLibraryDirectories=[/vendor/lib64, /system/lib64]]] couldn't find "libhello_world.so"

I looked into the apk, and found the native dynamic library at position:

/data/app/bazel.android-1/lib/arm64/libhello_world.so

After I cp libhello_world.so /system/lib64/, I re-run the App, and it worked.

Environment info

• Operating System:
  Ubuntu 14.04

• NDK,SDK Info:
  android_sdk_repository(
  name = "androidsdk",
  path = "/usr/local/android/sdk",
  api_level = 22,
  build_tools_version="22.0.1",
  )
  android_ndk_repository(
  name = "androidndk",
  path = "/usr/local/android/ndk",
  api_level = 22,
  )

• Bazel version (output of bazel info release):
  release 0.4.0

Thank you, any suggestion would be appreciated.

[aj-michael]

Thanks for the detailed report, I'm investigating now. For reference, does this occur with any Bazel app with JNI that you mobile-install, or just one particular app? Oops, totally blanked here, your report includes the hello world sample app. This is certainly a bug.

[aj-michael]

Actually, would you mind trying to run your app via bazel build and adb install? I would be curious to know if the error persists, because it appears this may be a bug in our incremental deployment code that mobile-install uses.

[nxty2011]

Hello, I understand it's quite a dirty work to deal with so many issues, so I try my best to describe it clear.

I have checked with bazel build and adb install with both my project and hello_world example project, as a result, the error is gone for both projects. So I guess it is the problem of installation process in bazel mobile-install.

Two more things hope to provide more info.

• I have tried use default parameter for --fat_apk_cpu, which is set to armeabi-v7a I guess from commands bazel processed. Then the app could successfully run on android device following the same process(except without setting the --fat_apk_cpu option).

• When looking into the generated apk with a archive tool, I found the structure is like:

app.apk/lib/arm64-v8a/native.so

But after I check the app directory on android device via adb shell, I found this:

app/lib/arm64/native.so

Notice the different names of abi directories under lib directory, either bazel mobile-install or bazel build, adb install would produce this. The difference is former method worked, while the latter one threw out lib not found error.

That's about it, thank you.

[aj-michael]

Thanks for all of the details! I suspect the problem is due to a bug in Bazel using special code to handle native libs during mobile-install here:

bazel/src/tools/android/java/com/google/devtools/build/android/incrementaldeployment/StubApplication.java

Line 353 in c3f6a6a

private String copyNativeLibs(String dataDir) throws IOException {

It is very strange that this works for armeabi-v7a but not arm64-v8a.

[jin]

Turns out that the native libs were never incrementally installed because --android_incremental_native_libs is off by default.

By switching it on, you can now use --fat_apk_cpu=arm64-v8a correctly, and you'll get the correct error message while trying to install a .so built for armeabi-v7a on an arm64-v8a device:

12-21 14:30:16.272 10596 10596 E AndroidRuntime: java.lang.UnsatisfiedLinkError: dlopen failed: "/data/data/com.example.android.bazel/incrementallib/libapp.so" is 32-bit instead of 64-bit

I'll send a change to switch it on by default.