[question] [v.0.13.1] [android] getting IncompatibleClassChangeError when trying to bind .aar in C# project [SOLVED by adding ProGuard rules]

[dodikk]

Describe the bug

Getting Java.Lang.IncompatibleClassChangeError: no non-static method "Lcom/cossacklabs/themis/SecureCellSeal;.decrypt([B[B)[B" in Release configuration in C# android project.
When decrypting "obfuscated" string constant on app start.

Any ideas? Have you seen anything like this in some java or kotlin android project?

To Reproduce

On app start I try to decrypt an "obfuscated" string constant

_secureCell = SecureCell.SealWithKey(masterKeyData);
_secureCell.Decrypt(cipherTextBytes, context);

Getting an error in Release configuration:

Java.Lang.IncompatibleClassChangeError: no non-static method "Lcom/cossacklabs/themis/SecureCellSeal;.decrypt([B[B)[B"
[orion.mobile]   at Java.Interop.JniEnvironment+InstanceMethods.GetMethodID (Java.Interop.JniObjectReference type, System.String name, System.String signature) [0x0005b] in <42d2b7086f0a46efb99253c5db1ecca9>:0 
[orion.mobile]   at Android.Runtime.JNIEnv.GetMethodID (System.IntPtr kls, System.String name, System.String signature) [0x00007] in <3080427739614e60a939a88bf3f838d5>:0 
[orion.mobile]   at Com.Cossacklabs.Themis.SecureCell+ISealInvoker.Decrypt (System.Byte[] p0, System.Byte[] p1) [0x00017] in <cd618986d1ce4194b63cdd3366dad291>:0 
[orion.mobile]   at Themis.Droid.CellSealDroid.UnwrapData (Themis.ISecureCellData cipherTextData, System.Byte[] context) [0x0007e] in <a492e7118e094c3296442a386fe5d80e>:0 
[orion.mobile]    --- End of inner exception stack trace ---

Expected behavior

N/A - this issue is a question

Environment (please complete the following information):

• OS: Android 10, build 00WW_2_250
• Hardware: Nokia 7.2
• Themis version: 0.13.1
• Installation way:
  • via package manager
  • built from source

Additional context

Sorry for asking in a wrong place if I'm violating any of your policies with this ticket.

I've spent a while debugging it and am a bit desperate at the moment.
I know you do not support that C# and Xamarin.Forms but filing this question just in case you've seen a similar issue in some java or kotlin android project.

Unable to share a sample project

since that does not reproduce on https://github.com/dodikk/themis-xamarin-prototype/tree/bugfix/v0.13.2/droid-strip-symbols
Only in a project under NDA, unfortunately.

• I've checked the data I'm getting the failure on. It has been encrypted with wasm-themis CLI tools. Also I can decrypt the
  data collected from my app's exception (again, with wasm-themis CLI tools)
• The same app code and bindings work in debug configuration
• apk seems to have SecureCellandSecureCellSeal class symbols (checked via "profile apk" UI in android studio)
  

[vixentael]

Hm, that doesn't look right.

@dodikk thank you for notifying, although we don't support C# at the moment, but let's try to find a workaround. Do I understand correctly, that the sample project doesn't have the same issue?

Right now, I do not have meaningful suggestions yet, but I do have a set of experiments that might help to locate the issue.

---

Quick ideas to try on the "real" project

1. Does the issue on real project happens on some particular Android emulator or device? Can you try different device / emulator? Please try 64-bit emulator if you're using x32.

2. Does the issue on real project happens only for Android 10? Can you try different Android version?

3. Can you please try older versions of Themis, like 0.13.0 or 0.12.2 to see if the issue still persists on your main project? Themis 0.13.0 has introduces some changes in API that might be causing this issue (but pure guess). Please install 0.12.2 using maven { url "https://dl.bintray.com/cossacklabs/maven/" }, as it's not available in JCenter.

4. Can you please specify how exactly do you install Themis in your real project? I see that you're installing Themis using maven for your sample project.

Please try using Themis 0.13.1 from JCenter directly (just indicate jcenter(), don't indicate maven { url "https://dl.bintray.com/cossacklabs/maven/" }) to see if this will have any changes. We haven't published officially that Themis supports JCenter yet, so please refer to the non-published docs if you have any questions.

---

More experiments trying to find the difference between sample project and real project

1. Can you please list all the differences between the sample project and the real: the way of Themis installation, usage of same/different Themis APIs, usage of same/different Themis versions, usage of same/different gradle versions, etc?

2. Can you please use the exact data (encrypted message) that breaks real project to see if it breaks a sample project?

[dodikk]

Do I understand correctly, that the sample project doesn't have the same issue?

Right.
Same device, same thems.aar binary ==> different results for some reason.

[dodikk]

Please try using Themis 0.13.1 from JCenter directly (just indicate jcenter(), don't indicate maven

Actually, I've installed the package like this.

curl -L --output themis-0.13.1.aar https://bintray.com/cossacklabs/maven/download_file?file_path=com%2Fcossacklabs%2Fcom%2Fthemis%2F0.13.1%2Fthemis-0.13.1.aar

P.S. I had some difficulties integrating maven/gradle into C# project build, so I took that shortcut.
It seems to have been working fine for v0.12.0

[dodikk]

Can you please specify how exactly do you install Themis in your real project? I see that you're installing Themis using maven for your sample project.

That sample project might have some failed experiments "leftovers".
So, as shown in the comment above,

I'm using curl to download themis binary

[dodikk]

Can you please use the exact data (encrypted message) that breaks real project to see if it breaks a sample project?

Will come back with this a bit later today.
As well as with "another device" and "another droid version" checking.

[dodikk]

@vixentael thanks for blazing fast reply. I appreciate a lot.

[ilammy]

Java.Lang.IncompatibleClassChangeError: no non-static method "Lcom/cossacklabs/themis/SecureCellSeal;.decrypt([B[B)[B"

Hm... Can this be because SecureCellSeal is not public? It's a package-private class implementing a public SecureCell.Seal interface, and the user code is expected to use the interface. SecureCell#SealWithKey returns this interface. It seems like C# interop tries to be smart and devirtualize the call but that doesn't work quite well.

Another idea is that the method name may be not correct. JNI has some peculiar rules, especially when it comes to overloads. It seems correct – decrypt method of com.cossacklabs.themis.SecureCellSeal which takes two []byte arrays and returns one – but there's also another decrypt method which may throw a spanner in the works. But again, the signature and the method name seem to be correct.

[dodikk]

Hm... Can this be because SecureCellSeal is not public?

Thanks @ilammy . I've thought about that. But I did not have enough time to fork themis and rebuild it with the mentioned changes just yet. Will try that when all other "low hanging fruit" ideas prove being unrelated to the issue.

[dodikk]

Can you please use the exact data (encrypted message) that breaks real project to see if it breaks a sample project?

That same data from the "real project" does not break my "sample project".
Checked on the very same nokia device.

/cc @vixentael

[dodikk]

Can you please list all the differences between the sample project and the real:

• the way of Themis installation ==> same.
  Just downloaded that aar with curl and generated xamarin bindings on top of it. No significant diff between the correspondingcsproj files (C# projects) discovered.
• usage of same/different Themis API ==> same.
  The C# code of binding/wrapper projects is identical (for "sample" and "real" projects). That code is available in dodikk/themis-xamarin-prototype repository.
• usage of same/different Themis versions ==> same
  My "real project" bindings are based on the very same aar binary as the "sample project".
• usage of same/different gradle versions, etc? ==> N/A.
  In my case that was just curl. Copying the same binary on the file system does not help either.

/cc @vixentael

[dodikk]

@vixentael @ilammy can that be caused by mismatch of the android studio versions?
Which one did you use to produce that themis-0.13.1.aar artefact?
Might updating to the most recent toolchain help mitigating the issue?

[dodikk]

Can you try different device / emulator? Please try 64-bit emulator if you're using x32.

Same results on x64 - Nexus 5 - Android Pie emulator (both for the "real project" and for the prototype; on CipherText data from "real app")
/cc @vixentael

[UPD] same results on Samsung SM-J200H and android v5.1.1 hardware

[ilammy]

can that be caused by mismatch of the android studio versions?

Well, everything is possible, but I don't think that the toolchain versions matter very much here. The issue seems to be on the Java side of things, not in the native interop, so NDK should not affect it. Java/Android alone have a very good story with compatibility (I'm looking at you, Apple), so I doubt that Java compiler is doing something weird here.

Which one did you use to produce that themis-0.13.1.aar artefact?

We use a standalone toolchain to build it, not Android Studio. It's available in cossacklabs/android-build Docker image. It is somewhat outdated, but that's a result of brittle equilibrium of bugs and incompatibilities that is able produce results. We try updating it from time to time, but it's not kept on the bleeding edge.

[dodikk]

Well, everything is possible, but I don't think that the toolchain versions matter very much here.
We use a standalone toolchain to build it, not Android Studio.

Ok. Then I'll try other things before applying your toolchain to my app's build environment.
Thanks for providing context, @ilammy

[UPD] a link to cossacklabs/android-build just un case for future readers of this topic
https://github.com/cossacklabs/dockerfiles/blob/master/android-build/Dockerfile#L26

[dodikk]

I'm trying the guides below to check the "troubles from C# world"

1. https://gist.github.com/JonDouglas/dda6d8ace7d071b0e8cb
2. https://www.jon-douglas.com/2017/04/13/linker-bitdiffer/

But no success fo far.

Things start to break when C# apk size optimizer (which they call "linker" for some reason) is set to more aggressive mode (None ==> Sdk Assemblies Only modes mentioned in ).
So the high-level hypothesis "the apk size optimizer is stripping too much" looks like the best candidate. On the other hand, the symbols mentioned in the exception are there in the apk

P.S. This comment is not a question but rather an update on "what I've discovered so far"

[vixentael]

thank you @dodikk, we're watching your progress! Linker's optimization is a good guess, do you think it's possible to disable it / change optimization level?

[dodikk]

do you think it's possible to disable it / change optimization level?

@vixentael sure thing it is possible. But I cannot use None since that would bloat the apk size with the entire mono runtime. Which is highly unwanted (ok for debug and in-house; but bad for release and playmarket).

So I need a more fine-tuned fix and can't solve the issue that easily.

Also, xamarin allows specifying manually "which symbols should stay" (see the link below). But I'm viewing that as "weapon of last resort" since that approach would be rather time-consuming (just like writing a makefile without those cmake or autotools helpers). So I'm hoping to find an easier fix and still "on a quest" to find one...
https://docs.microsoft.com/en-us/xamarin/cross-platform/deploy-test/linker

[UPD] this docs page of MSDN seems relevant and helpful

• https://docs.microsoft.com/en-us/xamarin/android/deploy-test/release-prep/proguard?tabs=macos
• https://devblogs.microsoft.com/xamarin/androids-d8-dexer-and-r8-shrinker/
• https://developer.android.com/studio/build/shrink-code
• [!!!] Issues around getting D8 and R8 to work with new Contributor Gallery dotnet/android#2693
  ** sharing for future readers of this thread, just in case

[dodikk]

Solved

What had to be done:

1. Add proguard.cfg file to the build
2. Add instructions to "ask" proguard to not remove themis classes

-keep class com.cossacklabs.themis.**
-keep class com.cossacklabs.themis.** {*;}

/cc @vixentael @ilammy

[dodikk]

Do you accept pull requests to your documentation?

@vixentael , could you please share the URL of the corresponding repo?

As far as I've understood so far... Native android devs might run into such issue as well. So documenting this pitfall (in the Installing JavaThemis for Android development) guide might make sense.
Unless either

1. you've done so already but I have not noticed that hint
2. maven or gradle deals with proguard automatically by default
3. Having the solution documented in this ticket as is good enough

[ilammy]

Do you accept pull requests to your documentation?

@vixentael , could you please share the URL of the corresponding repo?

Sure we do. The docs live here: https://github.com/cossacklabs/product-docs

Though, I'm not sure if this is an issue strictly related to Themis. Android Studio sets up ProGuard in Android projects by default, but the issue seem to be generic enough to happen with any other library if that's ProGuard 'optimizing away' parts that it does not deem essential when in fact they are.

I think, if this issue affects multiple projects then it's worth documenting to caution developers. Otherwise, I don't think that Themis docs should bloat into a list of trivia and howtos on configuring (effectively) third-party components of the build. I guess, if this issue pops up for someone else they can be directed here at first.

@vixentael, what's your take?

[vixentael]

I'd mention this on Installation page, maybe like a separate paragraph "Configuring ProGuard rules" under "Installing stable version with Gradle".

I totally agree with @ilammy that we shouldn't put all the tips&tricks into docs, but this is a tricky issue which might take some time to troubleshoot, so having it mentioned somewhere with link to this issue makes sense to me.

@dodikk can you add these proguard rules to your sample project? As an illustration.

[dodikk]

can you add these proguard rules to your sample project? As an illustration.

@vixentael sure. Of course I'll do that.
Also I was planning to document this stuff in the sample's readme.

[vixentael]

Amazing, really waiting forward!

[dodikk]

Amazing, really waiting forward!

@vixentael , I have updated develop branch of the demo project. Both the project and readme.
https://github.com/dodikk/themis-xamarin-prototype/tree/develop

Will update the default branch of demo repository later, after more testing.

[vixentael]

Add info to docs here cossacklabs/product-docs#28