JNI crash when reading russian / emoji character using simpleQueryForString

[mdvacca]

Hi, we are having a JNI crash when using the simpleQueryForString method to read russian / emoji characters.
This seems to be related with: https://code.google.com/p/android/issues/detail?id=81341
So far we reproduced it in Android 5.1 and 5.0.2
Example character that throws the exception: 😃

Exception:

F/art ( 2028): art/runtime/check_jni.cc:65] JNI DETECTED ERROR IN APPLICATION: input is not valid Modified UTF-8: illegal start byte 0xf0
F/art ( 2028): art/runtime/check_jni.cc:65] string: '........' // JSON DATA REMOVED FOR SECURITY

F/art ( 2028): art/runtime/check_jni.cc:65] in call to NewStringUTF
F/art ( 2028): art/runtime/check_jni.cc:65] from java.lang.String net.sqlcipher.database.SQLiteStatement.native_1x1_string()
F/art ( 2028): art/runtime/check_jni.cc:65] "AsyncTask #5" prio=5 tid=20 Runnable
F/art ( 2028): art/runtime/check_jni.cc:65] | group="main" sCount=0 dsCount=0 obj=0x12f4d880 self=0xb755c590
F/art ( 2028): art/runtime/check_jni.cc:65] | sysTid=2064 nice=10 cgrp=bg_non_interactive sched=0/0 handle=0xb755cbd0
F/art ( 2028): art/runtime/check_jni.cc:65] | state=R schedstat=( 0 0 0 ) utm=1 stm=0 core=2 HZ=100
F/art ( 2028): art/runtime/check_jni.cc:65] | stack=0xa3b36000-0xa3b38000 stackSize=1036KB
F/art ( 2028): art/runtime/check_jni.cc:65] | held mutexes= "mutator lock"(shared held)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #00 pc 00004f60 /system/lib/libbacktrace_libc++.so (UnwindCurrent::Unwind(unsigned int, ucontext_)+23)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #1 pc 00003665 /system/lib/libbacktrace_libc++.so (Backtrace::Unwind(unsigned int, ucontext_)+8)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #2 pc 002569c1 /system/lib/libart.so (art::DumpNativeStack(std::1::basic_ostream<char, std::1::char_traits >&, int, char const, art::mirror::ArtMethod)+84)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #3 pc 00239577 /system/lib/libart.so (art::Thread::Dump(std::1::basic_ostream<char, std::1::char_traits >&) const+158)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #4 pc 000b19fb /system/lib/libart.so (art::JniAbort(char const, char const)+610)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #5 pc 000b2135 /system/lib/libart.so (art::JniAbortF(char const, char const, ...)+68)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #6 pc 000b46ef /system/lib/libart.so (art::ScopedCheck::Check(bool, char const_, ...) (.constprop.129)+922)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #7 pc 000be21d /system/lib/libart.so (art::CheckJNI::NewStringUTF(JNIEnv, char const)+44)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #8 pc 00007bb3 /data/app/com.medallia.mobile-1/lib/arm/libdatabase_sqlcipher.so (???)
F/art ( 2028): art/runtime/check_jni.cc:65] native: #9 pc 00688e3f /data/dalvik-cache/arm/data@app@com.medallia.mobile-1@base.apk@classes.dex (Java_net_sqlcipher_database_SQLiteStatement_native_11x1_1string*+82)
F/art ( 2028): art/runtime/check_jni.cc:65] at net.sqlcipher.database.SQLiteStatement.native_1x1_string(Native method)
F/art ( 2028): art/runtime/check_jni.cc:65] at net.sqlcipher.database.SQLiteStatement.simpleQueryForString(SQLiteStatement.java:154)
F/art ( 2028): art/runtime/check_jni.cc:65] at com.medallia.mobile.model.SqliteStore$3.doInBackground(SqliteStore.java:317)
F/art ( 2028): art/runtime/check_jni.cc:65] at com.medallia.mobile.model.SqliteStore$3.doInBackground(SqliteStore.java:305)
F/art ( 2028): art/runtime/check_jni.cc:65] at android.os.AsyncTask$2.call(AsyncTask.java:292)
F/art ( 2028): art/runtime/check_jni.cc:65] at java.util.concurrent.FutureTask.run(FutureTask.java:237)
F/art ( 2028): art/runtime/check_jni.cc:65] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
F/art ( 2028): art/runtime/check_jni.cc:65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
F/art ( 2028): art/runtime/check_jni.cc:65] at java.lang.Thread.run(Thread.java:818)
F/art ( 2028): art/runtime/check_jni.cc:65]
D/BtGatt.GattService( 7094): stopScan() - queue size =1

Thanks!
David.

[brodycj]

FYI I found the following: http://stackoverflow.com/questions/16786739/how-to-use-unicode-in-android-resource

It does seem related to https://code.google.com/p/android/issues/detail?id=81341, which is now slated for a future release.

I will add a test case to reproduce this issue when I get a chance.

[brodycj]

I contributed a test scenario in sqlcipher/sqlcipher-android-tests#10.

[developernotes]

Hello @mdvacca

From the link provided by @brodybits, there appear to be issues with various Android OS versions, including the versions you reported above supporting character above the Unicode Basic Multilingual Plane (U+0000 – U+FFFF) which would include the 😄 (i.e., U+1F603) character you mention above.

[mdvacca]

Hi @developernotes, Is there any plan to fix this bug for versions previous to Android 6.0?
In that case what is the ETA?

Thanks,
David.

[developernotes]

Hello @mdvacca

The issue as we see it lies in the Android OS in terms of how it performs string handling for characters outside of the BMP, so it is not something the SQLCipher for Android project can address directly.

[mdvacca]

Yes, I was wondering if any workaround can be applied to avoid using the NewStringUTF method.

Here https://code.google.com/p/android/issues/detail?id=64892 narayan suggests:
{comment}
It's also worth mentioning that we could also convert UTF-8 from the input to UTF-16 and call NewString instead of NewStringUTF.
{comment}

What do you think?

Thanks for your help!
David.

[developernotes]

Hello @mdvacca

Converting usage from UTF-8 to UTF-16 and then using NewString instead of NewStringUTF would be interesting. There are a few places where this is used within the JNI layer. Is this something that you would like to take on yourself?

[mdvacca]

Yes, I will work on it and then send a pull request with the change.
We are adding support for several languages and we are currently blocked because of this issue.

Thanks!