Investigate Scoped Storage in Android 11 and document how it works with Realm

[johnsabilla]

Goal

App with Realm runs fines on previous and current version of Android except on Android 11. I understand this is a preview OS. I just want to know if this issue is known, if anyone else is experiencing this.

Actual Results

2020-06-02 10:16:31.702 2915-2915/com.patapon.disconnect E/REALM_JNI: jni: ThrowingException 5, flock() failed: Function not implemented in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 107, .
2020-06-02 10:16:31.703 2915-2915/com.patapon.disconnect E/REALM_JNI: Exception has been thrown: Unrecoverable error. flock() failed: Function not implemented in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 107
2020-06-02 10:16:31.704 2915-2915/com.patapon.disconnect E/EventBus: Could not dispatch event: class com.patapon.ble.events.TanthenticationState to subscribing class class com.patapon.disconnect.components.AppComponentStore
io.realm.exceptions.RealmError: Unrecoverable error. flock() failed: Function not implemented in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 107
at io.realm.internal.OsSharedRealm.nativeGetSharedRealm(Native Method)
at io.realm.internal.OsSharedRealm.(OsSharedRealm.java:175)
at io.realm.internal.OsSharedRealm.getInstance(OsSharedRealm.java:251)
at io.realm.BaseRealm.(BaseRealm.java:137)
at io.realm.BaseRealm.(BaseRealm.java:104)
at io.realm.Realm.(Realm.java:163)
at io.realm.Realm.createInstance(Realm.java:499)
at io.realm.RealmCache.createInstance(RealmCache.java:507)
at io.realm.RealmCache.doCreateRealmOrGetFromCache(RealmCache.java:473)
at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:414)
at io.realm.Realm.getInstance(Realm.java:428)
at com.patapon.disconnect.repo.RealmRepository.initialize(RealmRepository.kt:98)
at com.patapon.disconnect.components.TKG.createRepo(TKG.kt:40)
at com.patapon.disconnect.components.AppComponentStore.onTanthenticationState(AppComponentStore.kt:215)
at java.lang.reflect.Method.invoke(Native Method)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:507)
at org.greenrobot.eventbus.EventBus.invokeSubscriber(EventBus.java:501)
at org.greenrobot.eventbus.HandlerPoster.handleMessage(HandlerPoster.java:67)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loop(Looper.java:223)
at android.app.ActivityThread.main(ActivityThread.java:7478)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:549)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:941)
2020-06-02 10:16:31.704 2915-2915/com.patapon.disconnect E/AbbaZaba: [M] Caused by EventBus event: com.patapon.ble.events.TanthenticationState@238c987

Steps & Code to Reproduce

It crashes when instantiating realm

val config = RealmConfiguration.Builder()
.directory(File(Environment.getExternalStoragePublicDirectory(location + "/RealmDb"))
.name("testdb")
.also {
k?.let { key ->
it.encryptionKey(key)
}
}
.build()

Realm.getInstance(config)

Version of Realm and tooling

Realm version(s): io.realm:realm-gradle-plugin:7.0.0

Realm Sync feature enabled: No

Android Studio version: 4.0

Android Build Tools version: 28.0.0

Gradle version: 3.6.3

Which Android version and device(s): Android 11 (R) preview, Google Pixel 3

[cmelchior]

That sounds really surprising flock is a standard Linux system call. I would be very surprised if they changed anything around this.

Most likely it is related to using Environment.getExternalStoragePublicDirectory(). They introduced a lot of restrictions around how you can access External Storage and there is a chance that Realm will no longer be able to store data there, but we haven't investigated that yet: https://developer.android.com/preview/privacy/storage

[Kyba1985]

I also encountered this problem. With Android 11 and the scope model we cannot use external storage but only the app's internal space.

I know that i will migrate all data of the users but at the moment only for test:

migrate this:

File directory = new File(Environment.getExternalStorageDirectory(), ConstantsApp.ROOT_DIRECTORY_FOLDER);
File realmDBFolder = new File(directory, ConstantsApp.REALM_ARCHIVE_DIRECTORY_FOLDER);

to this:

File directory = new File(getExternalFilesDir(null), ConstantsApp.ROOT_DIRECTORY_FOLDER);
File realmDBFolder = new File(directory, ConstantsApp.REALM_ARCHIVE_DIRECTORY_FOLDER);

the app works well after init realm with this file configuration.

[mmarich]

I am experiencing the same issue as originally reported in this ticket under Android 11 on a recent update to a number of customer-owned Samsung devices.

Where it gets interesting is that the error does not occur when accessing the built-in external storage, but only the removable external storage (SD card). In both cases, the Realm database should be located under one of the appropriate directories returned by Content.getExternalFilesDirs().

The specific error is:

Unrecoverable error. flock() failed: Function not implemented in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 101

The devices models reporting this error so far are:

SM-N975U
SM-G781V
SM-G975U
SM-N986U1

All are Samsung devices running Android 11.

I will reach out to these customers and investigate this further. What is rather confusing is that this error does not occur when accessing the built-in external storage.

[vok1980]

OPPO CPH2043 also have this problem.

[mmarich]

An update: I can reproduce this on an emulator running Android API 30 (11) by configuring a secondary SD card as well as forcing my app to use scoped storage in the developer settings.

Again, this only occurs when accessing the secondary SD card.

[vok1980]

The problem is that flock does not work with fat32 fs. Before Android 11 it was not working too, but without error ret code. Beginning with Android 11 it returns ENOSYS.

[mmarich]

The problem is that flock does not work with fat32 fs. Before Android 11 it was not working too, but without error ret code. Beginning with Android 11 it returns ENOSYS.

@vok1980 That makes sense, thanks for commenting. Unfortunately the emulator only supports FAT32, but I will try and test with an ext4 format on a physical device. In the meantime, @cmelchior you may want to change the title of this issue again to reflect the need to address the behaviour of flock() on Android 11.

[kkaefer]

For anyone arriving here: flock indeed fails with errno 38, but POSIX advisory locks with fcntl work on Emulator + Android 11 + Environment.getExternalStorageDirectory() if MANAGE_EXTERNAL_STORAGE is requested (and enabled by the user in the settings).

[kkaefer]

(and be sure to check out the caveats of POSIX advisory locks explained in https://www.sqlite.org/src/artifact/c230a7a24?ln=994-1081)

[Susko3]

Seems like things have changed with the new Android 14 beta. Users are reporting crashes when using directories that the app should have access to.

We're using getExternalFilesDir and realm is crashing with the same assertion:

Assertion failed: r == 0 && "File::unlock()" with (r, (*__errno())) =  [-1, 38]

The docs mention that there should be no problems with accessing this location:

Starting in Build.VERSION_CODES.KITKAT, no permissions are required to read or write to the returned path; it's always accessible to the calling app.

[cmelchior]

@Susko3 getExternalFilesDir should not have been working since Android 11: https://developer.android.com/about/versions/11/privacy/storage ?

[Susko3]

Unless I'm misunderstanding something, the docs you linked explicitly permit using getExternalFilesDirs() (note the plural):

Starting in Android 11, apps cannot create their own app-specific directory on external storage. To access the directory that the system provides for your app, call getExternalFilesDirs().

In the getExternalFilesDirs() docs, it's mentioned that

The first path returned is the same as getExternalFilesDir(java.lang.String).

Besides, I can confirm this working fine on an Android 11 devices (the app targets API 31 - Android 12).

Reading the Android 14 summary page, I'm not noticing anything that could break this functionality.