Slowdown of Termux when accessing shared internal storage

[arocoun]

Problem description
An image processing and uploading program I use, mapillary_tools, has started running much slower on Termux -- at less than 1/10 normal speed.

I suspect it has something to with the update to Android 11, and how Android allocates resources to Termux. Termux actually SPEEDS UP DRAMATICALLY for a second while my phone is adjusting to being rotated!

Video of this odd behavior: https://youtu.be/O5wTd0V7610

I've tried resetting my phone, reinstalling Termux, reinstalling mapillary_tools, and making sure battery optimization is off for Termux. After taking WAY too long searching, I haven't found that anyone else has had this problem.

Steps to reproduce
Not sure what lead to this.

Expected behavior
Termux and any programs on it run at normal speeds.

Additional information

• Termux application version: 0.101 from Play Store
• Android OS version: 11
• Device model: Galaxy A51 5G

[arocoun]

I tested Termux's speed at copying files (using cp command) while on verbose. Its copy speed is ludicrously slow -- but once again speeds up for a second when I rotate my phone.

Why might Termux speed up while adjusting to rotation, and how do I keep its speed decent?

[agnostic-apollo]

Well, all these can be easily solved by using a servo motor and mounting your phone on it and keep rotating it while running termux commands.

[agnostic-apollo]

But can be looked into. Does it still happen if you acquire a wakelock, even though it's to prevent sleep.

[arocoun]

Ooh! Big discovery! Termux ONLY slows down when dealing with files from outside it's own directory -- that is, files that can only be accessed after the termux-setup-storage command.

[agnostic-apollo]

Well, external shared storage is emulated with sdcardfs, so performance would definitely be slower. But weird that rotation has an effect on it.

[arocoun]

Hmm. After looking it up, it seems that many programs access storage more slowly since Android 11. Probably something to do with beefed up security for every interaction with storage.

So this is likely more of an Android problem than a Termux problem. Still, I'll still happily accept advice, haha!

Edit: I guess a good temporary solution is to move files to Termux's on directory with the "mv" command, to minimize programs' interactions with shared storage.

This has worked very well for me, since mapillary_tools interacts with tens of thousands of small files multiple times while processing images.

[agnostic-apollo]

Scoped storage is currently not being applied to termux, however, android would still be doing extra processing internally to handle scoped storage itself, so performance impact could/would likely be there.

Moving would depend on what you are actually doing. If you are repeatedly accessing the same file at different times or are in place editing, then the one move operation would be worth it for faster operations later. However, if there is only a one time read operation for processing a file, then moving it first would make that two operations so would be twice as slow.

And you are right, slow external storage issues are not a termux issue, and can't likely be solved by it. Your rotation issue is weird currently, that can be looked into.

[ghost]

Does touching the screen (scrolling a terminal or notification list) have similar effect as rotating screen?

This seems like either misconfigured bus or cpu frequency scaling. This also can be done intentionally by manufacturer, so device reaching its full performance only in certain cases for battery power saving when user doesn't interact with the device.

Never observed such behavior except on custom kernels with "powersaving" configs.

[agnostic-apollo]

Actually, you may be right, it might be a boost in CPU speed instead of storage speed, duh!. The interactive CPU governor for the kernel has settings like the input_boost for boosting CPU freq for a few milliseconds when touchscreen is touched. Not sure what android 11 (or his device) is using as the default governor these days but other governors have similar settings and this could be related. Rotating requires redrawing of activities, so a boost at that time would make sense, that's likely what's happening.

[ghost]

Stock Samsung kernel uses interactive. That's on Galaxy S7, not sure what is used on other devices.

[agnostic-apollo]

Yeah, can't really guess without checking. There are also ARM big.LITTLE CPU cluster specific governors. On my LG G5, the big cores (0 and 1) are set to interactive, while the little cores (2 and 3) are set to performance governor. Although, I'm using a custom kernel, so not sure what the default was. OP can check CPU-Z for detecting CPU scaling.

[RalfWerner]

@arocoun My Galaxy device was also updated on Android 11 yesterday. Your mapillary_tools example is poor to reproduce. If it is I/O problems on /sdcard, you could please create a simple repeatable bash example with which you and I can repeat that?

[arocoun]

@arocoun My Galaxy device was also updated on Android 11 yesterday. Your mapillary_tools example is poor to reproduce. If it is I/O problems on /sdcard, you could please create a simple repeatable bash example with which you and I can repeat that?

My problem has nothing to do with an SD card, but rather the phone's internal storage (ie when you first open Termux, the directory you go to with cd storage and all of its subdirectories).

I have no idea how to form an easily-reproduced situation where Termux interacts with internal storage thousands of times, then interacts with files in it's own directory thousands of times to compare the speed.

(Probably something using the dd command to test I/O for locations "storage/test" and "test", but I'm too much of a newbie to do that without breaking something).

I use Termux / Linux command line for one thing, mapillary_tools. My experience beyond that is limited. Perhaps someone with more experience could come up with something.

[RalfWerner]

but rather the phone's internal storage

ls -l /sdcard /storage/self/primary are the symlinks for the same storage as the symlinks in the ls -l ~/storage. Everything internal storage, if no external card is inside the device. So we mean the same!
I am not sure if your problem is related to I/O on /sdcard. It occurs at Configure* Events (rotation) and performs a complex process with pictures/videos that may also have to do with network activity, cache etc. This is not visible in your YouTube Reference.

I do not wont to manage an account for mapillary. That's why the request a simple bash script like:
typeset -i n=0; while true; do n+=1; f=$(find /sdcard/DCIM -type f); cp $f storage; echo $n; done

Here all camera recordings of your device are copied in Termux ~/storage until you stop the process. (Ctrl+c) e.g. at 1000. The rotation of my device has no influence on the performance (expected) - yours?
But simple packages as fluxbox feh also not -more below.

[arocoun]

Oh, thanks for clarifying what you meant by /sdcard! Like I said, I don't have much experience in Termux. Also, thanks of thinking so much about this!

What you saw in the video was not files being copied. Rather, it was one small part of image processing, where the images are analyzed. Thousands of tiny descriptive files are created, and they are written to or read from hundreds of thousands of times.

When these files are somewhere in shared storage, this process takes over 30 minutes. When these files are in Termux's directory, the process takes less than 2 minutes.

Regarding your copying experiment: I'm thinking that if the slowdown for each file interaction is, say, 10 milliseconds, you wouldn't notice a slowdown copying 1000 image files.

When I did the copying mentioned in my second post, I was copying almost 10,000 tiny files created by the aforementioned process -- so the per-interaction delay played a bigger role. When I copy 1000 image files now, I see almost no slowdown.

[ghost]

When these files are somewhere in shared storage, this process takes over 30 minutes. When these files are in Termux's directory, the process takes less than 2 minutes.

Even though that shared storage (/sdcard, /storage/emulated/0) and Termux home directory are stored on same partition, there still some differences. As shared storage is mounted through overlay, certain performance penalty is expected, especially in random-access mode.

[arocoun]

When these files are somewhere in shared storage, this process takes over 30 minutes. When these files are in Termux's directory, the process takes less than 2 minutes.

Even though that shared storage (/sdcard, /storage/emulated/0) and Termux home directory are stored on same partition, there still some differences. As shared storage is mounted through overlay, certain performance penalty is expected, especially in random-access mode.

Probably true there'll be some slow down in any case! But before Android 11, the slowdown wasn't noticeable.

I now find myself wondering if mapillary_tools is uncommon in the number of file interactions it does, and if this problem is not likely to affect other folks.

[ghost]

But before Android 11, the slowdown wasn't noticeable.

Probably because Android 11 implemented changes to the overlay filesystem (sdcardfs or how it is called) to meet the requirements of the further scoped storage enhancements.

[RalfWerner]

there still some differences. As shared storage is mounted through overlay

@xeffyr I can reproduce that! When I use in the example above cp $(find storage -type f) ../usr/tmp (termux only - repeat 100 times) I need 11 sec instead of 16 from /sdcard and exactly the same relation, when I repeat it 200 times (60% more). When I repeat this on an Android 8 device, the relation is revers. So you are right - Android 11 reduced the /sdcard performance significant! But this don't explain the Configure* Event behavior of @arocoun

I use Termux / Linux command line for one thing, mapillary-tools

@arocoun is this the *first time, that you use mapillary on a _rotatable device? fluxbox e.g. reconfigure the graphic screen, when you rotate (on a servo motor:) or resize it in Pop-View (possible in Android 11).

[annihat]

Hi, another interested but unskilled user here, @arocoun have you tried using <sdcard>/Android/data/<package name> ? For some time it's been the only directory on the sdcard usable by an app without specific permissions so might not be slowed by the system. (on Android 2.3.5 it's <sdcard>/.android/data/<package> I don't know when it changed)

On my device it's /storage/emulated/0/Android/data/com.termux but Termux doesn't have permission to create the directory, you have to do it with a file manager (I was very surprised the setup routine hadn't created it). It's created with Owner: <package> & Group: sdcard_rw & permissions: rwxrwx--x so other apps can access it.

[agnostic-apollo]

The /sdcard is a "symlink" to /storage/emulated/0 as the default on android, which is emulated by sdcardfs. So /sdcard/Android/data/<package_name> would have the same performance degradation as any other path under /sdcard.

You can create the directory from termux by running termux-setup-storage.

[annihat]

You can create the directory from termux by running termux-setup-storage.

Termux, like Android, is a steep learning curve.

What does this command do, exactly? What would happen if I ran it now that I've created the directory manually? Would it do anything extra?

[ghost]

@annihat It will create that directory again. termux-setup-storage sends an intent to Termux application which by itself will wipe that directory (if exists) and rebuild symlink set from scratch.

Here is a Java function which does that: https://github.com/termux/termux-app/blob/master/app/src/main/java/com/termux/app/TermuxInstaller.java#L192

[annihat]

@xeffyr , @agnostic-apollo , thanks for that, it produces a useful-looking group of directories (including the root of the sdcard) & asks for permission to use them. It doesn't do anything with the one already-writable directory though.

[agnostic-apollo]

Welcome.

The termux setupStorageSymlinks() function only deletes the ~/storage directory. It won't delete any Android/data directory, the call to externalFilesDirs() automatically creates any directories that are missing, on any external storage.

https://wiki.termux.com/wiki/Termux-setup-storage

[annihat]

@agnostic-apollo , I like termux-setup-storage, I really do, but it's an additional feature & correctly so.

My opinion is that the installer from the appstore should create <sdcard>/Android/data/com.termux & provide a link to it in $HOME because no permissions are needed & it can still be used if the user chooses not to grant them (special-use installations like the one in this thread wouldn't seem to need general permission when a specific permission is already available)
(I also think a readme.txt should be placed in there explaining that this directory is on the sdcard & accessible to other apps so can be used for file transfers without needing the user to grant permissions)

[RalfWerner]

The authorization can be granted with the Android Setup as well astermux-setup-storage.
@annihat If you have an Android 11 device (which deals with this issue) and/or additionally external card there are special features (duplex). The patha=Android/data/com.termuxis also special (is disposed of with the current app with targetSDK<30).

[agnostic-apollo]

My opinion is that the installer from the appstore should create /Android/data/com.termux & provide a link to it in $HOME because no permissions are needed & it can still be used if the user chooses not to grant them (special-use installations like the one in this thread wouldn't seem to need general permission when a specific permission is already available)
(I also think a readme.txt should be placed in there explaining that this directory is on the sdcard & accessible to other apps so can be used for file transfers without needing the user to grant permissions)

Accessing Android/data directory of other apps has been revoked in android 11 unless apps use hacks, check here.

And everybody doesn't need symlinks in their $HOME directory, it's a user choice. For some absolute paths are preferred, if termux scripts of users (that may be shared with others) are relying on the symlinks, then it needs to be ensured that termux-setup-storage has been run after termux installation or after/during script installation on a different device and that symlinks exist and refer to correct paths. This is specifically important for removable external sd cards. With absolute paths such checks aren't required, user just needs to grant storage permission on installation and nothing more. It's better for background scripts too for which a prompt can't be shown to users.

Automatically creating the Android/data directories could be considered although that is an additional IO process to all slow storages on every termux startup, but automatically creating only their symlinks likely isn't the best way. The termux-setup-storage checks if the ~/storage directory already exists before asking user if it should be wiped to recreate the symlinks. So if we start creating only the Android/data symlinks without the others, then the ~/storage directory would be created and it may be incorrectly assumed by termux-setup-storage and current user scripts that ALL symlinks exist if only the check for ~/storage directory existence is done. At the very least it would require more checks for it to be doable, there are probably other issues too.

[RalfWerner]

Back to this Issue - with my 29+ check process I've got:

lrwxrwxrwx 1 u0_a170 u0_a170     34 Feb 15 15:35 fd -> /storage/1A55-0F06
lrwxrwxrwx 1 u0_a170 u0_a170     34 Feb 26 09:51 fda -> fd/Android/data/com.termux/files
lrwxrwxrwx 1 u0_a170 u0_a170     34 Feb 15 15:34 sd -> /storage/emulated/0
lrwxrwxrwx 1 u0_a170 u0_a170     34 Feb 26 09:52 sda -> sd/Android/data/com.termux/files
drwx------ 2 u0_a170 u0_a170   4096 Feb 26 10:20 storage

So I find (duplex) withls fd/D* sd/D* 6 paths (sd,fd)x(DCIM,Documents,Download) but only 2 paths with ls storage/d* (miss documents). I had already documented the performance loss (60%) above but for the ??a path be does not yet checked.
This I have done now with the same process on an Android 11 Pixel emulation. Here I could not detect losses, but rather a performance improvement of 30-50% on the??a paths.

[agnostic-apollo]

Closing since likely an Android 11 OS issue. Users should use termux private app data directory ~/ (except ~/storage) for multi-access processing of files instead of shared storage. Can be opened again if issue reason is proven otherwise.

[RalfWerner]

Users should use termux private app data directory for multi-access processing of files instead of shared storage

This requires enough memory. My normale Termux usage would then fail to 2 of my 3 devices. With Target 28, however, it is never a real problem to save everything except u/ into fd (above). But with Android 11 *setup-storage is sometimes not sufficient and manual Deny->Allow toggle (workaround) is required.
Maybe here it could be changed. Also at 29+ I have not found any real restrictions.

[agnostic-apollo]

This requires enough memory.

Then go complain to android OS devs, although unlikely to be solved since its part of design now or use root to access /data/media/0 directly without sdcardfs.

Maybe here it could be changed

It's possible to detect the condition from within termux about storage permission being granted but still permission denied errors, but I don't know how to reproduce that situation to test it. Also don't know if that happens when targeting sdk 29+. If someone does, let me know.

[RalfWerner]

... or use root to access /data/media/0 directly without sdcardfs.
Also don't know if that happens when targeting sdk 29+. If someone does, let me know.

root would be nice to the check, but is not a solution for the IP Obj on Android (Checks with 29+ Artifakt). In AVD I use one virtual sdcard (ldcard) on different PCs and with various emulators/SDK-versions. They are all different. Please tell me which emulator you used to test, so that I can generate test cases?
As with real devices, only one emulator+ldcard can be used at the same time and change the data to ldcard should always lead to restart on other emulators.
With my real devices I currently have a maximum of 64G, which I have also exchanged (with needle). In all cases, since my Success Message, I was able to use the sdcard - with patience and workaround. So we do not need to strive to the Android developers.

The updated-flavor Branch by @fornwall generates two *.apk, but not the right for 29+ as Android-10.
But app-current-debug.apk is identical with your last v113 artifact.