Could run any linux under proot in Android 8.1

[darkgeek]

Hi,

I have an OnePlus One phone with Android Extended ROM v5.4 (say, it's Android 8.1.0) flashed. I installed the latest Termux from Google Play yesterday. Then I tried to install termux-ubuntu, termux-fedora and termux-archlinux, but none of them work unfortunately.

For example, I try to start termux-ubuntu, it says:

u0_a105@localhost:~/Shares/termux-ubuntu$ ls
README.md  binds  start-ubuntu.sh  ubuntu-fs  ubuntu.sh  ubuntu.tar.gz
u0_a105@localhost:~/Shares/termux-ubuntu$ ./start-ubuntu.sh 

[1]+  Stopped                 ./start-ubuntu.sh
u0_a105@localhost:~/Shares/termux-ubuntu$ 

No other errors though.

And the content of start-ubuntu.sh is:

#!/data/data/com.termux/files/usr/bin/bash
cd $(dirname $0)
## unset LD_PRELOAD in case termux-exec is installed
unset LD_PRELOAD
command="proot"
command+=" --link2symlink"
command+=" -0"
command+=" -r ubuntu-fs"
if [ -n "$(ls -A binds)" ]; then
    for f in binds/* ;do
      . $f
    done
fi
command+=" -b /system"
command+=" -b /dev"
command+=" -b /sys"
command+=" -b /proc"
## uncomment the following line to have access to the home directory of termux
#command+=" -b /data/data/com.termux/files/home"
command+=" -w /root"
command+=" /usr/bin/env -i"
command+=" HOME=/root"
command+=" PATH=/usr/local/sbin:/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/games:/usr/local/games"
command+=" TERM=$TERM"
command+=" LANG=$LANG"
command+=" /bin/bash --login"
com="$@"
if [ -z "$1" ];then
    exec $command
else
    $command -c "$com"
fi

If I try other distros, they all say Stopped like this. However, they all work flawlessly when my phone is Android 7.

So how could we fix this? If you need more information or test, just tell me. Thanks.

[michalbednarski]

The `[1]+ Stopped` was recent regression, run `pkg up` to install fixed version (5.1.107-8)

[darkgeek]

@michalbednarski Hi, thanks for your help. After updating to the latest version 5.1.107-8, it complains another error:

u0_a105@localhost:~/Shares/termux-ubuntu$ ./start-ubuntu.sh 
proot info: vpid 1: terminated with signal 4

EDIT:

I stop and restart Termux, now it complains Stopped again:

u0_a105@localhost:~/Shares/termux-ubuntu$ ./start-ubuntu.sh 

[1]+  Stopped                 ./start-ubuntu.sh

u0_a105@localhost:~/Shares/termux-ubuntu$ pkg info proot
Hit:1 https://termux.net stable InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
proot is already the newest version (5.1.107-8).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

[michalbednarski]

I don't have ready answer for that, I've recently tested with TermuxArch, I'll try with termux-ubuntu when I'll be at home. Generally I ask for results of dpkg --print-architecture uname -a Then I'll see if I can reproduce it myself

[darkgeek]

@michalbednarski OK.

u0_a105@localhost:~$ dpkg --print-architecture 
arm
u0_a105@localhost:~$ uname -a
Linux localhost 3.4.113-bacon-g487bcfba375 #1 SMP PREEMPT Wed Apr 11 09:51:18 IST 2018 armv7l Android

[darkgeek]

And I tried TermuxArch just now, it still complains the same error:

u0_a105@localhost:~/Shares/arch$ bash setupTermuxArch.sh 

 🕛 > 🕛 TermuxArch v1.6 will attempt to install Linux in /data/data/com.termux/files/home/arch.  Arch Linux in Termux PRoot will be available upon successful completion.  To run this BASH script again, use `!!`.  Ensure background data is not restricted.  Check the wireless connection if you do not see one o'clock 🕐 below.  Checking prerequisites…

 🕛 > 🕧 Prerequisites: OK  Downloading TermuxArch…

setupTermuxArch.sha 100%[===================>]     153  --.-KB/s    in 0s      
setupTermuxArch.tar 100%[===================>]  19.03K  26.8KB/s    in 0.7s    

 🕛 > 🕐 TermuxArch download: OK

 🕛 > 🕜 TermuxArch v1.6 integrity: OK

 🕛 > 🕝 Detected armv7l Android operating system.

 🕛 > 🕒 Activating termux-wake-lock: DONE  

 🕛 > 🕞 Contacting worldwide mirror http://mirror.archlinuxarm.org/: DONE  

 🕛 > 🕓 Downloading the checksum file and ArchLinuxARM-armv7-latest.tar.gz from the geographically local mirror http://tw.mirror.archlinuxarm.org/.  If contact with the local mirror is not successful, run bash setupTermuxArch.sh again.  Should the worldwide mirror not provide another geographically nearby server after a couple of attempts, use bash setupTermuxArch.sh manual after locating a local mirror from the Internet; See bash setupTermuxArch.sh help for additional options.  Download of ArchLinuxARM-armv7-latest.tar.gz pending Internet connection:

ArchLinuxARM-armv7- 100%[===================>]      67  --.-KB/s    in 0s      
ArchLinuxARM-armv7- 100%[===================>] 354.17M  2.24MB/s    in 4m 30s  

 🕛 > 🕠 Checking download integrity with Termux busybox md5sum.  This may take a little while:

 🕛 > 🕕 System image file download integrity: OK

 🕛 > 🕡 Uncompressing ArchLinuxARM-armv7-latest.tar.gz into /data/data/com.termux/files/home/arch.  The option to create Arch Linux system users will be available through addauser and addauserps.  Arch Linux user login from Termux with startarch is now implemented.  See Ability for Scripts to Launch Commands for Arch Linux in Termux PRoot on Device https://github.com/sdrausty/TermuxArch/issues/54 for more information about these brand new options.  Additional features of TermuxArch are also listed at https://github.com/sdrausty/TermuxArch/releases.

While waiting, you can use df, du -hs, htop, ps, top and watch in a new Termux session to watch the uncompressing while the session completes.  Use info query and man query to learn more about your Linux system in the palm of your hand.  See The Linux Documentation Project http://tldp.org to learn more about Linux and CLI commands.  Uncompressing ArchLinuxARM-armv7-latest.tar.gz will take a long time; Be patient:

 🕛 > 🕘 Cleaning up installation files: DONE  

 🕛 > 🕤 Arch Linux in Termux PRoot is installed.  Configuring and updating Arch Linux 📲

:: Synchronizing package databases...
error: failed retrieving file 'core.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update core (download library error)
error: failed retrieving file 'extra.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update extra (download library error)
error: failed retrieving file 'community.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update community (download library error)
error: failed retrieving file 'alarm.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update alarm (download library error)
error: failed retrieving file 'aur.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update aur (download library error)
error: failed to synchronize any databases
error: failed to init transaction (download library error)

==> Generating locales...
  en_US.UTF-8.../usr/bin/locale-gen: line 41: 12752 Bus error               localedef -i $input -c -f $charset -A /usr/share/locale/locale.alias $locale

 🕛 > 🕙 Releasing termux-wake-lock: DONE  

 🕛 > 🕥 Use startarch to launch Arch Linux in Termux PRoot.  Alternatively, run ~/arch/startarch in a BASH shell to start Arch Linux in Termux PRoot for future sessions also.

 🕛 > 🕦 startarch copied to /data/data/com.termux/files/home/bin.

 🕛 = 🕛 Information about "Starting Arch Linux from Termux?" at https://github.com/sdrausty/TermuxArch/issues/25.  Use tour to run a short tour, and get to know the new Arch Linux in Termux PRoot environment you just set up better.  If there was more than one error during the update procedure and you would like to refresh the installation, use setupTermuxArch.sh refresh.  This will update and recreate the configuration provided.  Use the TermuxArch command keys to install and generate Arch Linux keyring keys.

Arch Linux in Termux PRoot is installed in /data/data/com.termux/files/home/arch.  This project is in active development and contributions are welcome; See https://sdrausty.github.io/TermuxArch/CONTRIBUTORS for information.  The documentation repository for TermuxArch https://sdrausty.github.io/TermuxArch/docs/ is a Termux Arch submodule that is located at https://github.com/sdrausty/docsTermuxArch.  Pull requests and contributions through the issues pages are open to improve the ux and this Termux PRoot installation script.

Use ~/arch/startarch and startarch in a BASH shell to launch Arch Linux in Termux PRoot for future sessions.  See https://wiki.archlinux.org/index.php/IRC_channel for available Arch Linux IRC channels.


[1]+  Stopped                 bash setupTermuxArch.sh

[michalbednarski]

I couldn't reproduce locally (on emulator), so I have to ask for detailed log

script
PROOT_VERBOSE=9 ./start-ubuntu.sh

(Exit script with Ctrl-D, transcript of session is saved to file called typescript

[darkgeek]

@michalbednarski OK, this is the transcript:

typescript.txt

[michalbednarski]

I think I have this fixed:
proot_5.1.107-8_arm-793b09.deb.zip
(Install with dpkg -i proot_5.1.107-8_arm-793b09.deb)

Please test and if it works I'll push update to apt repo

[darkgeek]

@michalbednarski Thanks for your work! By using your new proot, I can start TermuxArch and termux-ubuntu successfully, say, I can see the shell prompt on both of them.

However, I could not install any packages by using pacman or apt. For TermuxArch, I see these errors:

[02:01 home ]$ pacman -Syu
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update core (download library error)
error: failed retrieving file 'extra.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update extra (download library error)
error: failed retrieving file 'community.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update community (download library error)
error: failed retrieving file 'alarm.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update alarm (download library error)
error: failed retrieving file 'aur.db' from tw.mirror.archlinuxarm.org : Send failure: Function not implemented
error: failed to update aur (download library error)
error: failed to synchronize any databases
error: failed to init transaction (download library error)

And I can confirm that mirror site is available:

[02:01 home ]$ ping tw.mirror.archlinuxarm.org 
PING tw.mirror.archlinuxarm.org (163.13.240.201) 56(84) bytes of data.
64 bytes from 163.13.240.201: icmp_seq=1 ttl=41 time=384 ms
64 bytes from 163.13.240.201: icmp_seq=2 ttl=41 time=396 ms
64 bytes from 163.13.240.201: icmp_seq=3 ttl=41 time=395 ms
^C
--- tw.mirror.archlinuxarm.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 384.045/391.999/396.114/5.672 ms

For termux-ubuntu, I see:

u0_a105@localhost:~/Shares/termux-ubuntu$ ./start-ubuntu.sh 
groups: cannot find name for group ID 3003
groups: cannot find name for group ID 9997
groups: cannot find name for group ID 20105
groups: cannot find name for group ID 50105
root@localhost:~# apt install fortune-mod
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  fortunes-min librecode0
Suggested packages:
  fortunes x11-utils bsdmainutils
The following NEW packages will be installed:
  fortune-mod fortunes-min librecode0
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/573 kB of archives.
After this operation, 1563 kB of additional disk space will be used.
W: chown to _apt:root of directory /var/cache/apt/archives/partial failed - SetupAPTPartialDirectory (38: Function not implemented)
W: chmod 0700 of directory /var/cache/apt/archives/partial failed - SetupAPTPartialDirectory (38: Function not implemented)
Do you want to continue? [Y/n] y
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
debconf: delaying package configuration, since apt-utils is not installed
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)
dpkg: error processing archive /var/cache/apt/archives/librecode0_3.6-23_armhf.deb (--unpack):
 unable to securely remove '/var/lib/dpkg/tmp.ci': Function not implemented
dpkg: error while cleaning up:
 unable to securely remove '/var/lib/dpkg/reassemble.deb': Function not implemented
dpkg: error processing archive /var/cache/apt/archives/fortune-mod_1%3a1.99.1-7_armhf.deb (--unpack):
 unable to securely remove '/var/lib/dpkg/tmp.ci': Function not implemented
dpkg: error while cleaning up:
 unable to securely remove '/var/lib/dpkg/reassemble.deb': Function not implemented
dpkg: error processing archive /var/cache/apt/archives/fortunes-min_1%3a1.99.1-7_all.deb (--unpack):
 unable to securely remove '/var/lib/dpkg/tmp.ci': Function not implemented
dpkg: error while cleaning up:
 unable to securely remove '/var/lib/dpkg/reassemble.deb': Function not implemented
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Function not implemented
W: chown to root:adm of file /var/log/apt/term.log failed - OpenLog (38: Function not implemented)
W: chmod 0640 of file /var/log/apt/term.log failed - OpenLog (38: Function not implemented)
E: Sub-process /usr/bin/dpkg returned an error code (2)

I provide the transcript of both of them here:

typescript-archlinux.txt

typescript-ubuntu.txt

Thanks:)

[michalbednarski]

I've added workarounds for more legacy syscalls that are now blocked (see commit message above for details). pacman seems to work now, further workarounds for apt will be needed but for today these are ones I've implemented.

Binary: proot_5.1.107-8_arm-0c1ab5.deb.zip

@corbinlc Above commit includes change based on your symlink -> symlinkat patch, one of things that came out during testing is that rewinding by SYSTRAP_SIZE when thumb mode was active (e.g. in dpkg (apt)) actually reexecuted two instructions, and additional instruction clobbered syscall argument, proot seen invalid argument and failed syscall by providing own errno and changing syscall to PR_void.

[darkgeek]

Thanks for your work. I have tested your provided proot, now I can confirm pacman works again.

[corbinlc]

This is how I handled utimes...

    //int utimes(const char *filename, const struct timeval times[2]);
    case PR_utimes:
            //convert to:
            //int utimensat(int dirfd, const char *pathname, const struct timespec times[2], int flags);
            prepare_restart_syscall_after_seccomp(tracee);
            set_sysnum(tracee, PR_utimensat);
            if (peek_reg(tracee, CURRENT, SYSARG_2) != 0) {
                    signal = read_data(tracee, times, peek_reg(tracee, CURRENT, SYSARG_2), sizeof(times));
                    if (signal < 0) {
                            poke_reg(tracee, SYSARG_RESULT, signal);
                            push_specific_regs(tracee, false);

                            /* Swallow signal */
                            signal = 0;
                            break;
                    }
                    timens[0].tv_sec = (time_t)times[0].tv_sec;
                    timens[0].tv_nsec = (long)times[0].tv_usec * 1000;
                    timens[1].tv_sec = (time_t)times[1].tv_sec;
                    timens[1].tv_nsec = (long)times[1].tv_usec * 1000;
                    signal = write_data(tracee, peek_reg(tracee, CURRENT,  SYSARG_2), timens, sizeof(timens));
                    if (signal < 0) {
                            poke_reg(tracee, SYSARG_RESULT, signal);
                            push_specific_regs(tracee, false);

                            /* Swallow signal */
                            signal = 0;
                            break;
                    }
            }
            poke_reg(tracee, SYSARG_4, 0);
            poke_reg(tracee, SYSARG_3, peek_reg(tracee, CURRENT, SYSARG_2));
            poke_reg(tracee, SYSARG_2, peek_reg(tracee, CURRENT, SYSARG_1));
            poke_reg(tracee, SYSARG_1, AT_FDCWD);
            restart_syscall_after_seccomp(tracee);

            /* Swallow signal */
            signal = 0;
            break;

That seems to work well. I should probably check that the two struct types are the same size (really that time_t is the same size as a long), but that seems to work fine for the cases tried on armhf

[corbinlc]

Also, this is what I did for statfs

    //int statfs(const char *path, struct statfs *buf);
    case PR_statfs:
            prepare_restart_syscall_after_seccomp(tracee);
            set_sysnum(tracee, PR_statfs64);
            restart_syscall_after_seccomp(tracee);

Again, there could be a problem if the statfs64 struct and the statfs struct are different sizes. This is not tested as well.

[corbinlc]

@michalbednarski With regards to the statfs, this is being called on /dev/pts . statfs or statfs64 fails on this, because it appears to not be readable on a pixel running 8.1 . This used to be readable when I was developing before. Is this a known issue, is there a known workaround? I notice that /dev/ptmx is readable. Do you guys use some sort of binding to work around this issue?

[corbinlc]

@michalbednarski I like this restarting method better than what I was doing to fully emulate the system calls in an extension. I tried to copy your method for symlink originally, but wasn't smart enough or diligent enough to figure out it was rewinding by an incorrect amount. Thanks for figuring out the issue when thumb mode was active.

[corbinlc]

So, if I add support for utimes and statfs as shown, but make statfs just return 0 when /dev/pts is checked, I can apt-get update and then apt-get install wget and then run wget so that is pretty close to a general use case given that apt-get is one of the hardest things to get working

[marksteward]

Does this mean https://github.com/termux/proot/blob/b41e439/src/syscall/chain.c#L149 also needs fixing? I don't have a thumb device to test.

[michalbednarski]

Just pushed 29a1f08 which adds utimes support so apt can work, tomorrow I'll send pull request to termux-packages so all those fixes will be included in Termux apt repo

@corbinlc Thanks for utimes handling, code in my commit is based on your implementation

@marksteward Thanks for noting syscall chain rewind, actually I was thinking about it as well previously but in the end I've forgot about it during previous commit.

[darkgeek]

Thanks for your awesome work. I think it's fixed now.

[michalbednarski]

I was going to close this today as update is now live in apt repo.

[infinity0]

I'm running into this issue with LineageOS 16 (Android 9) and Debian Testing, same error about "error creating new backup file"

[infinity0]

Workaround is to run proot as root.