make termux self-hosting

[shawnl]

it would be awesome is termux could build its own apt.

similar: #63

If termux was self-hosting I would be able to do some packaging. But the current build system is too hacked up, I can't get it working.

[hyperpallium]

I've got termux to install from within termux (i.e. without Gradle), but crashes with [abbv]:

Caused by: android.view.InflateException: Binary XML file line #7: Error inflating class android.support.v4.widget.DrawerLayout
  at android.view.LayoutInflater.createView(LayoutInflater.java:652)
[snip]
caused by: java.lang.NoClassDefFoundError: android.support.v4.widget.DrawerLayout$ChildAccessibilityDelegate
  at android.support.v4.widget.DrawerLayout.<init>(DrawerLayout.java:191)

BTW: it's a NoClassDefFound and an Error, not a ClassNotFoundException..

I've got my own (simple) apps to install and run, as proper android apps (on the homescreen etc). So I partly know what I'm doing.

For building termux, everything compiles (including native), all junit tests successful, packaged (including annotation and core-ui support libraries), installed.
Oh, and changed the manifest package name (with aapt's --rename-manifest-package), so it doesn't replace the termux I'm working from...

I think packaging resources with aapt p is the problem (and is causing DrawerLayout's components to lack necessary info to inflate properly). Unfortunately, there's little documentation for aapt... and it's rarely directly used anymore, so google/Stackoverflow isn't much help.

Anyone know how to package termux's subprojects and support libraries with aapt (i.e. without Gradle)?

[ I might try modifying termux and app's res/layout/drawer_layout.xml to omit DrawerLayout, and just setContentPane with a TerminalView directly. But I'm not familiar with layout resources, so I might be missing a lot.]

PS: I'm on Android 5.1 Lollipop.
@SDRausty @joakim-noah

[joakim-noah]

I'm a little unclear, you want to build the Termux apk in Termux? I thought this issue was about building Termux deb packages in Termux.

Take a look at the instructions I wrote up on building simple apks using aapt on the D wiki, it might help.

[hyperpallium]

Thanks for the quick reply!
Yes... Ah, I read build its own apt as build its own apk. A different kind of self-hosting... (Other comments mention building Termux in Termux, confirming my misreading.)

Should I start a new issue for this?

Thanks for the link - the packaging there is similar to your Feb 15, 2016 comment, which was one of the references I used to get native code working. Unfortunately, it doesn't cover packaging resources from multiple sources, which I think is the problem here - termux has three subprojects and uses two support libraries.

EDIT Apparently, Gradle will output the aapt commands it issues if run with the --debug option. Although the output is "extremely verbose", it would be great if someone could build termux on a PC and post the output here (I don't have a PC myself). One option, to cover future changes, is if https://travis-ci.org/termux/termux-app ran gradle with --debug.

[joakim-noah]

I haven't used a PC in three years either, only been using Android and Termux on a tablet and smartphone since my ultrabook died. You could submit a pull to dump the debug output to the termux-app github, then look at the debug log in the CI log.

[hyperpallium]

@joakim-noah Good solution. Though I saw on the logs it uses aapt2, so might differ. EDIT though it would give some guidance, so I should give it a go. EDIT2 the big difference seems that aapt2 has separate aapt2 compile and aapt2 link phases

The same problem happens with the AccessibilityDelegate of PagerView, from the same core-ui support aar. The resources files don't seem relevant to these particular classes, so might be something else after all, a version problem maybe?
For the moment, I've just commented-out these classes, to get the main thing working.

I've got it running, and installing the bootstrap-arm.zip - but this is output on the terminal itself:

exec("/data/data/com.termux2/files/usr/bin/login"): Permission denied

(copied by hand, but I think it's right). This string is assembled from within termux.c, output with perror.

I comfirmed that all the executables get Os.chmod.

Could it be a file-permission problem, because it's com.termux2, and com.termux is hardcoded somewhere? I added the 2 to TerminalService.FILE_PATH, and it doesn't seem to appear anywhere else.

I'm also just using plain gcc -shared for termux.c, and putting it in /lib/armeabi/libterm.so - supposedly deprecated. However, I can't see how that would deny permission, and the c code is running... but who knows. I should follow your guide, and try a proper -v7a version.

[joakim-noah]

Never dug much deeper than the sample apks, so don't know about more complex resources, and never built Termux either.

[hyperpallium]

No worries, thanks for your responses.

I got it working by using bash instead of login for the shell.

It turns out the base directory /data/data/com.termux/files is hard-coded in many programs, which the new com.termux2 can't access, only its own /data/data/com.termux2/files. e.g. the following programs need:

bash: /data/data/com.termux/files/usr/etc/bash.bash.rc
login: /data/data/com.termux/files/usr/bin/sh
apt: /data/data/com.termux/files/usr/etc/apt/apt.conf.d

Fortunately, bash still runs. The busybox utilities also work (e.g. its basic vi).

Need to patch the sources and build, though I'm not sure how to make them truly directory independent (I don't want to make a hardcoded termux2! just to build termux in termux) I think most linuxes depend on an apriori known root directory structure...

Anyway... building apt within termux will be one of them, so this issue was apt after all.

[hyperpallium]

To clarify: I got it to start with bash instead of login by putting "bash" first in app/src/main/java/com/termux/app/TermuxService at line 265 (NB: line will differ in unmodified source)

  for (String shellBinary : new String[]{"login", "bash", "zsh"}) {

Some features of the terminal weren't working - backspace, arrows - and vim was printing out strings at the bottom instead of modifying text in-place.

By using a new name of the same length, com.termuz, binary patching works. From the prompt in the new app:

grep -r -l "com\.termux" ../usr/lib | xargs sed -i 's/com\.termux/com.termuz/g'
grep -r -l "com\.termux" ../usr/bin/bash | xargs sed -i 's/com\.termux/com.termuz/g'

I tried patching everything in ../usr, but this caused a fail to start. (I haven't investigated further, but I suspect that using the complete path /data/data/com.termux might fix it, because some com.termux
strings are getting modified that shouldn't be.)

I got vim working by copying ../usr/share/vim from the old termux (NB: actually, tarring it, cp to sdcard, then cp to new termux), and copying the binary ../usr/bin/vim and patching it as above.

Of course, it's very helpful to also copy across your .inputrc, .vimrc and .bashrc.

Curiously, .bashrc is not run when the terminal starts (even though it is bash); but if you then execute bash, it is run. I guess there's some other com.termux hardcoded somewhere.

[finagolfin]

Any interest in adding this, maybe by installing a Termux environment in a VPS so that cloud builds of Termux packages can continue? I ask because I just got much of Swift built and working in Termux, swiftlang/swift@796d6ad, and to get a Termux package for Swift built next, I now have to figure out how to get it all cross-compiled too.

It would be easiest if these Termux build scripts simply supported building in Termux instead. I can understand why this wasn't done from the beginning, but now that Termux is being used more and more for development, maybe it's time to add that option too.

[ghost]

I now have to figure out how to get it all cross-compiled too.

Cross-compiling now available for everyone. Just setup a CI for your repository, example configuration available in https://github.com/termux/termux-packages/blob/master/.cirrus.yml.

It would be easiest if these Termux build scripts simply supported building in Termux instead.

There several problems:

1. Build environment uses clang from NDK and Termux uses vanilla clang. Program successfully compiled with NDK toolchain may not be compiled inside Termux.

2. Build environment is based on Ubuntu which provides several packages that are not available in Termux but needed for building packages. This also provides standard file system layout (FHS) and ensures that all build scripts are work as expected - in Termux you may need to do either additional patching or run build steps under proot.

3. Certain things require Android SDK.

4. Required access to /data. Not only /data/data/com.termux/files/usr. But I guess that can be changed.

Above problems are derived from how build scripts are written. But there actually more which are package-specific (though maybe related to either case 1 or 2).

now that Termux is being used more and more for development, maybe it's time to add that option too.

Can you tell for which purpose self-hosting is needed ?

Previous versions of packages can be built on CI (no PC needed, only Termux) but will require some initial configuration (easy + config sample at https://github.com/termux/termux-packages/blob/master/.cirrus.yml).

[ghost]

I got it build-package.sh partially working in Termux:

$ bash ./build-package.sh -s ed
termux - building ed for arch aarch64...
Downloading https://mirrors.kernel.org/gnu/ed/ed-1.15.tar.lz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 67343  100 67343    0     0  33789      0  0:00:01  0:00:01 --:--:-- 33772
/data/data/com.termux/files/home/.termux-build/ed/src/configure[24]: CC?=gcc: not found
/data/data/com.termux/files/home/.termux-build/ed/src/configure[25]: CPPFLAGS?=: not found
/data/data/com.termux/files/home/.termux-build/ed/src/configure[26]: CFLAGS?=-Wall -W -O2: not found
/data/data/com.termux/files/home/.termux-build/ed/src/configure[27]: LDFLAGS?=: not found
configure: WARNING: unrecognized option: '--disable-dependency-tracking'
configure: WARNING: unrecognized option: '--libdir=/data/data/com.termux/files/usr/lib'
configure: WARNING: unrecognized option: '--disable-rpath'
configure: WARNING: unrecognized option: '--disable-rpath-hack'
configure: WARNING: unrecognized option: '--host=aarch64-linux-android'
configure: WARNING: unrecognized option: '--disable-nls'
configure: WARNING: unrecognized option: '--enable-shared'
configure: WARNING: unrecognized option: '--enable-static'
configure: WARNING: unrecognized option: '--libexecdir=/data/data/com.termux/files/usr/libexec'

creating config.status
creating Makefile
VPATH = /data/data/com.termux/files/home/.termux-build/ed/src
prefix = /data/data/com.termux/files/usr
exec_prefix = $(prefix)
bindir = $(exec_prefix)/bin
datarootdir = $(prefix)/share
infodir = $(datarootdir)/info
mandir = $(datarootdir)/man
program_prefix = 
CC = aarch64-linux-android-clang
CPPFLAGS = -I/data/data/com.termux/files/usr/include
CFLAGS =  -fstack-protector-strong -Oz
LDFLAGS = -L/data/data/com.termux/files/usr/lib -Wl,-rpath=/data/data/com.termux/files/usr/lib,--enable-new-dtags -Wl,--as-needed -Wl,-z,relro,-z,now
OK. Now you can run make.
aarch64-linux-android-clang -I/data/data/com.termux/files/usr/include -fstack-protector-strong -Oz -c -o buffer.o /data/data/com.termux/files/home/.termux-build/ed/src/buffer.c
aarch64-linux-android-clang -I/data/data/com.termux/files/usr/include -fstack-protector-strong -Oz -c -o carg_parser.o /data/data/com.termux/files/home/.termux-build/ed/src/carg_parser.c
aarch64-linux-android-clang -I/data/data/com.termux/files/usr/include -fstack-protector-strong -Oz -c -o global.o /data/data/com.termux/files/home/.termux-build/ed/src/global.c
aarch64-linux-android-clang -I/data/data/com.termux/files/usr/include -fstack-protector-strong -Oz -c -o io.o /data/data/com.termux/files/home/.termux-build/ed/src/io.c
aarch64-linux-android-clang -I/data/data/com.termux/files/usr/include -fstack-protector-strong -Oz -DPROGVERSION=\"1.15\" -c -o main.o /data/data/com.termux/files/home/.termux-build/ed/src/main.c
aarch64-linux-android-clang -I/data/data/com.termux/files/usr/include -fstack-protector-strong -Oz -c -o main_loop.o /data/data/com.termux/files/home/.termux-build/ed/src/main_loop.c
aarch64-linux-android-clang -I/data/data/com.termux/files/usr/include -fstack-protector-strong -Oz -c -o regex.o /data/data/com.termux/files/home/.termux-build/ed/src/regex.c
aarch64-linux-android-clang -I/data/data/com.termux/files/usr/include -fstack-protector-strong -Oz -c -o signal.o /data/data/com.termux/files/home/.termux-build/ed/src/signal.c
cat /data/data/com.termux/files/home/.termux-build/ed/src/red.in > red
chmod a+x red
aarch64-linux-android-clang -L/data/data/com.termux/files/usr/lib -Wl,-rpath=/data/data/com.termux/files/usr/lib,--enable-new-dtags -Wl,--as-needed -Wl,-z,relro,-z,now -fstack-protector-strong -Oz -o ed buffer.o carg_parser.o global.o io.o main.o main_loop.o regex.o signal.o
if [ ! -d "/data/data/com.termux/files/usr/bin" ] ; then install -d -m 755 "/data/data/com.termux/files/usr/bin" ; fi
install -m 755 ./ed "/data/data/com.termux/files/usr/bin/ed"
install -m 755 ./red "/data/data/com.termux/files/usr/bin/red"
if [ ! -d "/data/data/com.termux/files/usr/share/info" ] ; then install -d -m 755 "/data/data/com.termux/files/usr/share/info" ; fi
rm -f "/data/data/com.termux/files/usr/share/info/ed.info"*
install -m 644 /data/data/com.termux/files/home/.termux-build/ed/src/doc/ed.info "/data/data/com.termux/files/usr/share/info/ed.info"
if /bin/sh -c "install-info --version" > /dev/null 2>&1 ; then \
	install-info --info-dir="/data/data/com.termux/files/usr/share/info" "/data/data/com.termux/files/usr/share/info/ed.info" ; \
fi
if [ ! -d "/data/data/com.termux/files/usr/share/man/man1" ] ; then install -d -m 755 "/data/data/com.termux/files/usr/share/man/man1" ; fi
rm -f "/data/data/com.termux/files/usr/share/man/man1/ed.1"*
rm -f "/data/data/com.termux/files/usr/share/man/man1/red.1"*
install -m 644 /data/data/com.termux/files/home/.termux-build/ed/src/doc/ed.1 "/data/data/com.termux/files/usr/share/man/man1/ed.1"
cd "/data/data/com.termux/files/usr/share/man/man1" && ln -s "ed.1" "red.1"

Later will post which issues are happens there.
Probably will even submit necessary Termux compatibility changes for build-package.sh.

---

Actual issues are:

1. Not possible to **cross-**compile packages for other architectures since everything relying on $PREFIX and it obviously can't be switched from ARM to x86 for example.

2. Termux environment will be messed up since everything is installed to $PREFIX.
   Experimenting with core packages may even break environment and you will need to re-install Termux. Some build.sh scripts contain commands that are dangerous to be executed in Termux, example is a perl package https://github.com/termux/termux-packages/blob/master/packages/perl/build.sh#L43.

3. Step termux_step_massage() is very slow on Android devices (only parts where find is getting executed). Same goes to configure part with autotools.

4. Issues described in make termux self-hosting #1756 (comment).
   Packages that are not available in Termux but required to build some packages:

   asciidoc
   asciidoctor
   help2man
   intltool
   meson
   openjdk-8-jdk
   xmlto
   
   # These should be available via `pip`.
   python3-docutils
   python3-sphinx
   python3-recommonmark
   

[ghost]

#4149

[finagolfin]

Cross-compiling now available for everyone. Just setup a CI for your repository

Thanks, but that's not really the problem, as I have a linux x86_64 VPS and can cross-compile other packages from this repo just fine. I'm simply saying that writing a Termux build.sh to cross-compile Swift will be more work than one that natively compiles it.

I doubt I'm alone, as I see others who've built projects natively in Termux, #1438 say(looks like that guy cross-compiles, I didn't actually look at his build script till now, assumed before that he natively compiled from what he wrote there), then never bothered to get it cross-compiled with the Termux scripts.

Anyway, I can get it cross-compiled, it's not insurmountable, simply more work. I thought I'd try and spur a discussion about having a native build option in this repo, leaving aside the Swift package and speaking generally.

Build environment uses clang from NDK and Termux uses vanilla clang. Program successfully compiled with NDK toolchain may not be compiled inside Termux.

I doubt this is going to cause problems, as it's unlikely that they patched clang and other build tools much for the NDK. It is possible though.

Build environment is based on Ubuntu which provides several packages that are not available in Termux but needed for building packages.
Certain things require Android SDK.

Sure, native builds won't work for all packages: I figured there could be a variable set for those, so that they're only cross-compiled from Ubuntu.

This also provides standard file system layout (FHS) and ensures that all build scripts are work as expected - in Termux you may need to do either additional patching or run build steps under proot.

I don't see why, I only see patches that add the Termux sysroot, not the Ubuntu layout. That makes sense, as the packages ultimately run in Termux. If you mean the main build scripts in scripts/build and not the individual package scripts in packages/, yes, the idea is to patch the former to work in Termux too, which provides an even more standard layout than linux distros. 😉

Can you tell for which purpose self-hosting is needed ?

As I said above, many of the Swift repos have been cross-compiled by others before, swiftlang/swift#26298, so I should be able to get them cross-compiled from linux. But I doubt the Swift package manager has been, so I think that will take more work.

And just generally, after getting a package like this built natively, packagers have to go figure out how to cross-compile it. While I can do it for Swift, it would be nice if Termux started supporting native package builds also.

Probably will even submit necessary Termux compatibility changes for build-package.sh

Thanks, great to see you trying it out. I thought there was some repo that already ported the build scripts to Termux some time back, but I can't find it now.

Not possible to **cross-**compile packages for other architectures since everything relying on $PREFIX and it obviously can't be switched from ARM to x86 for example.

Ah, yes, that is a problem, didn't think of that, was hoping cross-compiling from Termux would work too.

Termux environment will be messed up since everything is installed to $PREFIX.
Experimenting with core packages may even break environment and you will need to re-install Termux. Some build.sh scripts contain commands that are dangerous to be executed in Termux, example is a perl package

Yes, some subset of packages will still require cross-compiling from Ubuntu for reasons like this, so they can just have a variable like TERMUX_LINUX_BUILD=yes set so that they're never built on Termux.

Thanks for trying it out, I think this repo should definitely have a native option. While I think that's doable over time, I figured the bigger issue is where those native packages would actually be built: how do you imagine that working, say if I submit a Termux package that only builds natively?

Do you think it would be that hard to set up a Termux environment in Cirrus or whatever is currently used to build official packages? I think figuring out that part, how to continue cloud builds of native Termux packages, is what needs some thought. I've built code in an Android emulator or virtual environment like that before, but I don't know how hard it would be for you guys to set one up for this project, so that Termux-native packages could be automatically built in the cloud.

[ghost]

I doubt this is going to cause problems, as it's unlikely that they patched clang and other build tools much for the NDK.

As I remember, some packages are building fine with NDK but in Termux throw compiler errors. With same set of patches.

Maybe something was changed with recent versions, but I need verify this.

so they can just have a variable like TERMUX_LINUX_BUILD=yes set so that they're never built on Termux.

I already started to fix the Termux build system for self-hosting and it now exports variable TERMUX_ON_DEVICE_BUILD so these packages can be fixed.

I figured the bigger issue is where those native packages would actually be built: how do you imagine that working, say if I submit a Termux package that only builds natively?

I guess that will be rare case. Mostly happens when program's authors haven't implemented support for cross-compiling.

Headers in ndk-sysroot are same as in build environment so they should not be a problem.

Do you think it would be that hard to set up a Termux environment in Cirrus or whatever is currently used to build official packages?

There no problem with launching Termux in Android virtual device inside Cirrus. But there are problems related to:

• Communicating with Termux app.

• Speed (even if x86).

[ghost]

Closing since on-device build support was implemented. Not all packages can be built on device, but hope that will be changed in future.