Trim Busybox, supply other essential commands

[frno7]

Busybox is currently 2.6 MB and comes with a lot of unnecessary functions. Remove at least half of them. See also frno7/iopmod#7 (comment).

Currently defined functions:
	[, [[, acpid, addgroup, adduser, adjtimex, ar, arch, arp, arping,
	ascii, ash, awk, base32, base64, basename, bb, bbconfig, bbsh, bc,
	blkdiscard, blkid, blockdev, brctl, bunzip2, bzcat, bzip2, cal, cat,
	chat, chattr, chgrp, chmod, chown, chpasswd, chpst, chroot, chrt, chvt,
	cksum, clear, cmp, comm, conspy, cp, cpio, crc32, crond, cryptpw,
	cttyhack, cut, date, dd, deallocvt, delgroup, deluser, depmod, devmem,
	df, dhcprelay, diff, dirname, dmesg, dnsdomainname, dos2unix, du,
	dumpkmap, dumpleases, echo, ed, egrep, eject, env, envdir, envuidgid,
	ether-wake, expand, expr, factor, fallocate, false, fatattr, fbset,
	fdflush, fdformat, fdisk, fgconsole, fgrep, find, findfs,
	flash_eraseall, flash_lock, flash_unlock, flashcp, flock, free,
	freeramdisk, fsck, fsfreeze, fstrim, fsync, ftpd, fuser, getopt, getty,
	ginit, grep, groups, gunzip, gzip, halt, hd, hdparm, head, hexdump,
	hexedit, hostname, httpd, hwclock, i2cdetect, i2cdump, i2cget, i2cset,
	i2ctransfer, id, ifconfig, ifdown, ifenslave, ifplugd, ifup, init,
	insmod, install, ionice, iostat, ip, ipaddr, ipcrm, ipcs, iplink,
	ipneigh, iproute, iprule, iptunnel, kbd_mode, kill, killall, killall5,
	last, less, link, linux32, linux64, linuxrc, ln, loadfont, loadkmap,
	login, logread, losetup, lpq, lpr, ls, lsattr, lsmod, lsof, lspci,
	lsscsi, lsusb, lzcat, lzma, lzop, lzopcat, makedevs, man, md5sum, mdev,
	mesg, microcom, mim, minips, mkdir, mkdosfs, mke2fs, mkfifo, mkfs.ext2,
	mkfs.reiser, mkfs.vfat, mknod, mkpasswd, mkswap, mktemp, modinfo,
	modprobe, more, mount, mountpoint, mpstat, mt, mv, nameif, nanddump,
	nandwrite, nbd-client, nc, netcat, netstat, nice, nl, nmeter, nohup,
	nologin, nproc, nsenter, nslookup, ntpd, nuke, openvt, partprobe,
	passwd, paste, patch, pgrep, pidof, ping, pipe_progress, pivot_root,
	pkill, pmap, popmaildir, poweroff, powertop, printenv, printf, ps,
	pscan, pstree, pwd, pwdx, raidautorun, rdate, readahead, readlink,
	realpath, reboot, renice, reset, resize, resume, rev, rm, rmdir, rmmod,
	route, rtcwake, run-init, runlevel, rx, script, scriptreplay, sed,
	sendmail, seq, setarch, setconsole, setfattr, setfont, setkeycodes,
	setlogcons, setpriv, setserial, setsid, setuidgid, sh, sha1sum,
	sha256sum, sha3sum, sha512sum, showkey, shred, shuf, sleep, softlimit,
	sort, split, ssl_client, start-stop-daemon, stat, strings, stty, su,
	sum, svc, svok, swapoff, swapon, switch_root, sync, sysctl, tac, tail,
	tar, tc, tee, telnet, telnetd, test, tftp, tftpd, time, timeout, top,
	touch, tr, traceroute, true, truncate, ts, tty, ttysize, tunctl,
	tune2fs, ubiattach, ubidetach, ubimkvol, ubirename, ubirmvol, ubirsvol,
	ubiupdatevol, udhcpc, udhcpd, uevent, umount, uname, uncompress,
	unexpand, uniq, unit, unix2dos, unlink, unlzma, unlzop, unshare, unxz,
	unzip, uptime, users, usleep, vconfig, vi, vlock, volname, w, wall,
	watch, watchdog, wc, wget, which, who, whoami, whois, xargs, xxd, xz,
	xzcat, yes, zcat, zcip

[AKuHAK]

Maybe dropbear also can be provided as a must-have inside initramfs?

[frno7]

Indeed, dropbear is essential. People with wifi devices will want wpa_supplicant, and possibly iw. For advanced video mode configurations, fbset is needed. To drop out of the Linux kernel and drop into something completely different, back to wLaunchELF, for instance, one could use kexec. We should supply these commands too. :-)

[frno7]

I will also look into replacing

https://github.com/frno7/mipsr5900el-gentoo-linux-gnu/blob/17bb9c572c19e9675dcbed3c3f7f2940a03a837c/Dockerfile#L36-L52

with the Gentoo genkernel command.

[frno7]

@AKuHAK, the dropbear command came in commit 2ccb979, and fbset in 5072d48. They are the Musl variants which are much smaller, for busybox too. I don’t think there’s any reason to compile the GNU libc variants of them anymore, so those should probably be removed from the Dockerfile to make it a bit simpler and faster for the action runner.

All tools are statically linked, which means some duplication of Musl being separately linked into all of them. The advantage however is that they have no library dependencies, so commands can easily be added to or removed from the initramfs, as needed. Further on, it’d make sense to supply a set of different kinds of initramfs, tailored for different needs.

To be useful, Dropbear needs additional configuration files, for example /etc/dropbear/dropbear_ecdsa_host_key and /root/.ssh/authorized_keys, as well as being started from /sbin/init. For safety purposes I don’t think we should preconfigure network access for the default initramfs. It would make a nice article for the wiki however.

The lib/modules directory, by the way, is 4.5 MB. It could be made much smaller, especially if tuned for a specific use-case or set of devices.

[AKuHAK]

Great, probably we need to configure dispatch action, so after changes in this repo, produced binary in linux repo will also automatically change.

[frno7]

I removed redundant GNU libc linked packages in commit d2ab811. I also made proper scripts for the commands used, in commit 3017bca, so the exact same installation can be done in a plain Gentoo, with no Docker whatever, avoiding some of its obnoxious RUN syntax. With modular scripts, it’s also easier to rerun a subset of the installation procedure.

[frno7]

Great, probably we need to configure dispatch action, so after changes in this repo, produced binary in linux repo will also automatically change.

Since this repo compiles the kernel, we’ve got a cyclic dependency between the two repos. When updating the linux repo, it will rebuild itself. But then this repo has an old kernel, so it will have to be rebuilt too, which will trigger another rebuild of the linux repo... That’s a good reason to remove the kernel and IOP modules from this repo, and have it focus on Gentoo and tooling.

[AKuHAK]

we’ve got a cyclic dependency between the two repos.

Its not actually true cyclic, cause Docker image can contain old kernel, while linux repo should be up to date always. I only ask for linux artifacts regeneration, when Docker image updates, not vice versa. Buy yes, if this is confusing, you can remove linux kernel sources from Docker image and leave initramfs without kernel modules. This will also reduce Docker size heavily.

[frno7]

Busybox has got fbset, apparently:

gentoo-mipsr5900el/configs/busybox

Lines 615 to 617 in 87d58f1

	CONFIG_FBSET=y
	CONFIG_FEATURE_FBSET_FANCY=y
	CONFIG_FEATURE_FBSET_READMODE=y

I haven’t verified it has feature parity with sys-apps/fbset in

gentoo-mipsr5900el/scripts/install-packages

Lines 12 to 13 in 87d58f1

	# sys-apps/fbset
	ACCEPT_KEYWORDS="*" USE="static" emerge-mipsr5900el-unknown-linux-musl -v sys-apps/fbset

but if it does, we might as well remove this package, and save another 102424 bytes. :-)

[frno7]

Removed kernel in commit 9591cf1, and IOP modules and tools in commit 5e10f35. Added savedconfig for Busybox in commit 87d58f1. Most Busybox size savings were made when switching from GNU libc to Musl, as it shrank from 2.58 MB down to 1.66 MB. Configuration changes got it further down to 1.47 MB, but it could have been smaller had I not added IPv6 and some other useful features. As Busybox stands:

Currently defined functions:
	addgroup, adduser, adjtimex, arp, arping, ash, awk, basename, bb, bbsh,
	bc, blkdiscard, blkid, blockdev, brctl, cat, chattr, chgrp, chmod,
	chown, chpasswd, chpst, chroot, chrt, chvt, cksum, clear, cmp, comm,
	conspy, cp, crc32, crond, cryptpw, cttyhack, cut, date, dc, dd,
	deallocvt, delgroup, deluser, depmod, devmem, df, dhcprelay, diff,
	dirname, dmesg, du, dumpkmap, dumpleases, echo, ed, eject, env, envdir,
	envuidgid, ether-wake, expand, expr, fallocate, false, fatattr, fbset,
	fdflush, fdisk, fgconsole, find, findfs, flock, free, freeramdisk,
	fsck, fsfreeze, fstrim, fsync, fuser, getopt, getty, ginit, grep,
	groups, gzip, halt, hdparm, head, hexdump, hexedit, hostname, hwclock,
	id, ifplugd, init, insmod, install, ionice, iostat, ip, kbd_mode, kill,
	killall, killall5, klogd, less, link, ln, loadfont, loadkmap, logger,
	login, logread, ls, lsattr, lsmod, lsof, lsscsi, lsusb, makedevs,
	md5sum, mdev, mesg, microcom, mkdir, mkdosfs, mke2fs, mkfifo,
	mkfs.vfat, mknod, mkpasswd, mkswap, mktemp, modinfo, modprobe, more,
	mount, mountpoint, mpstat, mv, nameif, netcat, netstat, nice, nl,
	nmeter, nohup, nologin, nsenter, nslookup, ntpd, od, openvt, partprobe,
	passwd, paste, patch, pgrep, pidof, ping, pipe_progress, pivot_root,
	pkill, pmap, poweroff, printenv, printf, ps, pstree, pwd, pwdx, rdate,
	readahead, readlink, realpath, reboot, renice, reset, resize, resume,
	rev, rm, rmdir, rmmod, rtcwake, run-init, sed, seq, setconsole,
	setfattr, setfont, setkeycodes, setlogcons, setserial, setsid,
	setuidgid, sh, sha1sum, sha256sum, sha3sum, sha512sum, showkey, shred,
	shuf, sleep, softlimit, sort, split, ssl_client, start-stop-daemon,
	stat, strings, stty, su, sum, svc, svok, swapoff, swapon, switch_root,
	sync, sysctl, syslogd, tac, tail, tar, tee, telnet, test, tftp, tftpd,
	time, timeout, top, touch, tr, traceroute, traceroute6, true, truncate,
	ts, tty, ttysize, tunctl, tune2fs, udhcpc, udhcpc6, udhcpd, umount,
	uname, unexpand, uniq, unlink, unshare, unzip, uptime, usleep, vi,
	volname, watch, watchdog, wc, which, whoami, xargs, xxd, xz, zcip

[AKuHAK]

and IOP modules and tools

Why? IOP modules are essential for initramfs, this means that current artifacts in linux repo are broken. IOP tools also are nice addition. Also iop tools are taking less than 1mb.

[frno7]

Why? IOP modules are essential for initramfs, this means that current artifacts in linux repo are broken.

They’d be out-of-sync with the iopmod repo (having the same cyclic dependency to this repo as the linux repo had). The kernel modules are also essential for initramfs, but they’re built by the kernel of course. The IOP modules, as firmware, are closer to the kernel than Busybox and other Gentoo packages, in my opinion. So I think it makes sense to put the Gentoo packages into initramfs in this repo, and the rest (firmware and kernel modules) in the linux repo. Alternatively, the whole initramfs could be assembled in the linux repo.

Besides, the IOP modules and tools absolutely must be in-sync when the kernel compiles, so they’re compiled in .github/workflows/compilation.yml too.

Also, I’m considering removing the kernel modules from the downloadable initramfs archives, because they’re kernel version specific and generally won’t work with any other kernel. Hence, anyone who uses such initramfs archives with their own kernels will run into trouble with those modules, and should discard them anyway.

IOP tools also are nice addition. Also iop tools are taking less than 1mb.

IOP tooling could be an exception, but then again, they’re downloadable from the iopmod repo?

Anyone who wants either IOP tools or modules (or both) in the Docker can simply do

 # ACCEPT_KEYWORDS="**" USE="-modules tools static" emerge -v sys-firmware/iopmod
 # ACCEPT_KEYWORDS="**" USE="modules -tools" mipsr5900el-unknown-linux-musl-emerge -v sys-firmware/iopmod

and have them in a minute?

[AKuHAK]

The idea of Docker container is to keep as much as possible inside container to decrease the probability of user error, and so it is container's holder responsibilty that container is working. Qemu, iopmod modules, cross compiler, iopmod tools, all of that is kernel dependencies. Of course it is possible to remove everything from container and move it inside Github Action, but currently github action in linux is broken, cause produced initramfs will not contain iopmod modules.

Dispatch action cycle to my mind should be such:
Qemu repo produces dispatch action to Docker
Iopmod produces dispatch action to Docker
Docker produces dispatch action to linux repo

As I understand you suggest a bit different order:
Qemu repo produces dispatch action to Docker
Docker produces dispatch action to iopmod repo
Iopmod produces dispatch action to linux repo

As linux repo isnt a dependency for any other repo, it should not produce dispatch action to any other repos.

Maybe @fjtrujy (as ps2dev and pspdev container holder) can suggest something?

[frno7]

I think one can say that initramfs can have different levels of ‘completeness’:

1. configuration files (/etc/motd);
2. scripts (/init);
3. programs (/bin/busybox);
4. IOP module firmware (/lib/firmware/ps2/);
5. kernel modules (/lib/modules/5.4.169+/kernel/).

Of these, configurations, scripts, and programs are likely to work with almost any kernel, so supplying them in downloadable initramfs archives is helpful. It saves people some tedious work with details and cross-compilation of programs. IOP module firmware is somewhat more dependent on the kernel drivers used, and the kernel modules (the *.ko files) are generally entirely dependent on the exact kernel used, and are therefore almost unusable for any other kernel (even the same kernel version having a slightly different .config won’t work).

So a complete initramfs is actually generally only useful with a single, at one time compiled, specific kernel and nothing else.