Kernel Bug Detected, ohci-ps2 related, PAL SCPH-70003

[nickb834]

I've built a gentoo based cross compiler as per the wiki docs, and managed to get a bootable kernel first time. Unfortunately I get a segmentation fault so whilst I can see the output on the console, I can't interact with the shell.

I've commented out the modules in /sbin/init and discovered that as soon as modprobe ohci-ps2 is run, there's a kernel bug detected followed by a segmentation fault.

If I don't load any other module than ps2fb then there's no kernel bug nor segmentation fault - boots to a prompt just fine but of course no usb keyboard to play with.

I can't think of a method to get you a screenshot (can't ssh in for eg) other than litereally take a couple of photos:

PS2FB loaded only:

PS2FB & OHCI-PS2 loaded only:

[nickb834]

I wonder, with that request interrupt 86 failed are the irx modules getting loaded properly / at all...... given that two of them at least relate to irq (interrupts). maybe I put them in the wrong place? currently they're inside the /usr/src/initramfs/ps2/lib/firmware/ps2 dir

[frno7]

Hi @nickb834, congratulations on getting this far on the first try! The segmentation fault may be a consequence of USB probing failing, in which case it attempts to clean up and free a NULL pointer, which apparently isn’t supported. The lines at the top similar to

ohci_ps2: Unknown symbol usb_hcd_platform_shutdown (err -2)

are very interesting. I believe we need to fix those.

Does your /sbin/init use modproble in the same order as described in #33 (comment)? I’m also wondering whether some modules in arch/mips/configs/ps2_defconfig have been compiled into vmlinux rather than being separate .ko files to load dynamically? All combinations of linking (y) and loading (m) modules have not been tested yet. Perhaps your config is different somehow?

[nickb834]

This is my /sbin/init- taken from wiki verbatim (bar mode_option tweaks)

#!/bin/sh

PATH=/usr/sbin:/usr/bin:/sbin:/bin
export PATH

mount -t sysfs none /sys
mount -t tmpfs none /var
mount -t tmpfs none /tmp
mount -t proc none /proc
mkdir /dev/pts
mount -t devpts none /dev/pts
mount -t debugfs none /sys/kernel/debug

modprobe ps2fb mode_option=1280x720p@60 mode_margin=+60+0
modprobe ohci-ps2

modprobe sd_mod
modprobe ums-usbat
modprobe usbhid
modprobe hid-generic

uname -a

sh

while :
do
	sleep 1
done`

Only difference really between mine and yours in terms of ordering is I have ohci-ps2 before sd_mod - other than that I have previously tried adding in the other modules you have to no avail:

modprobe sif
modprobe iop-memory
modprobe iop-module
modprobe iop-irq
modprobe iop-fio
modprobe iop
modprobe iop-dev9

I took the ps2_defconfig from the repo and have left it well alone! I'm a little out of my comfort zone so changing minimal amount of things possible.

I'll rebuild vmlinuz with modules in same order as yours and report back

[frno7]

Some modules mentioned in #33 (comment) have been deprecated, so that (old) list is by no means perfect, but at least modprobe ordering ought to be good.

[frno7]

Also, don’t be alarmed about a cursor not blinking at the prompt, as one might expect. It’s a technical problem mentioned on the wiki. :-)

[nickb834]

Yep - I'd seen that about the cursor - I read the wiki 100% before I even started as I wanted to be sure it was doable! I even read all the open and closed issues - lots of useful info in there not that I understood a good chunk of it.

So- good news, module ordering appears to be an importing thing, the seg fault has gone away as has the kernel bug and even the failed interrupt - but there's still issues with usb etc:

[frno7]

Could there be a problem with module dependencies?

% ls initramfs/ps2/lib/modules/5.4.169+
build                      modules.builtin.bin      modules.softdep
kernel                     modules.builtin.modinfo  modules.symbols
modules.alias              modules.dep              modules.symbols.bin
modules.alias.bin          modules.dep.bin          source
modules.builtin            modules.devname
modules.builtin.alias.bin  modules.order

ohci-ps2.ko should have some, like this:

% grep ohci-ps2 initramfs/ps2/lib/modules/5.4.169+/modules.dep
kernel/drivers/usb/host/ohci-ps2.ko: kernel/drivers/usb/host/ohci-hcd.ko kernel/drivers/usb/core/usbcore.ko kernel/drivers/usb/common/usb-common.ko kernel/drivers/ps2/iop-heap.ko kernel/drivers/ps2/sif.ko kernel/drivers/ps2/iop-memory.ko kernel/drivers/ps2/iop-registers.ko

By the way, I’m currently using the following commands to build the kernel:

make -j58 ARCH=mips CROSS_COMPILE=mipsr5900el-unknown-linux-gnu- vmlinux

rm -rf ../initramfs/ps2/lib/modules
make -j58 ARCH=mips CROSS_COMPILE=mipsr5900el-unknown-linux-gnu- modules
make -j58 ARCH=mips CROSS_COMPILE=mipsr5900el-unknown-linux-gnu- INSTALL_MOD_PATH=../initramfs/ps2 INSTALL_MOD_STRIP=1 modules_install

make -j58 ARCH=mips CROSS_COMPILE=mipsr5900el-unknown-linux-gnu- vmlinuz

[nickb834]

so my modules in initramfs looks like:

gentoo /usr/src/initramfs/ps2/lib/modules/5.4.169+ # ls -w 80
build                      modules.builtin.bin      modules.softdep
kernel                     modules.builtin.modinfo  modules.symbols
modules.alias              modules.dep              modules.symbols.bin
modules.alias.bin          modules.dep.bin          source
modules.builtin            modules.devname
modules.builtin.alias.bin  modules.order

looks same?

ohci-ps2.ko deps:

gentoo /usr/src/initramfs/ps2/lib/modules/5.4.169+ # grep ohci-ps2 modules.dep
kernel/drivers/usb/host/ohci-ps2.ko: kernel/drivers/ps2/iop-heap.ko kernel/drivers/ps2/sif.ko kernel/drivers/usb/host/ohci-hcd.ko kernel/drivers/ps2/iop-registers.ko kernel/drivers/ps2/iop-memory.ko kernel/drivers/usb/core/usbcore.ko kernel/drivers/usb/common/usb-common.ko

Which looks same too?

I'm building kernel same way as you but without the -j58 as I'm running in a 4 core virtuablbox vm on a 3 year old macbook pro and it's not that grunty in comparison!

[frno7]

It’s puzzling because the errors mentioned in #66 (comment) are for generic modules. Nothing specific about PlayStation 2, as it seems. It looks like the kernel has difficulties finding the kernel modules it wants to load via dependencies.

Hmm. Could it be a simple permission problem? Make sure UID and GID correspond to the owner of the files in ../initramfs/ps2, and that all files and directories have the same owner:

% grep CONFIG_INITRAMFS .config
CONFIG_INITRAMFS_SOURCE="../initramfs/ps2"
CONFIG_INITRAMFS_ROOT_UID=1000
CONFIG_INITRAMFS_ROOT_GID=1000
CONFIG_INITRAMFS_COMPRESSION=".xz"

I'm building kernel same way as you but without the -j58 as I'm running in a 4 core virtuablbox vm on a 3 year old macbook pro and it's not that grunty in comparison!

No worries, except be careful not to overheat the poor processor with all compilations. For that reason I’ve down-clocked my dual core MacBook 2011 model, running Gentoo, that I normally used until summer 2018, when I bought a POWER9 dual socket 64 core machine. :-)

[nickb834]

checked dirs and perms:

gentoo /usr/src # cd linux-5.4.169-ps2/
gentoo /usr/src/linux-5.4.169-ps2 # ll ../initramfs/ps2/
total 60
drwxr-xr-x 14 root root 4096 Feb  1 22:20 .
drwxr-xr-x  3 root root 4096 Jan 31 23:36 ..
drwxr-xr-x  2 root root 4096 Jan 31 21:40 bin
drwxr-xr-x  2 root root 4096 Jan 31 21:39 dev
drwxr-xr-x  2 root root 4096 Jan 31 21:39 etc
-rwxr-xr-x  1 root root  164 Jan 31 21:41 init
drwxr-xr-x  4 root root 4096 Feb  1 00:33 lib
drwxr-xr-x  2 root root 4096 Jan 31 21:39 mnt
drwxr-xr-x  2 root root 4096 Jan 31 21:39 proc
drwxr-xr-x  2 root root 4096 Jan 31 21:39 root
drwxr-xr-x  2 root root 4096 Feb  8 17:12 sbin
drwxr-xr-x  2 root root 4096 Jan 31 21:39 sys
drwxr-xr-x  2 root root 4096 Jan 31 21:39 tmp
drwxr-xr-x  4 root root 4096 Jan 31 21:39 usr
drwxr-xr-x  2 root root 4096 Jan 31 21:39 var
gentoo /usr/src/linux-5.4.169-ps2 # grep CONFIG_INITRAMFS .config
CONFIG_INITRAMFS_SOURCE="../initramfs/ps2"
CONFIG_INITRAMFS_ROOT_UID=1000
CONFIG_INITRAMFS_ROOT_GID=1000
CONFIG_INITRAMFS_COMPRESSION=".xz"
gentoo /usr/src/linux-5.4.169-ps2 # find ../initramfs/ps2/ ! -user root
gentoo /usr/src/linux-5.4.169-ps2 # find ../initramfs/ps2/ ! -group root

Looks right I think?

bought a POWER9 dual socket 64 core machine. :-)

POWER9 peaks my interest - that's a hefty amount of cores!

[frno7]

Try nought

CONFIG_INITRAMFS_ROOT_UID=0
CONFIG_INITRAMFS_ROOT_GID=0

as root likely has UID 0 and GID 0 on your host (MacBook) machine, similar to

% id root
uid=0(root) gid=0(root) groups=0(root)

The file linux/usr/Kconfig has this to say about them:

config INITRAMFS_ROOT_UID
        int "User ID to map to 0 (user root)"
        depends on INITRAMFS_SOURCE!=""
        default "0"
        help
          If INITRAMFS_SOURCE points to a directory, files owned by this UID
          (-1 = current user) will be owned by root in the resulting image.

          If you are not sure, leave it set to "0".

config INITRAMFS_ROOT_GID
        int "Group ID to map to 0 (group root)"
        depends on INITRAMFS_SOURCE!=""
        default "0"
        help
          If INITRAMFS_SOURCE points to a directory, files owned by this GID
          (-1 = current group) will be owned by root in the resulting image.

          If you are not sure, leave it set to "0".

It’s normally recommended to compile the kernel as someone else than root, which is why the mapping is needed since INITRAMFS must have many if not all files owned by root. The kernel and some commands such as dropbear may otherwise refuse to load certain files, for security reasons, which I suspect is happening with the module dependencies in your case.

The default in linux/arch/mips/configs/ps2_defconfig is 1000 which may not map to anything useful, so we should probably change this. And the wiki ought to have a note about this, of course.

[nickb834]

Good news!

USB now works and I can interact with the shell.

All I did to get things working was:

make -j4 ARCH=mips CROSS_COMPILE=mipsr5900el-unknown-linux-gnu- vmlinux

rm -rf ../initramfs/ps2/lib/modules
make -j4 ARCH=mips CROSS_COMPILE=mipsr5900el-unknown-linux-gnu- modules
make -j4 ARCH=mips CROSS_COMPILE=mipsr5900el-unknown-linux-gnu- INSTALL_MOD_PATH=../initramfs/ps2 INSTALL_MOD_STRIP=1 modules_install

make -j4 ARCH=mips CROSS_COMPILE=mipsr5900el-unknown-linux-gnu- vmlinuz

I suspect - it was the rebuild of the modules (after the rm) that did it. Perhaps I didn't build things in the right order first time round.

[frno7]

Bliss! Nevertheless I’d like to keep this issue open as a reminder to

• fix the error recovery NULL pointer exception in ohci-ps2.c;
• add the SCPH-70003 model to the hardware model compatibility wiki page, below known to be compatible, perhaps you’d like to do this?
• clarify the wiki on UID and GID permissions for INITRAMFS;
• somehow improve the wiki on compiling the kernel, to avoid these dependency problems.

You’ve got permission to edit the wiki, so if there’s anything you think could be improved just edit, grammar, spelling, technical things, etc. Note that the wiki pages are reStructuredText, in common with Linux kernel documentation, as opposed to Markdown which is generally more common on Git Hub.

/ # date
Thu Jan  1 00:02:44 UTC 1970

The PlayStation 2 real time clock (RTC) is actually supported in Linux. Maybe its battery is drained? :-)

[nickb834]

I've updated wiki for 70003 known compatability, - for the compiling kernel I think I messed up the order of operations (as best I can tell) so I'll make it clearer based on what worked for me.

The PlayStation 2 real time clock (RTC) is actually supported in Linux. Maybe its battery is drained? :-)

Yeah my battery is dead, I bought another and I have no idea what I've done with it!

[frno7]

Thanks! The astute observer will notice some Unknown symbol with (err -2) in your latest log in #66 (comment). I fear the issue may resurface. The modules affected seem to be a bit random... What are the file and directory ownerships on the PlayStation 2 hardware, especially in / and /lib/modules/5.4.169+/?

[nickb834]

I've updated the wiki for building the Building a PlayStation 2 Linux INITRAMFS root filesystem to point to Building a PlayStation 2 Linux kernel to build the modules, and then steered the reader back to Building a PlayStation 2 Linux INITRAMFS root filesystem#install kernel modules to install them.

I don't like how the user journey flows to kernel build, back to initramfs and back to kernel build - but I couldn't think of a better page structure that kept initramfs and kernel build as seperate pages.

It should however be clearer now.

[nickb834]

All files and dirs are owned by ‘0’, a “root" user doesn’t actually exist.

…

On 9 Feb 2022, at 15:13, frno7 ***@***.***> wrote: Thanks! The astute observer will notice some Unknown symbol with (err -2) in your latest log in #66 (comment) <#66 (comment)>. I fear the issue may resurface. The modules affected seem to be a bit random... What are the file and directory ownerships on the PlayStation 2 hardware, especially in / and /lib/modules/5.4.169+/? — Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIUOBM5OR2PQAH6P2ZWYXS3U2KABBANCNFSM5NYJNT2A>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.

[frno7]

Could you try to run depmod, before any invokations of modprobe? It’s got various options to update and verify module dependencies. The one in Busybox is simplified. The original has more interesting options, such as

       -w
           Warn on duplicate dependencies, aliases, symbol versions,
           etc.

The keyword here being ‘etc’. :-)

[frno7]

The BUG_ON observed in #66 (comment) is

linux/lib/genalloc.c

Lines 251 to 253 in 59a11ab

	end_bit = chunk_size(chunk) >> order;
	bit = find_next_bit(chunk->bits, end_bit, 0);
	BUG_ON(bit < end_bit);

with bit as 0 and end_bit as 16384. The error path is

__BUG_ON
gen_pool_destroy
release_nodes
really_probe
device_driver_attach
__driver_attach
bus_for_each_dev
bus_add_driver
driver_register
do_one_initcall
do_init_module
load_module
sys_finit_module
syscall_common

in generic kernel code, due to missing kernel modules (the preceding unknown symbol errors), seemingly completely unrelated to the PlayStation 2, although it’s triggered by ohci-ps2.c and nonfunctional kernel module dependencies. In effect, it’s an error caused by another error. A likely fix is described in #69 (comment).

[frno7]

I’ve updated the wiki article on building a PlayStation 2 Linux INITRAMFS root filesystem with details on depmod, and UID and GID filesystem ownership.