Enable harddisk devices via the expansion bay

[frno7]

Enable harddisk devices via the PlayStation 2 expansion bay. A starting point is drivers/ata/pata_ps2.c. DMA via the subsystem interface (SIF) is needed for reasonable performance. Plan:

• initialise expansion bay and parallel ATA hardware;
• register kernel block device file;
• mount file systems for reading;
• unaligned transfer via bounce buffer;
• interrupt optimisation;
• reading via DMA;
• writing via DMA;
• performance tuning.

See also #11 and #17.

[motolav]

I'm waiting on a ABitTop SATA upgrade board and I'd be willing to test out compatibility with it when there's progress with this.

[frno7]

Thanks, @motolav! I believe the most complex issue when making a good driver will be efficient use of DMA, especially when sharing DMA with other IOP drivers, for example networking.

[rickgaiser]

There's already a highly optimized (P)ATA driver for linux v3.8. Fully using linux scatter-gather DMA, and reaching speeds close to the theoretical maximum of UDMA66. The IOP drivers needed can be found here:
https://gitlab.com/ps2max/linux/linux-firmware

It consists of:

• dev9_dma module for sharing DMA between multiple devices. State: finished
• pata_ps2 driver for pata. State: finished
• smaprpc driver for ethernet. State: working, but can be cleaned up and optimized. It uses the MDIO/PHY drivers from linux to setup the physical connections and speeds/duplex.

Also essential for debugging IOP modules, is the eedebug/iopdebug (name is confusing...) driver:
https://gitlab.com/ps2max/linux/linux-firmware/tree/master/eedebug
https://gitlab.com/ps2max/linux/linux/blob/ps2-v3.8/arch/mips/ps2/iopdebug.c
This helps to seend all print/debug messages from the IOP to the EE, so they can then be printed to the default debug output (screen or serial port)

The serial port output is IMO a valabl tool when debugging the kernel. It's been extended to also accept input, so you can start a tty console over serial:
https://gitlab.com/ps2max/linux/linux/blob/ps2-v3.8/drivers/tty/serial/ps2_uart.c

The linux implementation for pata can be found here:
https://gitlab.com/ps2max/linux/linux/blob/ps2-v3.8/drivers/ata/pata_ps2.c

The linux implementation for ethernet can be found here (files: smap*):
https://gitlab.com/ps2max/linux/linux/tree/ps2-v3.8/drivers/ps2

I would like to help out on getting these drivers to work with mailine linux, but I'm not sure if I can get a decent development environment anytime soon since I'm in the middle of moving to another country.

EDIT: For testing I have multiple PATA adapters, a SATA upgraded adapter and a sata clone adapter.

[frno7]

Great, @rickgaiser! Your contributions are most welcome. Take your time, we’ll get there.

This helps to seend all print/debug messages from the IOP to the EE, so they can then be printed to the default debug output (screen or serial port)

I have a similar printk IOP module here: https://github.com/frno7/iopmod/blob/master/module/printk.c

The serial port output is IMO a valabl tool when debugging the kernel. It's been extended to also accept input, so you can start a tty console over serial:

I’m unable to test this serial driver myself, since I haven’t soldered a serial port. As an alternative, one can use an early putc with the Graphics Synthesizer, as described in #9. It’s practically equivalent to a true serial port when debugging unmodified hardware, because it will print the same things on the video screen, although it doesn’t accept any input for obvious reasons.

[frno7]

@rickgaiser, I have started reviewing your ATA driver and the links you supplied. This will obviously take some time, as there are significant changes between the 3.8 and 5.4 Linux kernels as well as their corresponding IOP module structures. Nevertheless your driver looks like an excellent starting point.

[frno7]

@rickgaiser, I get a bus error when reading the ATA registers (from the EE), which I suspect is caused by the DEV9 not being powered on properly. So I have a iop-dev9 module that is supposed to power it on, but it seems something is amiss. It follows the steps of ps2dev9 (expbay_init) of the PS2SDK. And not entirely wrong either, I assume, because it properly detects the expansion device by printing dev9: Expansion device interface during initialisation. It also claims that the DEV9 is already powered on, by printing dev9: Expansion device already powered on, which might be wrong, because reading the registers corresponding to SPD_R_REV_1 and SPD_R_REV_3 in the PS2SDK, for example, give bus errors too.

Any ideas on what might be wrong?

There’s quite a bit of obscure hardware poking going on, as can be seen, during expansion device initialisation:

linux/drivers/ps2/iop-dev9.c

Lines 153 to 216 in f6ec831

	static int __init exp_dev_init(void)
	{
	bool power;
	int err;
	err = iop_ssbus_write_1420(0x51011);
	if (err < 0)
	return err;
	err = iop_ssbus_write_1418(0xe01a3043);
	if (err < 0)
	return err;
	err = iop_ssbus_write_141c(0xef1a3043);
	if (err < 0)
	return err;
	err = iop_dev9_power(&power);
	if (err < 0)
	return err;
	if (!power)
	{
	u16 reg_1464;
	printk("dev9: Expansion device power on\n"); /* FIXME */
	err = iop_dev9_write_1466(1);
	if (err < 0)
	return err;
	err = iop_dev9_write_1464(0);
	if (err < 0)
	return err;
	err = iop_dev9_read_1464(&reg_1464);
	if (err < 0)
	return err;
	err = iop_dev9_write_1460(reg_1464);
	if (err < 0)
	return err;
	err = exp_dev_reset();
	if (err < 0)
	return err;
	} else {
	printk("dev9: Expansion device already powered on\n"); /* FIXME */
	}
	#if 0
	err = iop_dev9_write_1464(0x103 & 0x3f); /* ??? */
	if (err < 0)
	return err;
	#endif
	err = iop_dev9_write_1466(0);
	if (err < 0)
	return err;
	iop_set_dma_dpcr2(IOP_DMA_DPCR2_DEV9);
	return 0;
	}

[frno7]

I wrote a minimal dev9 IOP module replacement, and then the initialisation works. It prints

iop: dev9: init start
iop: dev9: rev 31
iop: dev9: power is already on
iop: dev9: spdrev 11
iop: dev9: init exit

in the kernel log, as one would hope for. And then the ATA driver proceeds with

scsi host1: pata-ps2
ata1: PATA max PIO3 irq 77
ata1.00: qc timeout (cmd 0x27)
ata1.00: failed to read native max address (err_mask=0x4)
ata1.00: HPA support seems broken, skipping HPA handling
ata1.00: ATA-11: KINGSTON SA400S37120G, SBFK71E0, max UDMA/133
ata1.00: 234441648 sectors, multi 16: LBA48 
scsi 1:0:0:0: Direct-Access     ATA      KINGSTON SA400S3 71E0 PQ: 0 ANSI: 5
sd 1:0:0:0: [sdb] 234441648 512-byte logical blocks: (120 GB/112 GiB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

which is to be expected, at the moment, because it’s not fully implemented. Although I started out on the EE, it probably is best to do most things on the IOP instead, since it can offload a lot of work for the benefit of the EE. Interesting things can be done if IOP would stream all data through the SIF.

[frno7]

It’s easy to get the driver to work with polling, that is without interrtupts, by ap->flags |= ATA_FLAG_PIO_POLLING:

ata1: PATA max PIO3 irq 77
ata1.00: ATA-11: KINGSTON SA400S37120G, SBFK71E0, max UDMA/133
ata1.00: 234441648 sectors, multi 16: LBA48 
scsi 1:0:0:0: Direct-Access     ATA      KINGSTON SA400S3 71E0 PQ: 0 ANSI: 5
sd 1:0:0:0: [sdb] 234441648 512-byte logical blocks: (120 GB/112 GiB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdb: sdb1 sdb2
sd 1:0:0:0: [sdb] Attached SCSI disk

I’ve prepared two partitions that are now detected properly:

# fdisk -l /dev/sdb
Disk /dev/sdb: 112 GB, 120034123776 bytes, 234441648 sectors
14593 cylinders, 255 heads, 63 sectors/track
Units: cylinders of 16065 * 512 = 8225280 bytes

Device  Boot StartCHS    EndCHS        StartLBA     EndLBA    Sectors  Size Id Type
/dev/sdb1    0,0,2       124,254,63           1    2008124    2008124  980M 82 Linux swap
/dev/sdb2    125,0,1     256,254,63     2008125  234436544  232428420  110G 83 Linux

Mounting an EXT4 file system works too:

# mount /dev/sdb2 /mnt
# ls /mnt
lost+found

Of course, with its polling mode, it’s very slow at the moment, as seen with the dd command:

# dd if=/dev/sdb of=/dev/null bs=4096 count=4096
4096+0 records in
4096+0 records out
16777216 bytes (16.0MB) copied, 9.129766 seconds, 1.8MB/s

@rickgaiser mentioned UDMA/66, which is more than 30× faster than this. The device itself advertises UDMA/133, as shown above. The next task is to get interrupts working. Somehow the registered IRQ_IOP_DEV9 isn’t asserted. Perhaps some part of the DEV9 hardware isn’t enabled, or some special quirk is needed somewhere?

[AKuHAK]

A little offtop:
@frno7 just want to mention, that if you want to get a fast answer for some specific question, you can enjoy the discord channel. Almost all active developers are there.

[rickgaiser]

@rickgaiser, I get a bus error when reading the ATA registers (from the EE)

Although I started out on the EE, it probably is best to do most things on the IOP instead, since it can offload a lot of work for the benefit of the EE. Interesting things can be done if IOP would stream all data through the SIF.

The EE can access many things on the IOP bus (memory/devices), but these accesses are always much slower than doing things on the IOP, and transferring the data using DMA. I'm not sure what you mean by "stream all data through the SIF", but it's impossible to stream data directly from HDD -> EE. It always involves 2 DMA transfers (HDD -> IOP -> EE).

@rickgaiser mentioned UDMA/66, which is more than 30× faster than this. The device itself advertises UDMA/133, as shown above. The next task is to get interrupts working. Somehow the registered IRQ_IOP_DEV9 isn’t asserted. Perhaps some part of the DEV9 hardware isn’t enabled, or some special quirk is needed somewhere?

All needed quirks should be in here:
https://github.com/ps2dev/ps2sdk/blob/master/iop/dev9/dev9/src/ps2dev9.c

About the speed, there's a few things to note:

• The old ATA protocol is not very efficient. UDMA66 will never be able to reach 66MB/s, not even on a modern PC. More like 40 to 50MB/s.
• The fastest route to get data from HDD to the EE is 2x DMA transfers. The first from HDD -> IOP. The second from IOP -> EE. Since both transfers use the IOP memory, that will become a bottleneck. With the IOP memory bandwith a little over 100MB/s, the theoretical max speed is half of that.

To gain max speed:

• both DMA transfers have to be active at the same time.
• scatter-gather lists from linux have to be sent to the IOP. The IOP can transfer the data directly to the correct memory location on the EE.

That being said, 40+ MB/s has been proven possible on ps2/linux.

[frno7]

I'm not sure what you mean by "stream all data through the SIF", but it's impossible to stream data directly from HDD -> EE. It always involves 2 DMA transfers (HDD -> IOP -> EE).

That’s what I had in mind. The SIF can exchange data with several devices too, such as the image processing unit (IPU) in #15 etc. and so isn’t limited to memory transfers only. In principle, I suppose one could make a command such as cat somefile >/dev/ipu do all data copying in hardware. That could be interesting.

Incidentally, would you happen to know the details on how to acknowledge interrupts with DEV9 (see frno7/iopmod#5), more specifically the ones for SPD_REG16(SPD_R_INTR_STAT) and SPD_REG16(SPD_R_INTR_MASK)? I would expect that to be done somewhere near dev9_intr_dispatch in the PS2SDK, but I couldn’t find it.

Related to frno7/iopmod#5 is also finding documentation on the following DEV9 registers, if someone has figured them out and invented better names for them:

enum _dev9_regnames {
       DEV9_R_1460, DEV9_R_1462, DEV9_R_1464, DEV9_R_1466, DEV9_R_1468,
       DEV9_R_146A, DEV9_R_POWER,DEV9_R_REV,  DEV9_R_1470, DEV9_R_1472,
       DEV9_R_1474, DEV9_R_1476, DEV9_R_1478, DEV9_R_147A, DEV9_R_147C,
       DEV9_R_147E,
       DEV9_R_MAX };

[rickgaiser]

Incidentally, would you happen to know the details on how to acknowledge interrupts with DEV9 (see frno7/iopmod#5), more specifically the ones for SPD_REG16(SPD_R_INTR_STAT) and SPD_REG16(SPD_R_INTR_MASK)? I would expect that to be done somewhere near dev9_intr_dispatch in the PS2SDK, but I couldn’t find it.

Perhaps you're looking for SMAP_REG16(SMAP_R_INTR_CLR), as in this example:
https://github.com/ps2dev/ps2sdk/blob/dd9b0b734b1cabf7342c5e2d9278967024745922/iop/network/smap/src/smap.c#L445
I'm not sure why the one is named "SMAP" and the other "SPD", but it seems to use the same flags for both registers in the link above.

[frno7]

Yes, thanks, I’ll try that! With cascading interrupts for DEV9, I believe some parts of the ATA driver can be simplified, and perhaps even more so the SMAP driver. The IRQ relay module will work automatically with DEV9 IRQs too, and so on.

[AKuHAK]

As about dev9 register references - I found only this, this and this.

[frno7]

@AKuHAK, the first link seems broken, no?

[frno7]

Interrupts are now working with the ATA driver, via a new IRQ_IOP_SPD_ATA0 virtual IRQ relay, as explained in frno7/iopmod#5 (comment). Data transfer performance isn’t any better for it, at the moment, but it’s nevertheless a prerequisite for DMA that is expected to be a major improvement.

[frno7]

Now reading via DMA works, as a proof of concept, with a useful 31 MB/s:

# dd if=/dev/sda of=/dev/null bs=4096 count=4096
4096+0 records in
4096+0 records out
16777216 bytes (16.0MB) copied, 0.513587 seconds, 31.2MB/s

That’s a 17× improvement over polling. Verifying file checksums on a mounted file system etc. works too. Before implementing writing via DMA, I’m going to refactor the driver. Redesign it entirely, in fact, to make better use of resources. This harddisk driver will eventually need to work concurrently and share resources with the network driver described in #19, for example.

[uyjulian]

It is possible to place APA and GPT on the same hard disk, so you could e.g. boot Linux from the hard disk (possibly with a kernel bootstrapper as __mbr (not to be confused with MBR partitioning scheme)) and use e.g. ext4 on that same hard disk. https://www.psx-place.com/threads/apa-jail.34847/

[frno7]

@uyjulian, I’m unfamiliar with APA, but the general steps for Linux 5.x are these:

1. a boot loader such as wLaunchELF starts a compressed ELF image of a Linux 5.x kernel, from any medium that the boot loader can manage;
2. the INITRAMFS embedded in the compressed Linux 5.x kernel ELF configures the necessary devices and starts any custom init scripts, and so on;
3. optional file systems are mounted etc. and the root_pivot command is performed to switch the root from the RAM disk to some other mounted file system, whatever that is;
4. the INITRAMFS can be unmounted and discarded, to reuse its memory for more useful things.

If the Linux kernel doesn’t understand APA, then this could of course, in principle, be implemented. I did some searching and found that someone has written a guide for APA partition support that claims that for Linux 2.x this allows your PS2 Linux and PS2 Game Data to co-exist on the same hard-drive. Linux 5.x may need some additional adaptions, though.

In short, yes, it should be possible with a bit of work.

[AKuHAK]

This hybrid scheme will allow to not use APA at all. The only thing, in such a case primary GPT partition header should be in sector 2, not in sector 1, first 2 sectors will be filled with customized APA mbr, and partitions should start at specific offsets. This approach differs from link that you posted, cause there is firstly mounted APA partition, but with customized hybrid scheme you can skip APA step completely.

[frno7]

@AKuHAK, thanks for the clarification. I’ve created #39 for the APA.

[frno7]

modprobe pata_ps2 will now enable reading harddisks, including mounting compatible preexisting filesystems via mount -o ro /dev/sda1 /mnt or suchlike. Something like dd if=/dev/sda of=/dev/null bs=4096 count=4096 can be used to test reading speeds.

Commits frno7/iopmod@5ccf9d9 and frno7/iopmod@5ccf9d9 have the corresponding DEV9 and ATA modules for the IOP. As usual, the IRX modules are compiled, installed and linked automatically on a properly updated and configured system.

Unfortunately, I’ve not yet been able to complete the refactoring, due to an apparent hardware timing issue. The DEV9 hardware appears to crash with the changes I was planning to do. Maybe something becomes slightly wrong with the interaction between the ATA and the DEV9 DMA registers? DEV9 triggers bus errors when attempting to access its registers. I won’t attempt to enable writing harddisks until this is solved.

A bounce buffer now handles all cases of unaligned transfers properly. Reading speed is currently about 20 MB/s with the particular hardware I have. At least 30 MB/s can be done with some tuning. Maybe even more than 40 MB/s with some effort.

[uyjulian]

One thing I noticed is that you are missing the delays when powering on dev9. If those delays are missing, they could be possibly causing a bus error.

Another possibility is the connection between the PS2 and the network adapter is not very good, which may also cause bus errors.

If the code is based on a Linux kernel that needs to be booted from kernelloader or RTE, it may be possible that dev9 was already powered on beforehand, so that code path was not tested very much.

[frno7]

One thing I noticed is that you are missing the delays when powering on dev9. If those delays are missing, they could be possibly causing a bus error.

Hmm. Yeah, but the pata_ps2 driver on the ps2-main branch actually does work with the current delayless DEV9 power on sequence. It’s stable without issues after hours of testing. :-)

Another possibility is the connection between the PS2 and the network adapter is not very good, which may also cause bus errors.

I suspect it’s an ATA module (iopmod:module/ata.c) software problem with the IOP, at least until I learn more details. The driver is fine when DEV9 DMA is serviced in its ata_sif_cmd_sg SIF command interrupt. However, I should like it to use a separate IOP thread, by waking it up with thsemap_isignal_sema(dev->sg_sema_id), rather than calling ata_sif_cmd_sg_transfer directly during the interrupt:

#if 0
	/* Waking up the thread to do DEV9 DMA crashes the DEV9 hardware. Timing problem? */
	thsemap_isignal_sema(dev->sg_sema_id);
#else
	/* Servicing DEV9 DMA directly in the SIF command interrupt is fine. */
	ata_sif_cmd_sg_transfer(dev);
#endif

Servicing DEV9 DMA requests in a separate IOP thread changes the timing, which I suspect is the issue with the IOP thread:

	for (;;) {
		thsemap_wait_sema(dev->sg_sema_id);

		ata_sif_cmd_sg_transfer(dev);
	}

If the code is based on a Linux kernel that needs to be booted from kernelloader or RTE, it may be possible that dev9 was already powered on beforehand, so that code path was not tested very much.

Yeah, the IOP DEV9 module iopmod:module/dev9.c isn’t exactly tidy, mostly because it’s a pile of magic numbers for undocumented DEV9 registers. dev9_power_off isn’t even implemented properly. Hopefully, all these registers and their bit fields can be given names and meanings, eventually.

[uyjulian]

For booting off HDD, you actually don't need to worry about APA. You can just stick with MBR or GPT while performing minimal modifications to sector 0.

In order to do hdd boot, only sector 0 (same as MBR) and the sector size/offset specified in sector 0 (can create a GPT partition to store this data) needs to be modified (which specifies the location and size of the program to be loaded).
In sector 0, it only cares about bytes 304-312. That is where the offset and size (4 byte integer each, in sectors) are stored
Normally, in MBR, that would be inside the bootstrap code part 2 location
https://en.wikipedia.org/wiki/Master_boot_record#Sector_layout
SoftDev2 is one such program that can be specified in that sector size/offset: https://github.com/parrado/SoftDev2

So it should be possible to make a disk that is bootable in both PC(BIOS+UEFI) and PS2. (But there isn't much of a use case for that.)

[frno7]

I assume some people will use the fdisk command that comes with the precompiled Busybox, see frno7/gentoo-mipsr5900el#5 (comment), so everything about partitioning and formatting disks can be done on the PlayStation 2. That’s how I prepared its hard disk, anyway. Is this boot tied to having wLaunchELF?

[uyjulian]

wLaunchELF is not required. The Linux chainloader or the Linux kernel itself can be stored on the specified sector size/offset.

However, the following restrictions reply:

1. The binary must be a raw binary to be loaded at 0x00100000 (-O binary argument of objcopy), wrapped in a KELF container. SoftDev2 has a sample of this
2. The binary must fit in the IOP memory. It won't be streamed into the EE (so basically this makes the size limit less than 1MB)
   So if the Linux kernel can't fit in less than 1MB, a chainloader is needed (just like BIOS needs it to fit in the MBR)

[frno7]

Ah, is standard Sony BIOS able to boot programs from hard disks?

[uyjulian]

Ah, is standard Sony BIOS able to boot programs from hard disks?

Only from SCPH-3xxxxx to SCPH-5xxxxx.

[frno7]

Ah, is standard Sony BIOS able to boot programs from hard disks?

Only from SCPH-3xxxxx to SCPH-5xxxxx.

Oh, this is very nice and something I think we should most definitely make good use of. :-)