Test PCIe switches and adapters #14
Comments
|
With the 1 to 2 ports adapter, I'm able to get multiple devices to load, so that's nice: I'm going to confirm I can only transfer data at up to 3.2 Gbps though. Maybe less with the overhead of the PCI bridge? |
|
To test NVMe + SATA, or anything plus anything requiring extra power... I'm going to need to grab an external PSU. I just don't have enough adapter cables to get from my single 4-pin molex power adapter to both the floppy connector on the PCIe switch plus SATA power for the drives. Might as well buy a PC case at some point to contain all this crazy mess haha. Anyways, I have my eye on the Redragon GC-PS003 600W fully modular PSU... mostly because it's close, relatively cheap, and available whenever. But I'm doing plenty of other stuff so it's not yet a priority. Edit: Just ordered that PSU, and I'll have a little more fun with the switch next week! |
|
The PSU is quite girthy, and also has to have a jumper across two pins on the motherboard connector so it will switch on. That was unforeseen, but obvious in hindsight. |
|
I found another one possibly worth testing: https://www.ebay.co.uk/itm/Excellent-PCI-e-Express-1X-to-3-1X-Switch-Multiplier-HUB-Riser-Card-USB-Cable-GN/353247990975
|
|
@Chaz6 - I'm actually going to try another switch, which might also be nice on the CM4, though it will likely cause a bit of strain on the connector... https://www.ebay.com/itm/PCIe-1x-to-Dual-PCIe-1x-90-Degree-Riser-Card-with-Power-PCIe-Switch-Expansion/273855896755?hash=item3fc3155cb3:g:fVwAAOSwHu5c4uvd |
|
@Chaz6 I have that card (or a variant of it, anyway. Same design). The sticker covers it, but the chip on the board is of the same series as the IOCrest card that Jeff already tested, albeit one with a slightly expanded feature set (adding an extra port). They utilize the same kernel driver, so should ideally function exactly the same. That said, I'm going to test it when my CM4 turns up one of these days and report back the results. |
|
I'm currently testing that card from @Chaz6 in my PC (CM4 not here yet), but I'm having trouble getting multi-lane cards to work. Have you been experiencing the same? The only 1x card I have is a Realtek Gigabit card, which seems to work fine. But I haven't gotten any other card to work, I tested a Samsung 970 EVO (with 4x adapter), Mellanox 2 10GbE card and a dual gigabit card pulled from a HP server, none working. I cut open the connector to plug in longer cards since I don't have a riser card right now, but the 1x card still works in that port (so I guess nothing broke). The cards not working do light up or get warm, but nothing in lspci, and nothing obvious in dmesg. |
|
@markus-k - I can't speak to @Chaz6's card, but the IO Crest seems to so far only work with devices plugged in through slot 2 (maybe slot 1 is dead, maybe something in the Pi's PCIe stack doesn't work with it)... but I haven't done any rigorous testing with it yet. I'm still focused right now on a number of individual cards and haven't yet done thorough testing with multiple cards. |
|
Just to give an update, I figured out what the problem was (or is) after some digging into the switch and PCIe documentation. Not really related to the CM4, but probably useful to everyone thinking about cutting up their 1x PCIe switch ports to plug many multi-lane cards into their CM4. When you just google about cutting open 1x PCIe ports everyone will tell you, it'll work no problem. That is not entirely true though, depending on how presence detection is implemented on the PCIe carrier board. PCIe cards have two pins, PRSNT1# and PRSNT2#, which need to be directly connected on the card. PCIe cards with more than one lane have multiple PRSNT2# pins, one for each lane (i.e. one for 1x, 4x, 8x, 16x), but only the one at the maximum number of supported lanes has to be connected to PRSNT1#. Now, when you cut open your 1x port, PRSNT1# is connected to a pin that is not even present on the slot connector, and the 1x PRSNT2# is floating since the card supports more lanes. Thus, the PCIe switch does not detect that a card is plugged in and disables the port. When you use a proper 1x to Nx adapter, the correct PRSNTx# pins will usually be connected and the card will be detected. Presence detection via PRSNTx# can be disabled in the switch, but the EEPROM is not fitted on the board, and you'd need to modify it's contents even if it was there. One other whacky solution, if you don't want to order a proper raiser, I used is to get out the soldering iron and botch a wire between PRSNT1# and PRSNT2# on the board, so the switch thinks there is always a card plugged in. PRSNT1# is connected to ground on this particular board, and PRSNT2# connected to the PCIe switch. I wouldn't really recommend doing this, but if you are too cheap to order yet an other PCIe adapter, it's probably fine. And see, now cards with more than one lane are detected on the host. |
|
@Chaz6 I have an almost identical 3 way card https://www.ebay.co.uk/itm/Excellent-PCI-e-Express-1X-to-3-1X-Switch-Multiplier-HUB-Riser-Card-USB-Cable-L-/154052637457. |
|
I also wanted to post an update—earlier I do remember having at least the Marvell 9215 (IO Crest) SATA card and Samsung 970 EVO NVMe working at the same time on my 2-slot switch, so it did work at least. I'll have to continue doing more testing. I have another new riser that has two slots and a different chip. |
|
This one seems to be working too: https://www.aliexpress.com/item/32960541989.html
|
|
@danielkucera - Can you confirm having more than one device plugged in works, identifying all the devices? I noticed that many times it would only show devices plugged into the 2nd slot on my switch. |
|
Can you try one of these PCIe to PCI(-X) cards too? Like this for example. It would make it possible to use many old PCI cards on the RPi |
|
@darkstar Tried already - https://www.raspberrypi.org/forums/viewtopic.php?p=1772997#p1772997 |
|
is anyone able to test ASM1184e chipset, I'm planing to build my own riser card with 1 x1 to 2 x4 + 2 M.2
It uses ASMedia chip ASM1184e: https://www.aliexpress.com/item/32868079091.html |
|
I have just put two cards in the first one pictured in Jeff's post and get this: What does seem to be the case though is that it will only recognise pciex1 ie no pciex4 on a riser, I cannot get it to recognise a single pciex4 device. |
|
I have tried the one dlednik recommended, this is the one for us guys!! On my Amazon it is listed as a different brand but looks like its the same as is the case with many of these devices: https://www.amazon.co.uk/gp/product/B07NZ5W8KJ/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1 To test it, I plugged in four devices, all four worked and the cool thing is that this is a mixture of x1 and x4 devices All four devices working. A 4x Intel 10G network card, a 1x 2 port network card, a marvel 1x 4 port sata card and a 4x nvme card Happy days :D |
@PBXForums Have a look at my previous comment: #14 (comment). Check if your Riser properly connects both PRSNT# pins. |
|
@markus-k They work in the one in my above post, the two port one is now being scrapped in favor of that one which seems to be working perfectly. |
|
Some devices (like the CM4IO) don't check presence with the PRSNT# pins but whether theres a 50 Ohm load on the PCIe lanes. You can achieve the same effect on the Pericom switches by just shorting the PRSNT# pins (or configuring the switch, which is a lot harder to do). |
|
@markus-k Thanks, I want at least three ports anyway. |
|
Testing this again with the IOCrest switch but with my 700W power supply for a NAS build... I finally seem to have it powered stably enough to get two devices working, and the system doesn't completely lock up when I run I was trying the same thing with the Exacq switch, and kept either having only one of the two cards work, or the Pi would lock up at various points through the boot process. Note that I tried powering it both via dedicated 2A molex power supply and via my 700W (100+W to the 5v bus) PSU, and it seemed to work more stably connected through my PSU. It still required one extra reboot before both cards were fully functional though. |
|
Seems to be extremely stable now (the IOCrest switch). I think for any of these switches, a solid power supply capable of handling 3-5A or more is going to be pretty important. But even then, some of the adapters and switches just... fail sometimes, and cause kernel panic. No idea why that happens randomly. |
|
Sorry if this is a dumb question - with using these that have a USB cable - do these perform as if they were simply USB devices or are they using a USB interface only? My goal is to connect a drive and a VPU (like the coral). In doing that, would it be better to just leave the drive on USB? If this isn't the place to ask, feel free to delete or point me in the right direction to learn more. |
They're using USB3 cables purely as a convenient physical cable. They do not convert to/from USB at an electrical or protocol level. As Jeff noted, there is also no standard wiring with these adapters, so ensure you use the correct pairs of adapters. |
|
I've had my eye on these for my mining shinanigans and they look perfect for Raspberry Pi usage given their connectivity is a USB socket instead of a PCIe slot but can't seem to find them anywhere else and the vendor didn't answer my request for quote but thought I'd share anyway incase anyone knows where to get them. https://www.alibaba.com/product-detail/pcie-splitter-pcie-to-pci-adapter_60572718142.html |
|
Was anyone able to test the PI7C9X2G404SL chip? |
|
@geerlingguy Wow dude, that is awesome what you are trying to do. |
I tried that chip in Ubuntu Core 20 without success -- but I haven't confirmed the adapter card definitely works. Interested to know experience of other people. -- not had chance to look to see if it was a driver issue for ubuntu |
|
A PCIe compliant switch is supported by the native pci driver. Unless the hardware is defective, the PI7C9X2G404SL should be enumerated across all platforms. Full disclosure, our company is a mfr of many PCIe designs and use this component with success. |
|
Thanks @juthi that's what I wanted to know. Looks like poor build quality, have another one on order so will be able to confirm for @RuhanSA079 . |
|
Bought three of the Exacq Technologies 1 to 2 ports PCIe x1 adapter none of them show up with lspci - all broken? |
|
@100nix, are you powering the PCIe adapter using the molex connector? That is required. |
|
Thanks @juthi - yes, see picture - are there any LEDs on the card that should light up when operational? I wonder if all three cards I bought are duds. |
|
Personally do not have this adapter and as a company, we did not use the TI bridge ages ago due to compatibility issues with the PCIe enumeration. We camped onto the PEX8112 bridge which worked in every brand we tested - we have 1M+ adapters in the field using their bridge. Early on a major brand name of PC had enumeration faults but they corrected the issue via BIOS code within 48 hours of notification.
Forgot to note, this is a standard PCIe bridge (fanout) design. You could also test the same inside of your Windows PC. Just be sure that the PCIe gold finger keying is correct & apply the PC power supply molex connector then check your Device Manager. If you are DOS person - you can use one of the freeware PCI utilities to scan & dump your PCIe bus to see if it enumerating on your PC. |
|
Thanks @juthi - plugged in one of my PCI to NVMe adapters - doesn't see it. Here is what I see in dmesg: [ 1.276644] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges: |
|
Hi. Not a rpi guy...yet but suggest the following:
lspci -vv ; that is 2 * 'v' as in Victor and not a 'w' this will display all PCI devices found by the host CPU. Inside this listing, check to see if you can locate the Texas Instrument PCIe switch. From the TI datasheet, you can find the PCI IDs (Vendor & Device ID). If found, then the PCIe switch is being detected by the host CPU. Then could be a s/w issue with the RPI distribution but not sure. Respectively, you can also seat the same adapter inside a normal PC and check to see if the switch is found. These switches (like others) use the CLASS CODE to be auto-enumerated by every operating system. Respectively, no device driver is required for your Windows / Linux OS running on your x86 / AMD / Intel CPU. This is to only validate that the adapter is working.
|
|
Silly but important concern, can you post a clear picture of the molex connector onboard your external power adapter? Also check against the industry standard pinout. A standard PCIe slot offers 3v3 and 12v0 rails. Be sure this molex connector is supplying these voltages to the proper PCIe adapter pins. |
|
Hey @juthi I moved on for the moment to play around with the ASHATA PCE4PCE-A01 1 to 4 Port PCIe Switch.
It seems to work, as you can see on the picture I am using the same power adapter with a sata power plug. I will revert back once I get a chance to try the Exacq Technologies 1 to 2 ports PCIe x1 adapter again. Thanks Alex |
I'm not sure if it's the issue here, but try powering the pi and the card from one power supply, e.g. a PC power supply, as there have been issues with using multiple power supplies (bad ground connection resulting in connection issues. Jeff had those issues with GPUs) |
|
Excellent advice @Coreforge. On a standard PC PCIe slot, the PCIe adapter must be fully powered and awake before the host bios can enumerate the PCIe function. On some protos, we intentionally had some soft start power rails to reduce in-rush currents and found that some branded PCs would miss the detection of our adapter. This may not be the same issue with the rpi but the idea to test with a PC power supply is a good one. Furthermore, the CM4 is powered from a 12V power adapter (coax / barrel jack). You could use a common PC power supply to power the molex connector on the PCIe switch + apply the yellow (+12v) and black (ground) from the same PC power supply onto the coax connector. Be sure to check on the polarity of the coax connector and their through hole location on the CM4 PCB. Do not reverse wire the power supply!! A few more comments - the TI PCIe switch is a Gen1 (2.5Gbps) switch for the fanout. The CM4 is reported to be a Gen2 PCIe switch (5.0Gbps). If the 5.0 Gbps is confirmed then it will be more practical to install a Gen2 PCIe switch to allow for the downstream adapters to run faster than with the TI switch. In the meantime, can you share the output from the following (found from Jeff's logs):
The above should list all enumerated devices on the CM4 and should also display the PCIe switch if it is being enumerated. reference: |
|
The IO Board also has a floppy connector for power that can just be used with an adapter and an atx power supply, or just an older one that still has a floppy connector. Especially with older ones, but maybe also with newer ones, they need some kind of base load though on either the 5V or 12V rail to have a stable output and some just shut off without one. Either a power resistor or a turn indicator bulb work well as one. |
|
Hello. Do you always need to connect additional power to the board? Or only when the device needs more power? |
|
It is best to apply the additional power. Unless the vendor claims otherwise, the additional power supply is mandatory. Then check if the PCIe switch is being enumerated. Next, of course, check if your docked widget is being found through the PCIe switch. |
|
I know I'm kind of late to the party, but what version of raspbian did you do your initial testing on ? My freshly installed Rasbian 11 32-bit just just outputs segmentation fault when running lspci as a regular user, and ust freezes when same command run as root. the board I'm using is the Exacq Technologies 1 to 2 ports PCIe x1 adapter using the TI XIO3130 PCIe switch chip powered through the J20 powerheader. Update: Running now on rasbian 11 64-bir and I have now been able to get the PCIe switch recognized by the OS. I have tried to plug both a NVMe adapter and a Realtek Nic into it and only the NIC is recognized, The driver for the nic is loaded but no device is present in the OS, just eth0. Could this be a power issue as it is not stable. Sometimes it just freezes when issuing lspci |
|
Testing with the Raspberry Pi 5 and an NVMe drive...
For some reason I'm not seeing the NVMe drive. Going to try something else. |
|
My soundblaster card shows up behind the bridge, though: Hmm... maybe it's just one of the two ports on my riser is having an issue, because swapping the NVMe to the other side gets it to show up: Edit: Hmm, now it's showing fine in slot 1 too, lol. Maybe just bad luck! |
|
Seems stable now, but like on the CM4, it looks like the firmware only looks for root-level NVMe drives—at least right now. I'll ask RPi about whether they'll support deeper NVMe enumeration for boot. |
|
In case anybody's using one of these Pericom PI7C9X2G304 / PI7C9X2G404 PCIe switches in production and experiencing random kernel panics, there is a fix in the works over here (which I have personally validated on 5.15.92): |
|
@theodric - Wow, excellent news! And great sleuthing in that thread. I'm pretty sure all the issues I've seen on the Pi 5's bus have been due to clock problems with signal integrity, but now I'm also wondering if certain weird behaviors on the CM4 with other devices could be related too. |
|
We have seen similar idiosyncratic PCIe behaviour with the CM4 on our in-house designed carrier board for the Viota Pi-Cube and initially blamed signal integrity. The design employs an ASM1182e two port chip with downstream to a VL805 and an M.2 connector for NMVE SSDs. We found this chip (and the 4 port variant) to be the most reliable of the several bridge chip vendors that we tested on various PCIe adapter boards on the CM4IO board. The problem was a random failure to enumerate one or other or any of the downstream devices or a failure to enumerate the NVME device at 5GTs - not getting beyond 2.5GTs. We burned many hours trying to track this problem down including re-iterating our board design (to improve signal integrity) but then after one of the several CM4 eeprom firmware upgrade releases the problem seemed to disappear and enumeration of both downstream devices and at 5GTs became significantly more reliable. No other changes were made on the test rig other than the firmware upgrade so our conclusion was that whilst good enough timing at the hardware level is important, the enumeration process is also likely relevant, as in what order things get done and when. The code is closed source so it is only a best guess but IMHO worth knowing. It sounds like there may be some similar "timing" problems with the Pi5 and as mine has just arrived this morning it is high on the list to investigate. A Pi5 PCIe connector pinout would be helpful, so if anyone has worked that out it would be a big help to post it here - with the usual caveat emptor !! We plan on making some Pi5 PCIe to M.2 adapter boards and will make them available to the community when done. |
|
Dr George Smart has posted his reverse engineering hack of the Pi5 PCIe connector pinout on Github and YouTube - and saved me cannibalising my Pi5! Big thank you George - will take it as valuable input for an M.2 HAT and keep you posted. |
I bought this ATX Power Breakout Board 20Pin 24Pin: |
|
Regarding PCIe, there are a couple of Raspberry NVME M.2 bridges, like this Geekworm X1001 PCIe to M.2 NVMe Key-M SSD Shield for Raspberry Pi 5: So far, I only tried a M.2 to SATA bridge, specifically this Bewinner M.2 to SATA3.0: However, when attaching some HDD to it, powering the HDD via my PSU, I just get this: (I'm also writing down my experience on this here.) So next I will try the PCIe to M.2 bridge and then attach a SATA controller to the PCIe. I wonder if that works better. I see that all the solutions here are PCIe to USB3. I wonder how that compares to solutions which directly use the PCIe from the Pi board? What would be the possible bandwidth? I see this in But I don't exactly understand what it means. What is GT/s? So how can I get this capable 15.752 Gb/s? Gb = Gbit? The Radxa Penta SATA HAT (#615) seems like another nice solution. But I cannot easily/cheaply get it currently in Germany. |
and others
I need to add a new section to the site for bridges/switches/adapters... because I'm finding that at least one of the three I've purchased doesn't seem to work with the Pi at all, giving me
pcie: link is downon any attempt to use it with any PCIe device (including known good ones that work when I plug them in directly or through other adapters.Right now I'm testing these:
IO Crest 1 to 2 ports PCIe x1 adapter — seems to work fine at least for some cards. Build quality of included floppy to molex power adapter is pretty bad, though:
Exacq Technologies 1 to 2 ports PCIe x1 adapter using the TI XIO3130 PCIe switch chip, which uses a 4-pin molex connector for power. This one seems like it came out of a video server of some sort. Got it on eBay:
PCIe 16x to 1x adapter (generic) - Works with everything so far, as if I plugged it straight in. It should though, since it's just a dumb extension cable:
Rosewill RCRC-17001 Mining Card 1x to 16x - Sometimes works, sometimes causes a kernel panic for some reason. I even bought two revisions (
003, with a PCIe power connector, and888, with a Molex 4-pin power connector), and both of them exhibit this weird behavior:PCE164P-N06 VER008S PCIe 1x to 16x riser - Haven't tested yet, but I like how it has a plethora of power options!
One common annoyance is they all have different power adapter cables.
I should probably just go buy a bulky PC power supply and drop it on my desk.I did by a 600W fully modular PSU.The text was updated successfully, but these errors were encountered: