50 MHz SDIO bus not working for some SD cards #23

pdp7 · 2021-04-28T21:57:32Z

reading from NFS vs SDcard:

NFS: 22.8 MB/s (the same on my dev box gets 170 MB/s via 10 GbE)
SDCard: 4.6 MB/s (and it's a highend card which will get > 50 MB/s normally).
So, for now and with production hardware, NFS > SDcard. I haven’t tried USB.

FWIW I tested both using dd bs=8192 status=progress,
e.g. sudo dd status=progress bs=8192 if=/dev/mmcblk1 of=/dev/null

@tekkamanninja @MichaelZhuxx: The Seeed FAQ shows 22 MB/s with SD card. I presume this means it was Linux 4.19 that Seeed tested with? Maybe there is a problem with 5.10 on the Fedora image that is causing it to be 4.6 MB/s?

The text was updated successfully, but these errors were encountered:

robertlipe · 2021-04-28T23:52:57Z

Is it possible this is a manifestation of #20 ?

tommythorn · 2021-04-29T00:40:34Z

I see no traffic in dmesg so I don’t think so

tgeek77 · 2021-05-04T12:55:16Z

I'm getting the same performance. I tested the microsd card on another machine and for 25-30 MB/s but on the BeagleV, I am getting the same performance as @tommythorn.

MichaelZhuxx · 2021-05-23T15:27:03Z

after update the frequency in dts as below:

--- a/arch/riscv/dts/starfive_vic7100_beagle_v.dts
+++ b/arch/riscv/dts/starfive_vic7100_beagle_v.dts
@@ -556,7 +556,7 @@
                        clocks = <&dwmmc_biuclk>;
                        clock-names = "biu";
                        clock-frequency = <100000000>;
-                       max-frequency = <10000000>;
+                       max-frequency = <50000000>;
                        fifo-depth = <32>;
                        card-detect-delay = <300>;
                        fifo-watermark-aligned;

the system will pop up the below message

[  911.725040] audit: type=1130 audit(1607040900.747:798): pid=1 uid=0 auid=4294967295 ses=429496729'
[  911.816845] audit: type=1131 audit(1607040900.747:799): pid=1 uid=0 auid=4294967295 ses=429496729'
[  920.872770] audit: type=1131 audit(1607040909.887:800): pid=1 uid=0 auid=4294967295 ses=429496729'
[  921.126239] audit: type=1130 audit(1607040910.147:801): pid=1 uid=0 auid=4294967295 ses=429496729'
[  921.177968] audit: type=1131 audit(1607040910.147:802): pid=1 uid=0 auid=4294967295 ses=429496729'
[  921.230732] audit: type=1130 audit(1607040910.237:803): pid=1 uid=0 auid=4294967295 ses=429496729'
[  921.281850] audit: type=1701 audit(1607040910.297:804): auid=4294967295 uid=0 gid=0 ses=4294967291
[  921.404099] audit: type=1334 audit(1607040910.417:805): prog-id=149 op=LOAD
[  921.444058] audit: type=1334 audit(1607040910.417:806): prog-id=150 op=LOAD
[  921.484401] audit: type=1334 audit(1607040910.417:807): prog-id=151 op=LOAD
[  921.524043] audit: type=1325 audit(1607040910.437:808): table=filter family=7 entries=0 op=xt_reg"
[  921.568324] audit: type=1325 audit(1607040910.437:808): table=broute family=7 entries=0 op=xt_reg"
[  946.284639] kauditd_printk_skb: 9 callbacks suppressed

pdp7 · 2021-05-23T20:18:26Z

@tekkamanninja @esmil @geertu interesting results from @MichaelZhuxx when changing the SD card frequency

MichaelZhuxx · 2021-05-24T08:53:42Z

sorry for missing test data

[root@fedora-starfive ~]# time dd if=/dev/mmcblk1p2 of=/dev/null bs=512k count=5000
5000+0 records in
5000+0 records out
2621440000 bytes (2.6 GB, 2.4 GiB) copied, 110.666 s, 23.7 MB/s

real    1m50.673s
user    0m0.033s
sys     0m15.706s

esmil · 2021-05-24T09:47:29Z

That's great. I see similar numbers after raising the max frequency to 50MHz.

Michael Zhu found that raising the max frequency on the SD Card controller to 50MHz seems to work and improve performance: starfive-tech/Fedora_on_StarFive#23 Signed-off-by: Emil Renner Berthing <kernel@esmil.dk>

esmil · 2021-05-24T22:13:39Z

While it seems to work for both me and @mcd500, @teknoraver has issues with raising the max frequency to 50MHz:
https://pastebin.com/vTHDjrKF

kraj · 2021-05-25T03:17:54Z

on yocto with 5.13-rc3

root@beaglev-starlight-jh7100:~# dd if=/dev/urandom of=rnd_data bs=512k count=1000
1000+0 records in
1000+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 23.6062 s, 22.2 MB/s

Michael Zhu found that raising the max frequency on the SD Card controller to 50MHz seems to work and improve performance: starfive-tech/Fedora_on_StarFive#23 Signed-off-by: Emil Renner Berthing <kernel@esmil.dk>

tommythorn · 2021-05-26T00:39:27Z

Confirm that I also now get 22+ MB/s with Akira-san's image ( Fedora-riscv64-vic7100-xfce-dev-Rawhide-20210516233526.n.0-sda-akira-5.13.rc3.raw.zst)

pdp7 · 2021-05-27T01:48:06Z

@MichaelZhuxx @tekkamanninja it appears @teknoraver had trouble when increasing the frequency so maybe there is variance between different boards? #23 (comment)

@esmil is that why you are keeping that frequency change out of https://github.com/starfive-tech/linux/tree/esmil_starlight ?

teknoraver · 2021-05-27T10:28:04Z

Yes, that change broke sdcard for me. It boots, and then the card start making errors until it seems removed:

[   26.648179] mmc0: card 1234 removed
[   26.735655] mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 400000Hz, actual 400000HZ div = 125)
[   26.794686] mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 50000000Hz, actual 50000000HZ div = 1)
[   26.804597] mmc0: error -84 whilst initialising SD card
[   26.823688] mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 300000Hz, actual 299401HZ div = 167)
[   26.887392] mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 50000000Hz, actual 50000000HZ div = 1)
[   26.897305] mmc0: error -84 whilst initialising SD card
[   26.916359] mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 200000Hz, actual 200000HZ div = 250)
[   26.989895] mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 50000000Hz, actual 50000000HZ div = 1)
[   26.999785] mmc0: new high speed SDHC card at address 1234
[   27.007687] mmcblk0: mmc0:1234 SA16G 14.4 GiB 
[   27.017953]  mmcblk0: p1 p2 p3
[   27.021757] debugfs: Directory 'mmcblk0' with parent 'block' already present!
[   28.728211] mmc0: card 1234 removed

esmil · 2021-05-27T12:36:46Z

@pdp7 Yes, that's exactly why. After you found the missing broken-cd; property broke sdcard detection for you others have also run into it, so when @teknoraver found it didn't work for him I didn't want to break the starlight branch for others too.

It'd be cool if we could somehow figure out if it's just @teknoraver's board/card. I guess one way to do that would be to just re-apply the change and wait for more people to complain, but hopefully we can find a nicer solution.

Michael Zhu found that raising the max frequency on the SD Card controller to 50MHz seems to work and improve performance: starfive-tech/Fedora_on_StarFive#23 Signed-off-by: Emil Renner Berthing <kernel@esmil.dk>

MichaelZhuxx · 2021-06-03T09:48:54Z

Beta developers who are having problems with SD card read/write, here is a suggestion that please add c_macro "_SET_SYSCON_REG_register50_SCFG_sdio0_cclk_dlychain_sel(v)" in INIT_FUNC_DEF(sdio0) to adjust delay shift between clk and data signal. the value of v is 0~31 which means the delay_shift(sel) = 0.16 * sel (ns). please have a try and sort out a appropriate value. thanks @mcd500, @teknoraver @pdp7

pdp7 · 2021-06-11T05:10:23Z

@teknoraver could you please try the fw_payload.bin.out that @mcd500 has linked in this post?
https://forum.beagleboard.org/t/new-firmware-boot-hang-is-fixed/29986/3

We think it may improve the problems that some people have been having with SD card stability

teknoraver · 2021-06-11T17:12:18Z

I can, but I didn't have stability with the SD card, except with the high frequency patch

teknoraver · 2021-06-11T17:35:25Z

ANyway, with the new fw and the patch, I still get this:

[   61.209784] mmc0: card 1234 removed
[   61.355262] mmc0: new high speed SDHC card at address 1234
[   61.363387] mmcblk0: mmc0:1234 SA16G 14.4 GiB 
[   62.060444]  mmcblk0: p1 p2 p3
[   62.064197] debugfs: Directory 'mmcblk0' with parent 'block' already present!
[   63.769728] mmc0: card 1234 removed
[   63.865560] mmc0: new high speed SDHC card at address 1234
[   63.877439] mmcblk0: mmc0:1234 SA16G 14.4 GiB 
[   64.549542] F2FS-fs (mmcblk0p2): Issue discard(479060, 479060, 1) failed, ret: -5
[   64.549567] F2FS-fs (mmcblk0p2): Issue discard(479064, 479064, 1) failed, ret: -5
[   64.580425]  mmcblk0: p1 p2 p3
[   64.591660] debugfs: Directory 'mmcblk0' with parent 'block' already present!
[   65.609730] mmc0: card 1234 removed
[   65.712979] mmc0: new high speed SDHC card at address 1234
[   65.721099] mmcblk0: mmc0:1234 SA16G 14.4 GiB 
[   66.420438]  mmcblk0: p1 p2 p3
[   66.424161] debugfs: Directory 'mmcblk0' with parent 'block' already present!
[   67.449692] mmc0: card 1234 removed
[   67.562955] mmc0: new high speed SDHC card at address 1234
[   67.571207] mmcblk0: mmc0:1234 SA16G 14.4 GiB 
[   68.270393]  mmcblk0: p1 p2 p3
[   68.274075] debugfs: Directory 'mmcblk0' with parent 'block' already present!
[   69.289689] mmc0: card 1234 removed
[   69.392869] mmc0: new high speed SDHC card at address 1234
[   69.400307] mmcblk0: mmc0:1234 SA16G 14.4 GiB 
[   70.100390]  mmcblk0: p1 p2 p3
[   70.104064] debugfs: Directory 'mmcblk0' with parent 'block' already present!

esmil · 2021-06-11T17:53:09Z

@teknoraver Which patch is that exactly? That looks like mmc0 is probed twice..

teknoraver · 2021-06-11T23:05:16Z

@esmil well, when I get stability issues the card seems being inserted and removed continuously, even it it's always inserted,
so I see those probe messages

pdp7 · 2021-06-11T23:18:03Z

@MichaelZhuxx any suggestions given what Matteo Croce (@teknoraver) is seeing in that log?

pdp7 · 2021-07-01T02:28:52Z

@teknoraver do you still seem performance issues with the newer SD cards that you have? I am trying to determine if there is anyone for which 50 MHz does not.

mcd500 · 2021-07-01T08:47:39Z

I finally had a time to investigate the sd card.
Without 50Mhz patch, it works, just not at the optimal speed.
With the 50Mhz patch, good speed but found two sd cards fail to boot.

The log for working SD

[root@fedora-starfive ~]# dmesg | grep mmc1
[    2.750257] mmc_host mmc1: card is non-removable.
[    2.827147] mmc_host mmc1: Bus speed (slot 0) = 100000000Hz (slot req 400000Hz, actual)
[    2.926297] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
[    3.033383] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    3.280961] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    3.379680] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[    3.446292] mmc1: queuing unknown CIS tuple 0x81 (9 bytes)
[    3.572092] mmc_host mmc1: Bus speed (slot 0) = 100000000Hz (slot req 10000000Hz, actu)
[    3.611552] mmc1: new high speed SDIO card at address 0001
[root@fedora-starfive ~]# dmesg | grep mmc0
[    2.591564] mmc_host mmc0: card is polling.
[    2.663562] mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 400000Hz, actual)
[    4.009235] mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 50000000Hz, actu)
[    4.080417] mmc0: new high speed SDHC card at address 0001
[    4.114898] mmcblk0: mmc0:0001 00000 14.9 GiB

The log for not working SD

mmc_host mmc0: card is polling.
mmc_host mmc1: card is non-removable.
mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 400000Hz, actual 400000HZ div =)
mmc_host mmc1: Bus speed (slot 0) = 100000000Hz (slot req 400000Hz, actual 400000HZ div =)
mmc_host mmc0: Bus speed (slot 0) = 100000000Hz (slot req 50000000Hz, actual 50000000HZ d)
mmc0: new high speed SDHC card at address b368
mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
mmcblk0: mmc0:b368       7.46 GiB
mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
systemd[1]: Detected architecture riscv64.
 mmcblk0: p1 p2
mmc_host mmc1: Bus speed (slot 0) = 100000000Hz (slot req 10000000Hz, actual 10000000HZ d)
mmc1: new high speed SDHC card at address 0001

The line in not working SD is suspicious.

mmc0: new high speed SDHC card at address b368

I will look into it deeper.

teknoraver · 2021-07-01T10:43:55Z

Indeed, some cards are failing.
Can you try with 25 Mhz? That worked for all cards for me

mcd500 · 2021-07-01T11:44:50Z

I will try 25MHz. The most difficulty was to find the sd cards which fails, only two out of all others I have.

pdp7 · 2021-07-07T17:59:00Z

@teknoraver @mcd500 any updates? We still have some cards that can not hit 50MHz right?

teknoraver · 2021-07-07T18:30:54Z

Yes, at least two models

pdp7 · 2021-07-15T02:36:11Z

I finally had a time to investigate the sd card.
Without 50Mhz patch, it works, just not at the optimal speed.
With the 50Mhz patch, good speed but found two sd cards fail to boot.

@mcd500 what manufacturer and model are those SD cards?

@MichaelZhuxx would like to try to reproduce does not have a SD card that fails at 50 MHz

teknoraver · 2021-07-15T10:54:28Z

@mcd500 what manufacturer and model are those SD cards?

For me, a Kingston 16 GB card, labeled SA16G

MichaelZhuxx · 2021-07-19T11:10:17Z

@mcd500 what manufacturer and model are those SD cards?

For me, a Kingston 16 GB card, labeled SA16G

could you please take a picture for this TF card, I can buy the same one for testing

teknoraver · 2021-07-19T11:36:50Z

We were collecting picture of the cards, but I don't remember where.
@pdp7 do you have the URL?

MichaelZhuxx · 2021-07-19T11:52:17Z

We were collecting picture of the cards, but I don't remember where.
@pdp7 do you have the URL?

do you mean this link?https://forum.beagleboard.org/t/known-good-sd-cards/29958

teknoraver · 2021-07-19T12:29:16Z

Yes, that's my card:

https://forum.beagleboard.org/uploads/default/original/2X/f/f8e0154b11fe6546d863902e5c4025b00f7cbabc.jpeg

MichaelZhuxx · 2021-07-19T15:27:51Z

Yes, that's my card:

https://forum.beagleboard.org/uploads/default/original/2X/f/f8e0154b11fe6546d863902e5c4025b00f7cbabc.jpeg

This TF card is no longer available in the market. we have verified three type of TF card and no issue happened.

MichaelZhuxx · 2021-07-22T01:32:49Z

Update the progress:
I got several type of TF card, these cards are working well, such as SanDisk, Lexar, SUMSUNG

but unfortunately, below cards(I guess it's only avaliable on China market) has a problem on booting, I need do a deep dive on that

MichaelZhuxx · 2021-07-27T11:25:34Z

Update:

I add "_SET_SYSCON_REG_register50_SCFG_sdio0_cclk_dlychain_sel(12)" in INIT_FUNC_DEF(sdio0) for u-boot and run it with 50MHz patch. All above cards(SanDisk, SUMSUNG, TOSHIBA, KINGSTON, Lexar, aigo, etc.) are working well.

@teknoraver please download the u-boot and dtb binary and try it. looking forward to hearing from your test result.

Beta developers who are having problems with SD card read/write, here is a suggestion that please add c_macro "_SET_SYSCON_REG_register50_SCFG_sdio0_cclk_dlychain_sel(v)" in INIT_FUNC_DEF(sdio0) to adjust delay shift between clk and data signal. the value of v is 0~31 which means the delay_shift(sel) = 0.16 * sel (ns). please have a try and sort out a appropriate value. thanks @mcd500, @teknoraver @pdp7

Michael Zhu found that raising the max frequency on the SD Card controller to 50MHz seems to work and improve performance: starfive-tech/Fedora_on_StarFive#23 Signed-off-by: Emil Renner Berthing <kernel@esmil.dk>

MichaelZhuxx · 2021-08-23T13:41:28Z

Verified in new Fedora Image Fedora-riscv64-developer-xfce-with-esp-Rawhide-20210822.n.0-sda.raw.zst, and test result is as below shown:

[root@fedora-starfive ~]# time dd if=/dev/mmcblk0 of=/dev/null bs=512k count=10000
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 147.18 s, 35.6 MB/s
real 2m27.188s
user 0m0.243s
sys 0m26.006s

tgeek77 · 2021-08-23T15:35:07Z

Verified in new Fedora Image Fedora-riscv64-developer-xfce-with-esp-Rawhide-20210822.n.0-sda.raw.zst, and test result is as below shown:

That URL doesn't work.

esmil · 2021-08-23T18:36:29Z

It seems the day was bumped. See https://fedora.starfivetech.com/pub/downloads/BeagleV-release/

MichaelZhuxx · 2021-08-24T12:13:05Z

@esmil @tgeek77 sorry for the inconvenience, the link is ok now

pdp7 assigned pdp7 and tekkamanninja and unassigned tekkamanninja and pdp7 May 18, 2021

pdp7 added the kernel label May 18, 2021

pdp7 assigned MichaelZhuxx May 18, 2021

pdp7 added the performance label May 18, 2021

pdp7 changed the title ~~SD card slow in kernel 5.10~~ 50 MHz SDIO bus not working for some SD cards Jul 7, 2021

MichaelZhuxx assigned MichaelZhuxx and unassigned tekkamanninja and MichaelZhuxx Jul 19, 2021

MichaelZhuxx closed this as completed Aug 23, 2021

50 MHz SDIO bus not working for some SD cards #23

50 MHz SDIO bus not working for some SD cards #23

Comments

pdp7 commented Apr 28, 2021 • edited Loading

robertlipe commented Apr 28, 2021

tommythorn commented Apr 29, 2021

tgeek77 commented May 4, 2021

MichaelZhuxx commented May 23, 2021

pdp7 commented May 23, 2021 • edited Loading

MichaelZhuxx commented May 24, 2021

esmil commented May 24, 2021

esmil commented May 24, 2021

kraj commented May 25, 2021

tommythorn commented May 26, 2021

pdp7 commented May 27, 2021

teknoraver commented May 27, 2021

esmil commented May 27, 2021 • edited Loading

MichaelZhuxx commented Jun 3, 2021 • edited Loading

pdp7 commented Jun 11, 2021

teknoraver commented Jun 11, 2021

teknoraver commented Jun 11, 2021

esmil commented Jun 11, 2021 • edited Loading

teknoraver commented Jun 11, 2021

pdp7 commented Jun 11, 2021

pdp7 commented Jul 1, 2021

mcd500 commented Jul 1, 2021

teknoraver commented Jul 1, 2021

mcd500 commented Jul 1, 2021

pdp7 commented Jul 7, 2021

teknoraver commented Jul 7, 2021

pdp7 commented Jul 15, 2021

teknoraver commented Jul 15, 2021

MichaelZhuxx commented Jul 19, 2021

teknoraver commented Jul 19, 2021

MichaelZhuxx commented Jul 19, 2021

teknoraver commented Jul 19, 2021

MichaelZhuxx commented Jul 19, 2021

MichaelZhuxx commented Jul 22, 2021

MichaelZhuxx commented Jul 27, 2021

MichaelZhuxx commented Aug 23, 2021 • edited Loading

tgeek77 commented Aug 23, 2021

esmil commented Aug 23, 2021

MichaelZhuxx commented Aug 24, 2021

pdp7 commented Apr 28, 2021 •

edited

Loading

pdp7 commented May 23, 2021 •

edited

Loading

esmil commented May 27, 2021 •

edited

Loading

MichaelZhuxx commented Jun 3, 2021 •

edited

Loading

esmil commented Jun 11, 2021 •

edited

Loading

MichaelZhuxx commented Aug 23, 2021 •

edited

Loading