BLE small throughput

[kristiankajak]

The problem
Hi, Zephyr community,

I am experiencing some difficulties when trying to reach a high throughput with BLE communication. I am using DLE enabled HCI-UART on nrf52-pca10040 and DLE enabled HCI-USB on the nrf52840 USB dongle, both working with Linux hosts.

I am sending raw data on the ACL data channel, and so far I am able to reach ~550 Kbps ( ~3.5 ms for one packet), whereas I was hoping higher speeds considering I am using BT5 compatible devices.

Hope you have some ideas.

Connection Setup

1. DLE enabled with TX/RX payload size of 251B
2. Connection interval of 7.5ms
3. 8000 packets sent (total of 2MB), whereas a new packet is sent right after receiving an acknowledgment of the previous one
4. Max TX/RX time 1048 us
5. PHY 2M

Expected behavior

I expect more than 2 packets to be transmitted within one connection interval. Ideally, I am aiming for 5 TX packet transfers, but the Zephyr API does not really document the connection event length, so I am not sure what to expect.

Other interesting observations

1. Switching between PHY = 1Ms/s and 2Ms/s does not really make a difference. For 1Ms/s connection I have to increase the connection interval and the TX/RX times, but the resulting throughput is roughly the same.
2. I have tried increasing the connection interval with no positive effect, and the time per packet is still the same for various combos I have tried.

Environment

1. Host zephyr-master_fork #1: Linux Mint Tina 19.2
2. Controller zephyr-master_fork #1: Zephyr 1.14 HCI-UART with DLE on nrf52-pca10040
3. Host This is a test commit #2: Linux Mint Tina 19.2 on VM
4. Controller This is a test commit #2: Zephyr 1.14 HCI-USB with DLE on nrf52840 USB dongle

With regards,
Kristian

[carlescufi]

@kristiankajak the UART is limited to 1Mb/s for starters, so that might be an issue. There's also the fact that, depending on how your host sends data, it might not be fully utilizing the buffers available in the controller.

Some suggestions:

• Take a look at this configuration file
• Disable the CONFIG_BT_HCI_ACL_FLOW_CONTROL Kconfig option to see if that speeds things up (it shouldn't if the Host manages flow control correctly)

@joerchan @jhedberg @cvinayak additional suggestions welcome.

[carlescufi]

Also @kristiankajak could you please use the Nordic BLE sniffer to take a sniffer trace and attach it here?

[joerchan]

@kristiankajak the UART is limited to 1Mb/s for starters, so that might be an issue

Definitely an issue. Assuming you even get 100% utilization of the UART it is still limiting the maximum throughput of BLE. (1.4 Mb/s).

Make sure that you have enough buffers available and that you keep them full at all times. Since an empty buffer can end the connection event. Look at the configuration that carles linked, especially CTLR_TX/RX_BUFFER(_SIZE).

And then you can use the sniffer that carles linked to to see how many packets you get per event (look at the event counter). At 7.5 ms you should get around 5, increasing the connection interval should improve throughput as long as you manage to enqueue/dequeue fast enough from the buffers.

[kristiankajak]

Hi @carlescufi and @joerchan,
Thank you for the reply. I figured that indeed the UART speed will definitely become a limiting resource when using BT5 for max throughput, but I didn't expect the bottleneck to appear as early as it seems right now.

Some debug info:

1. My controller configuration follows a similar setup to what @carlescufi posted, whereas BT_CTLR_RX_BUFFERS = 10, BT_CTLR_TX_BUFFERS = 10, BT_CTLR_TX_BUFFER_SIZE = 251.

2. Disabling CONFIG_BT_HCI_ACL_FLOW_CONTROL does not make any difference.

3. I have attached the sniffer logs for both 1Ms/s and 2Ms/s PHY speeds. I do not have multiple nrf52 dev kits, so I had to implement the only one I had as the sniffer, and the communication, in this test case, is between 2x NRF52 USB dongles running the HCI USB. I haven't been able to implement DLE on the USB dongle, and thus, the payload size is 24 bytes. Maybe this helps in terms of figuring out any pipeline issues, though, as I'm executing the same code as I did before (with just modified packet size and count).

   In this test, I am sending 20 packets sequentially (starting at transaction No. 755), with payloads alternating between "01010101..." and "20202020...". For PHY=2M, it seems that 12 packets are delivered within one connection event with the smaller payload size. This number seems totally reasonable, as ideally, 0.5 ms is the theoretical duration for one TX/RX cycle.

   However, there seems to be a mismatch between what I receive on the remote end versus what the sniffer receives. From what I see with btmon on the other end, all packets get send successfully in the correct order, whereas the sniffer catches some arbitrary package (see transaction No. 779), which causes a slight stall. Furthermore, the packets following do not follow the same "0101.." and "2020..." order anymore.

   We do not see this behavior with the 1M implementation, where 9 packets are sent per interval, with the correct order and without any random packages in-between. Based on this info, it seems that I have utilized the connection events quite highly, and maybe indeed the HCI implementation starts being the limiting part. Hope you have some ideas.

BTW, thank you for introducing me to the sniffer tool. It's been helpful.

With regards,
Kristian

[joerchan]

I haven't been able to implement DLE on the USB dongle, and thus, the payload size is 24 bytes.

@kristiankajak You are only sending L2CAP 23 bytes, and Link Layer 27 bytes. Even though the Link Layer connection is able to transmit 251 bytes. You need to enqueue larger packets to get your throughput up. (Bluetooth does not support more than one l2cap packets inside a single LL packet).

However, there seems to be a mismatch between what I receive on the remote end versus what the sniffer receives

The sniffer will not be able to handle the full throughput. Because of the 1M/s baudrate of the UART. When this happens the sniffer will drop packets. This is probably what you are seeing.

[kristiankajak]

Hi @joerchan ,

As I explained, I wasn't able to use the DLE in the sniffer logs due to using a slightly different setup (I had to implement the sniffer on the one NRF52 dev kit I had) than I did when measuring the throughput.

The throughput of 550 Kbit/s was reached with LL payload size of 251, so this is not the problem.

[joerchan]

@kristiankajak I don't understand the setup.
From your description:
In the first one you had one nrf52 dev-kit and one nrf52840 USB dongle.
In the second one you had two nrf52840 USB dongle.

So if you don't get DLE to work on the USB dongle, then how can you get DLE with 251 bytes in your first setup, wont' you be limited by the USB dongle?

[kristiankajak]

@joerchan , you are correct about the setup.

In terms on the DLE enabling, sorry for being shy on the details, but I did this intentionally to keep the focus of this topic. But to explain, I have been experiencing difficulties sending 251B payload with 2x Dongle setup. DLE is enabled, but the LL payload size cannot be increased further than 68B, otherwise the data does not reach the remote end. I wrote about this problem in a post linked here: https://lists.zephyrproject.org/g/devel/topic/problem_with_sending_longer/36485664?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,36485664. This problem was fixed when switching one of the dongle controllers (client) with the Nrf52 dev kit using HCI-UART.

As for the setup used in this issue (NRF52 dev kit + Dongle), DLE works fine and I do actually receive 251B LL payloads on the remote end correctly.

[cvinayak]

@kristiankajak This could appear to be a limitation of the USB driver implementation? @anangl, I remember Segger host dropped UART bytes beyond 64 bytes unless MSD was disabled on MacOS, could it be similar on Nordic's CDC implementation?

[kristiankajak]

@cvinayak This seems to be the case yes. As for the HCI UART implementation with 251B payload, I'm going to, at some point, test this setup with an android device on the remote end. This maybe will give some insight into where the bottleneck is. I will keep you guys updated.

[carlescufi]

@kristiankajak make sure you have disabled the MSD as per the instructions here:
https://docs.zephyrproject.org/latest/guides/tools/nordic_segger.html#disabling-the-mass-storage-device-functionality

[aescolar]

It would seem this question is now answered, so closing. If you disagree please reopen