Unit Testing w/ Devices

[bsiever]

Hi,

As I've been experimenting with (and enhancing) bleak I found testing to be bit cumbersome. I thought I'd:

• Ask other developers how you're testing your work --- Any info to share?
• Explain how I'm doing my testing and enhance/contribute my work if there's sufficient interest.

As for me: I'm mainly focused on the macOS backend and haven't been testing my work on other platforms. I think my current approach should be portable with a little effort. Here's what I'm doing/developing:

1. I'm using BBC Micro:Bits
   • They're pretty cheap test devices (Amazon Link : ~$18 US each).
   • They're easy to program with custom BLE services and advertising. I've been using PlatformIO's micro:bit support (with some modifications). The resulting firmware is a single file. They enumerate as a USB drive and firmware can be changed by simply copying a firmware file to the microbit drive.
2. I've created a custom service with a lot of BLE features that I wanted to test. (Here: https://github.com/bsiever/microbit-ble-testservice)
   • I'll be creating a few variations with other things to test (different advertising, services with "duplicate" UUIDs, etc). My test fixtures updates the firmware needed for the set of tests.
3. I've working on a variety of unit tests using pytest and alt-pytest-asyncio.
   • Current tests and testing approaches are still a bit rough, but it's a promising start. Initial tests are here: https://github.com/bsiever/bleak/tree/scan-filter/tests/microbit1 .
   • As of this moment I've got about 26 candidate tests and plan to do several more. So far I've been focused on reads/writes (characteristics: authorized, with response, without, unauthorized/failed; and descriptors), connects/disconnects, and notifies/indicates.
   • Next steps include:
     • Multiple simultaneous clients
     • Python behavioral stuff & context management
     • Scanning & Filtering
     • Lots of cleanup / improvements. These are all poorly structured proof-of-concept tests.

Slight tangent: As a result of the testing I've made a lot of changes to the macOS backend. Unfortunately I've been building off the release version instead of keeping up with the devel branch. I'll try to do a PR at some point...but it may be messy and merit rejection.

[hbldh]

This is definitely interesting, even though I can never use any hardware-in-the-loop testing on the CI systems. I test with a TI Sensortag on all platforms prior to releases. The test coverage is not 100% and the situation is pertty far from ideal, but I have set the bar pretty low to be able to cope with this project. Writing rigid testing for bleak is not my forte nor anything that piques my interest especially much I am afraid...

I will be glad to incorporate any PRs you have, but make sure that you mark them with e.g @pytest.mark.skipif and only run these tests if e.g. an enviroment flag BLEAK_HIL_ENABLED is set.

Thank you for doing this work though! I do think it is important, but I have no desire to do these chores myself...

[bsiever]

It's not my primary interest either, but it seems like a way to avoid some regression and improve consistency across platforms. My current tests are a bit messy and depend on the current API.
I'll probably try to improve my tests in conjunction with any updates regarding handling of #139 (updating API to handle duplicate values) and then do a PR.

[Ladvien]

Hello @bsiever,

I'd love to help out on the device testing front. I think I'll start by getting familiar with your MicroBit tests and then look at replicating them with the Arduino Nano 33 and ESP32.

[bsiever]

Hi @Ladvien ,

Any help would be appreciated. Quick question: Is there any major benefit to having the same tests on multiple pieces of hardware? I'm not stuck on the micro:bit, but it's cheap, readily available, and can be programmed by flash....so easy to integrate into unit tests (and I had a lot lying around...others probably have nanos & ESPs lying around instead).

[Ladvien]

Good question, @bsiever.

I think I was considering how different chipsets might introduce edge-cases; having several pieces of test hardware would help isolate bleak issues from hardware. That stated, thinking it through, it might make more sense to focus on one or two, rather than several. I'll defer to your opinion as I'm a hack.

[bsiever]

@Ladvien I really don't have a strong opinion.....Bleak runs at the upper level of the BLE stack and hopefully isn't too swayed by the underlying implementation (like MTU, etc.). My micro:bit service relies on an older, non-maintained code base...so there's certainly some motivation for something newer like ESP32.

[bsiever]

@Ladvien BTW, the primary service used for my tests is documented here: https://github.com/bsiever/microbit-ble-testservice/blob/master/README.md . It contains a lot of different attribute/characteristic permissions and behaviors. In hindsight some are not ideal for testing, like using periodic notifications/indications. My current tests using those are a little too fragile and may sometimes fail due to timing of the tests.

I also have two firmware versions that both include this service but advertise in different ways.

[Ladvien]

Thank you, @bsiever. I'm working on getting your testing environment setup on my MicroBit. My goal is to learn what I'm doing, as I'm a data engineer by day. Then, write firmware for the ESP32 using μPython to parallel your MicroBit firmware.

I saw your .edu email--can I shoot you specific questions there if I they come up?

[Ladvien]

Well, a mixed success on Linux.

I added a bit of code to make it compatible with Linux and to warn users about testing dependencies, but for the most part, it worked out of the box. Thanks, @bsiever!

I'd shoot you a PR, but I'm actually working off the https://github.com/workaroundgmbh/bleak/tree/proglove fork and branch, as it feels a bit more stable on Linux.

Hmm, now that I've got your unit tests running, I guess I can systematically figure out which is more stable, rather than go off "feels." Unit testing FTW! 🎉

One thought, may I write code to check if the firmware is already installed? This would allow bypassing firmware installation unless needed. My strategy would be using pyserial or pyserial-asyncio to check for your firmware's handshake. Should make running tests faster, I think.

[bsiever]

@Ladvien Wow --- that's much better than I expected. It's possible that the failures may be due to either the tests or unimplemented features in bleak. If you make changes to the files in the microbit directory, please do a PR and I'll merge them in.

As for contacting me directly: I'm on summer break and my response to my .edu email is spotty, but you're welcome to contact me at gmail (same username).

Unfortunately I don't think there's an easy way to determine which firmware is on the micro:bit. There's a "DETAILS.TXT" file that lists the underlying bootloader firmware, but the profile is application-level --- DETAILS.TXT doesn't change.

My test fixtures look for a --nofw flag to bypass the firmware update. I used it a lot when re-running specific tests.

[bsiever]

@Ladvien Whoops...Should have read more carefully. I'll try to include your updates to the fixtures in my repo (no need for a PR).

I'm open to the serial request....but it'll probably need some modification of the firmware to actually dump out the firmware or allow it to be interrogated. (Not a hard change, but we'd need to agree on a protocol)

[hbldh]

@bsiever @Ladvien You have convinced me of the benefits of trying to bring this into Bleak, to at least give me a decent testing solution for me before releases.
I have bought a Microbit and will try to set up your testing solution on it. Will update here when I have had time to make progress. My goal is to be able have a minimal test bench for a 0.7.X release sometime during July.

[Ladvien]

Thanks, @hbldh! I meant to offer to send you a Microbit. Thanks again for all the hard work; definitely a bleak fanboy.

--Thomas

[bsiever]

@hbldh Glad to hear it. I hope it helps. BTW, some of my tests of filtering require 2 micro:bits --- If you're in the US I can send one. If you're elsewhere I can chip in to support buying another (Venmo?).

I developed it partly to help cope with my own accidental regressions. I also knew that my PR (#209) was dumping a lot on you and tests would help verify that things weren't horribly broken.

I'm happy to make changes or add some new bluetooth services if needed. This is a starting point, but I'll let the general consensus decide on whether this is a platform/approach to stick with.

[hbldh]

I will start with the 1 micro:bit tests then. I will get back to you when and if another one is required!

[AGlass0fMilk]

Coming from the other side of things -- I'm looking for a good library with which to interact with custom BLE devices from a development laptop and bleak seems like the ideal candidate.

I primarily contribute to Mbed-OS (embedded RTOS) and work with their BLE stack quite a bit. They recently released some python tooling that may be of interest to bleak for their testing framework.

Essentially, it works like this:

• The Mbed target board (eg: nRF52840DK) runs a remote-procedure-call (RPC) server application (ble-cliapp). This allows a user to perform arbitrary BLE actions on the Mbed hardware, including create advertisement packets, connecting (central/peripheral), pairing, setting up a GattServer and so on.
• The corresponding python library creates a more "pythonic" interface that can be used to programmatically control one or several connected boards running the BLE RPC server. Results and data are returned in JSON format so it's easy to work with.

This is all facilitated through a standard virtual serial port (USB to UART).

Another nice feature is that Mbed-OS works on a number of different hardware targets and so the ble-cliapp could simply be recompiled for a different target that also supports BLE. This eliminates the need to maintain separate unit testing frameworks for different hardware as discussed above.

This whole thing was developed to be able to do integration testing on real embedded hardware. It looks like it could be useful for testing a desktop BLE library as well 😄

I just encountered this issue while reviewing bleak and figured I would comment...

If you're interested in learning more, let me know!