Add tests to verify lack of sync loss in RX

[tmiw]

Adds additional ctests to verify that FreeDV does not lose sync during RX.

Test Setup

• Test systems:
  • macOS: 2019 MacBook Pro (8 core 2.3 GHz Intel Core i9), 64 GB RAM, macOS 15.1
  • Linux: Raspberry Pi 5 (4 core ARM Cortex A76 @ 2.4 GHz), 8 GB RAM, latest Raspberry Pi OS
  • Windows:
    1. Lenovo IdeaPad Y410P (Intel Core i7-4700MQ, 8 GB RAM, Windows 10)
    2. Intel NUC (Intel Core i3-4010U, 8 GB RAM, Windows 10)
    3. Windows-10, Intel Core I7-4819MQ 2.8ghz 2801 4 Cores, 8 Logical, 32g of memory (Thanks Walter K5WH)
• "Radio" sound device: Griffin iMic USB sound card with 3.5mm cable between "Mic" and "Speaker" ports.
• "Analog" sound device:
  • macOS/Windows: Built-in speaker/mic on device
  • Linux: Generic USB sound device, no cables plugged in

Test Procedure

Linux/macOS

$ UT_ENABLE=1 ./build_linux.sh (or ./build_osx.sh)
$ cd build_[linux|osx]
$ for i in {1..10}; do FREEDV_COMPUTER_TO_RADIO_DEVICE="[radio device]" FREEDV_RADIO_TO_COMPUTER_DEVICE="[radio device]" FREEDV_COMPUTER_TO_SPEAKER_DEVICE="[analog device]" FREEDV_MICROPHONE_TO_COMPUTER_DEVICE="[analog device]" ctest -R fullduplex; done

Example execution from Ubuntu 22.04 aarch64 VM with above hardware:

for i in {1..10}; do FREEDV_COMPUTER_TO_RADIO_DEVICE="alsa_output.usb-Griffin_Technology__Inc_iMic_USB_audio_system-00.analog-stereo" FREEDV_RADIO_TO_COMPUTER_DEVICE="alsa_input.usb-Griffin_Technology__Inc_iMic_USB_audio_system-00.analog-stereo" FREEDV_COMPUTER_TO_SPEAKER_DEVICE="alsa_output.pci-0000_00_01.0.analog-stereo" FREEDV_MICROPHONE_TO_COMPUTER_DEVICE="alsa_input.pci-0000_00_01.0.analog-stereo" PYTHONPATH=$(pwd)/rade_src ctest -R fullduplex; done

(Optional) Load is added by running X number of yes >/dev/null & until top/Activity Monitor/other OS equivalent shows 75-80% CPU usage across all cores/threads before running the "With Load" tests.

Windows

Copy TestFreeDVFullDuplex.ps1 and freedv-ctest-fullduplex.conf.tmpl to the FreeDV install location (i.e. C:\Program Files\FreeDV 2.0.0...\bin), then open PowerShell and execute the following:

PS> cd 'C:\Program Files\FreeDV 2.0.0...\bin'
PS> Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser
PS> .\TestFreeDVFullDuplex.ps1 `
    -RadioToComputerDevice "Microphone (iMic USB audio system)" `
    -ComputerToSpeakerDevice "Speakers (Realtek High Definition Audio(SST))" `
    -MicrophoneToComputerDevice "Microphone Array (Realtek High Definition Audio(SST))" `
    -ComputerToRadioDevice "Speakers (iMic USB audio system)" `
    -NumberOfRuns 10 -ModeToTest RADE

(replace the various audio devices and mode to test as needed; the iMic devices above are the ones corresponding to the USB audio device with the loopback cable.)

Note that you may need to mute and unmute audio in Windows to allow it to be passed through to FreeDV prior to initially executing tests (Windows bug?)

(Optional) For load tests, see https://am2.co/2018/06/generating-artificial-cpu-load/ for a script that will generate the CPU load needed.

Success Criteria

ctest and the PowerShell script above start FreeDV in "full duplex" mode, causing TX and RX to occur at the same time. Because of the audio cable linking the Mic and Speaker ports on the USB audio device, FreeDV receives the signal that was just "transmitted". While this is happening, the tests look for "Sync changed" output on FreeDV's stderr and count the number of times this occurs. "Sync changed" occurs when freedv_get_sync() / rade_get_sync() changes from 0 to 1 (or vice versa). A "pass" is defined as this occurring only once, when FreeDV initially gets sync. Each scenario/mode must pass ten times in order for it to count as "passing".

Test Results

No Load (FreeDV only active task)

Platform	Commit tested	700D	700E	1600	RADE
macOS (PortAudio)	e21a3f0	10/10	10/10	10/10	10/10
Linux (pipewire)	e21a3f0	10/10	10/10	10/10	10/10
Windows 1 (PortAudio)	e21a3f0	10/10	10/10	10/10	10/10
Windows 2 (PortAudio)	e21a3f0	10/10	10/10	10/10	10/10
Windows 3 (PortAudio)	e21a3f0	10/10	10/10	10/10	10/10

Load Tests (Informational Only)

Platform	Commit tested	700D	700E	1600	RADE
macOS (PortAudio)	e21a3f0	10/10	10/10	10/10	10/10
Linux (pipewire)	e21a3f0	10/10	10/10	10/10	10/10
Windows 1 (PortAudio)	e21a3f0	10/10	10/10	10/10	5/10 (75% load) 10/10 (50% load)

[tmiw]

Need to figure out why snd-aloop isn't available in GH actions and then write the appropriate test script(s). Though it's possible this might only be a test that can be manually started (but 100% automated otherwise).

[tmiw]

Updated the test to check for and output any changes to the sync status once per second. This should probably happen more often, though. Would something like every 100ms be sufficient?

[tmiw]

@drowe67, OK, so I got RADE tests working on Windows in the GH action. (I know, not strictly necessary, but will be good once we start adding more test coverage.) Still waiting on Walter K5WH to give me results from his machine to update above. I'll ping him when I have a chance.

[drowe67]

@drowe67, OK, so I got RADE tests working on Windows in the GH action. (I know, not strictly necessary, but will be good once we start adding more test coverage.) Still waiting on Walter K5WH to give me results from his machine to update above. I'll ping him when I have a chance.

That's a neat trick - is there any correlation between the tests failing in the GH actions environment and tests failing with real HW? Just trying to make sure the GH actions tests can trap at least a subset of bugs in this area.

Yep results from a few typical machines/sound cards would be very useful.

[drowe67]

@tmiw - did the tests in this PR lead you to any improvements in the sound code, or was it all OK and passed the tests without modification?

[tmiw]

That's a neat trick - is there any correlation between the tests failing in the GH actions environment and tests failing with real HW? Just trying to make sure the GH actions tests can trap at least a subset of bugs in this area.

I've found some correlation on the Linux/PulseAudio side, at least. I also found some error handling bugs that will be handled separately: #772

@tmiw - did the tests in this PR lead you to any improvements in the sound code, or was it all OK and passed the tests without modification?

I ended up fixing some issues with the PulseAudio side. Otherwise I think things mostly matched up.

[tmiw]

@drowe67, just added the results I got from Walter to the top of the PR. Looks like we might be good?

[drowe67]

@drowe67, just added the results I got from Walter to the top of the PR. Looks like we might be good?

Well done @tmiw (and thanks Walter). I appreciate all your hard work on this one 👍

[tmiw]

@drowe67, just added the results I got from Walter to the top of the PR. Looks like we might be good?

Well done @tmiw (and thanks Walter). I appreciate all your hard work on this one 👍

No worries. OK to merge now?