These are client tools used to connect to the CTS infrastructure and receive RF data (IQ samples) over IP.
Forked from original + adapted to Python3. Tested with Python 3.9, GNU Radio 3.8.2 / Ubuntu 21.04
- RF Signals Over IP
- Tuning to a Frequency == Connecting to a Port
- Getting Started
- Start Receiving
- Receive with SDR Tools
- Building
- Use Your Own Tools
- Troubleshooting
- Hacker's Delight: Automatic Sample Rate
- Contact Us
In CTS contests radio signals are streamed from the challenge server over IP (RFoIP), and not over the air with antennae. The use of IP-based transports eliminates the complexity of deploying actual radios and transmitting RF over the air, not to mention per-country regulations about radio transmissions.
In particular, the "raw" RF signals are tunneled via ZeroMQ and can be received with traditional RF tools, as if they were coming from a software-defined radio (SDR). GNURadio, for example, supports ZeroMQ source blocks natively to receive signals as if they were coming from a regular source block. In this way, the entry barrier is fairly low and the players could focus on the real challenge: Decoding the signal. For the same reason, for some editions of the contest we provide VMs or Docker images to support the GNURadio toolchain, as setting it up can be a time consuming process and is not relevant to the core of the challenge. Of course, participants are welcome in using their own setup.
You won't need to develop an IP-to-RF translation in order to start receiving signals: we prepared a tool that will allow you to receive signals with common RF-analysis software. Many tools, like GNURadio, GQRX and Osmocom-based utilities (e.g., osmocom_fft) work natively with this streaming FIFO system. Some tools like URH and Baudline lack support for Linux FIFOs and can be used by first copying streaming data from the RX FIFO to a standard file and then operating on the standard file. Generally, the system was designed to operate as if you were streaming IQ samples from an actual SDR.
All you have to do is to choose the frequency you want to tune to, and the RFoIP server will send you the right metadata to adjust the sample rate automatically (see below). We'll provide you the frequency of the first signal to kickstart the game.
$ cd cts-cli/
$ python rx_to_fifo.py --help
Tags to Var imported
FIFO already exists (not creating): /path/to/cts.fifo
Usage: rx_to_fifo.py: [options]
Options:
-h, --help show this help message and exit
--rx-frequency=RX_FREQUENCY
Set RX Frequency [default=900000000]
--server-ip=SERVER_IP
Set Server IP [default=127.0.0.1]
--server-port-base=SERVER_PORT_BASE
Set Server Port Base [default=10000]
--throttle=THROTTLE Set Throttle [default=1]
--zmq-rx-timeout=ZMQ_RX_TIMEOUT
Set ZMQ RX Timeout [default=100]When using rx_to_fifo.py, frequency tuning is performed automatically (by connecting to the RFoIP server on the right port) and any downstream tool or process that has settings for tuning will have no effect on the delivered signal.
For instance, when GQRX is configured to read data from the FIFO created by rx_to_fifo, the tuning controls inside GQRX will have NO EFFECT as the data streaming to it is generated by our server and controlled by rx_to_fifo. To adjust tuning frequency, exit GQRX, exit rx_to_fifo and begin a new stream by supplying new tuning parameters to rx_to_fifo.
$ cd cts-cli/
$ python rx_to_fifo.py \
--server-ip=<this is given by the organizers> \
--rx-frequency=<each challenge has its own frequency>
Tags to Var imported
Tags to Var initialized
Created FIFO: /path/to/cts.fifo
IMPORTANT: Before you can see the source sample rate, you must start consuming samples from the FIFO
...Check that you can receive at least one sample:
$ cat /path/to/cts.fifo > /dev/nullAnd you should now see the original signal's sample rate reported on the standard output:
...continued from above...
rx_rate = 128000.0IMPORTANT: Each signal has it's own sample rate! Always check the reported sample rate before you start working on a signal. You can kill and restart the rx_to_fifo.py script as many times as you need (especially if it appears dead).
During some contests we stream a basic test signal to verify that all contestants are receiving correctly. To test your system:
$ python rx_test.py --server-ip=<given by the contest organizers> \
--rx-frequency=<the test frequency given>You should receive this signal:
To summarize, if you want to tune to 434.7MHz you must:
- run
rx_to_fifo.py --rx-frequency 434700000 --server-ip=<IP>, - point your standard receiver tool (e.g., GQRX) to the FIFO file,
- use the receiver tool (e.g., GQRX) to tune to 434.7MHz
To be crystal clear, the following will not work:
- run
rx_to_fifo.py --rx-frequency 434500000 --server-ip=<IP>(notice the 5 instead of the 7), - point your standard receiver tool (e.g., GQRX) to the FIFO file,
- use the receiver tool (e.g., GQRX) to tune to 434.5MHz + 0.2MHz
This implies that if you need to re-tune to a different frequency, you'll have to kill any rx_to_fifo.py process before starting a new one.
When started, the console may not show the sample rate until the FIFO is actually accessed. The sample rate may not update until a valid connection is made to the server, samples are delivered and read from the FIFO. If you're curious, these are GNU Radio Stream Tags.
Now you can consume RF data (IQ samples) from the FIFO. For example, with GQRX you can use the following device string:
file=/path/to/cts.fifo,freq=0,rate=<sample rate>,repeat=false,throttle=false
Again, you can keep a local copy of a the IQ samples of the signal by redirecting the output of the FIFO to your favorite file.
- start GQRX (e.g.,
$ gqrx -e), - under "Device:" choose "Other"
- under "Device string:", put:
file=/home/user/cts.fifo,freq=0,rate=32000,repeat=false,throttle=false - click "OK"
- choose the frequency you want to inspect,
- hit the play button,
- you should see some activity on the spectrum,
- and continue as usual.
Just run:
$ osmocom_fft
-F \
--args="file=/home/user/cts.fifo,freq=0,rate=32000,repeat=false,throttle=false"If this doesn't work for you, because gr-fosphor requires hardware acceleration, just drop the -F switch.
If you want to build the tools yourself, we recommend using our Docker images. However, if you're familiar with GNU Radio and have a working 3.7 installation, you can build them natively.
$ cd cts-cli/
$ ./docker-run.sh 'make all'$ cd cts-cli/
$ make allIf you run Linux and just want to install GNU Radio 3.7, we recomend using the following PPAs to get the basic tools installed (this is what we use in our Docker images).
$ sudo add-apt-repository -y ppa:bladerf/bladerf \
&& sudo add-apt-repository -y ppa:myriadrf/drivers \
&& sudo add-apt-repository -y ppa:myriadrf/gnuradio
$ sudo apt-get update \
&& sudo apt-get install -y \
gr-osmosdr \
gnuradioIf rx_to_fifo.py does not work for your situation, sample data can be saved
to a file within GNU Radio via the RF Over IP Source
(rf_over_ip_source.grc) block, or just copied from the FIFO and processed
statically by any tool.
Take a look at the rx_to_fifo.grc flowgraph as an example.
If no samples can be read from the FIFO, it is possible that there is no
connection to the server. In this case, check network connectivity, check that
the server_ip = <SERVER IP> in the config/cts_contestant.cfg file is
correct, terminate any process accessing the FIFO (don't feel bad if you need
to use kill -9, we all did it at least once in our lives), restart
rx_to_fifo.py, and then restart any downstream tool accessing the FIFO.
Occasionally, when starting or stopping a tool that accesses the FIFO, the following error:
thread[thread-per-block[7]: <block file_sink (12)>]: file_sink write failed with error 27
In this case simply kill and restart the script. Only one instance
rx_to_fifo.py should be run at a time. Multiple readers simultaneously
reading from the FIFO will result in corrupted data being read.
IQ samples are written to a named pipe (the FIFO file), which is an OS-dependent beast. In theory, you could run the 'rx_to_fifo.py' receiver within a Docker container and consume IQ-samples from outside the Docker container. However, this will work only if the host operating system is the same of the container operating system (i.e., Linux). We tried with a macOS host: didn't work!
$ ./docker-run.sh 'python rx_to_fifo.py \
--server-ip=<this is given by the organizers> \
--rx-frequency=<each challenge has its own frequency>'And then you can use ./cts.fifo as any other named pipe (FIFO), e.g.
$ dd if=cts.fifo of=samples.raw bs=2048 count=1000This will not work on any non-Linux host OS (because FIFOs are OS-specific) but a trick would be to use a second container that proxies that FIFO to a TCP socket (using socat), expose that port to the host, and then use socat on the host to re-proxy from the exposed port back to a file. Not clean, but should work.
Alternatively, simply use dd or cat from within docker:
$ ./docker-run.sh 'dd if=cts.fifo of=samples.raw bs=2048 count=1000'The resulting samples.raw file can then be read directly into tools like URH.
You may get compilation errors, which actually don't block the build process.
Should you encounter build errors on GNURadio 3.8, please check this out, or just use one of the provided VMs (they're tested). The cause is unknown, but a workaround exists (comment out global_blocks_path in ~/.gnuradio/grc.conf).
Other errors could be ignored: what really matters is the final rx_to_fifo.py file. If that's there and it loads without errors, you're good to go.
$ ls -lah *.py
ls: cannot access '*.py': No such file or directory
$ make
rm -rf *.pyc
rm -rf epy_*.py
for grc in tags_to_vars rx_to_fifo; do \
rm -rf $grc.py; \
done
rm -rf build
rm -rf __pycache__
for grc in tags_to_vars rx_to_fifo; do \
grcc -d ./ $grc.grc; \
done
Unhandled exception in thread started by <bound method Thread.__bootstrap of <Thread(Thread-1, stopped daemon 139634459854592)>>
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 774, in __bootstrap
self.__bootstrap_inner()
File "/usr/lib/python2.7/threading.py", line 814, in __bootstrap_inner
(self.name, _format_exc()))
File "/usr/lib/python2.7/traceback.py", line 241, in format_exc
etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'
FIFO already exists: /home/user/cts.fifo
FIFO already exists: /home/user/cts.fifo
>>> Warning: This flow graph may not have flow control: no audio or RF hardware blocks found. Add a Misc->Throttle block to your flow graph to avoid CPU congestion.
Unhandled exception in thread started by <bound method Thread.__bootstrap of <Thread(Thread-1, stopped daemon 139966222206720)>>
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 774, in __bootstrap
self.__bootstrap_inner()
File "/usr/lib/python2.7/threading.py", line 814, in __bootstrap_inner
(self.name, _format_exc()))
File "/usr/lib/python2.7/traceback.py", line 241, in format_exc
etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'
$ ls -lah *.py
-rw-rw-r-- 1 user user 93 Oct 15 15:19 epy_chdir.py
-rw-rw-r-- 1 user user 384 Oct 15 15:19 epy_mkfifo.py
-rw-rw-r-- 1 user user 1.6K Oct 15 15:18 epy_tags_to_vars.py
-rwxrwxr-- 1 user user 17K Oct 15 15:19 rx_to_fifo.pySample rates are set by the server on a per-signal basis. The RFoIP server
automatically informs the client (e.g., rx_to_fifo.py) about the original
sample rate of the tuned signal, so that the IP-to-RF translation is precise
and lossless. Obviously, clients can resample or process the signal as they see
fit.
Note that some GNU Radio blocks do not support the run-time adjustment of sample rates (indicated by block parameters that are not underlined), so the best strategy to address this is to tune to a signal, examine the returned sample rate and then set the sample rate of the tuned signal as the default sample rate in your flow graph or tool.
If you're curious about how this has been achieved, the sample rate indication works via the transmission of a GNU Radio tag attached to the tuned sample stream from the client to the server.
The tag is received by the RF Over IP Source (rf_over_ip_source.grc)
block and the variable samp_rate is automatically updated in the parent flow
graph. Individual signals have fixed sample rates that do not change, though
the samp_rate variable may initially change upon flow graph start if the
default value of samp_rate differs from the tuned signal.
Sample rate throttling is performed on both the server and client sides of the IP link by default. This does not by default create a two-clock problem as the IP link between server and client is asynchronous. If a hardware block is used on the RX side or throttling interferes with a particular tool, it can be disabled on the RX side by passing a 0 to the throttle parameter of the RF Over IP Source block.
You may notice the following warning message when using the RFoIP infrastructure:
Warning: the blks2.selector block is deprecated.
This is not a concern as the selector block is not being used to change signal paths dynamically, but only once upon flow graph startup. The selector block was deprecated by the GNURadio project as it is not sample-accurate when switching during runtime in that an unknown amount of samples are dropped during switching and its operation can be confusing to users.
Should you encounter blocking errors, just open an issue or contact us.