Sideband as JS8Call Interface

[simplyequipped]

Background

One of my overarching goals has been making radio-based communication more accessible to the average user. So far I have settled on the need for a simplified interface for something like JS8Call, primarily because JS8Call can obscure some of the challenges of HF propagation and the inefficiencies of a simple antenna installation. Also, since JS8Call also handles the radio interfacing part, simple radio solutions like the QDX become a great fit.

I have run into some challenges with sending encrypted data over JS8Call. It seems that JS8Call does not support sending byte strings directly, and JS8Call only supports a limited character set that cannot represent 256 individual characters in order to map byte values to characters. Basically, the already extremely low data rate takes a significant hit because you have to use multiple characters to represent over half of the byte values.

See #11 for low data rate discussions. See #399 for encryption in amateur radio discussion.

Main Topic

I have been considering using Sideband as an interface for JS8Call. The initial concept is to add an LXMF interface to pyjs8call that would map incoming messages to RNS destinations with the origin callsign as the display name. The device to receive JS8Call notifications would be specified by LXMF address when starting the pyjs8call LXMF interface. I am envisioning a command like the following to start pyjs8call, JS8Call, and the LXMF router:
python -m pyjs8call --lxmf LXMF_TARGET_DESTINATION_HASH

Incoming Message Processing:
incoming JS8Call message -> pyjs8call.Message -> pyjs8call LXMF message -> RNS transport -> Sideband message

Outgoing Message (Response) Processing:
outgoing Sideband message -> RNS transport -> pyjs8call LXMF message -> pyjs8call.Message -> outgoing JS8Call message

Obviously this would convert the encrypted Sideband message into plain text to be sent via JS8Call. It would be important to indicate to the Sideband user that the conversation will be unencrypted. Maybe a Plain destination type would be appropriate (I don't know offhand how that is handled or displayed in Sideband), or at least including a JS8Call prefix to the display name to indicate the source/destination of the conversation.

One limitation of this system would be that a Sideband user cannot initiate a JS8Call outgoing message without first receiving an incoming message to create the callsign destination on the pyjs8call end. This will require some consideration.

I am open to feedback on whether this should be pursued, and how it could be best configured for usability.

[faragher]

I wouldn't discourage pursuing it, even if it ends up not working, it'll be a great exercise in JS8Call and Reticulum. But I do wonder if it's the right tool for the job.

I wrote up a little pseudo-POP implementation (https://github.com/faragher/POPR) and encourage you to look through it for mechanics of creating LXMF messages and adding arbitrary message payloads. I learned so much writing that, and I probably went through ten iterations before it was actually clean and Reticulum appropriate.

And that's important. The flow of Reticulum is unique; not in a programming way, but in how it feels and the mindset behind it. I again highly encourage playing around with an implementation to get a feel for it. Play with the POPR code until you understand why I made all those decisions. It's MIT licensed, that's what it's there for!

To my comments on your proposed system:

You shouldn't modify Sideband. Not because you don't want new functionality, but if you can make it interface invisibly over LXMF, then any client can use it. POPR acts as an end point that sends and receives messages as a node, but can be accessed by any authorized POPR client to read or create new messages. If you have an LXMF to JS8Call "server" then it can do the same thing. I'm going to call the concept L2J because I'm lazy. It's a terrible name.

So, let's say you had an L2J server L2Js. Someone would send a message, and it would check the identity and either respond "Hey, you're not authorized to send messages, please contact the administrator at XYZ123 for authorization" or process the request. Since you're already keeping a directory of RNS addresses/identities to callsigns, you already need some sort of database.

You could also share contact information and identities over fast connections, cache radio broadcast identities, etc. Since the address size is considerable, it's worth preventing sending them repeatedly.

A plain destination is basically useless. It can't be routed, so it's really only for local broadcasts and while it does the job it's designed for very well, it's not what you're looking for.

So, what I would do:

Make an LXMF bot. Write up a script where when you send a message, it responds somehow. POPR or EchoBot (https://github.com/faragher/ReticulumExamples) are good places to start. Remember, you want to only send and receive LXM messages. You can use stock Sideband or NomadNet to test.

Now make sure you can figure out how to send/receive plaintext control messages. For example, try to get it to send "I should be sending message 'Foo' to address 'Bar'" if you send it the message "SEND: Bar Foo". Remember to test for spaces and error handling.

Then set it to display the destination and message on the server's console. Once you're there, you need to set up the JS8Call stuff. Write up a send routine and we're half done.

From the JS8Call receive routine, receive the message and destination callsign. Look up the callsign, check for the identity, make the LXM message like in POPR, set the destination and L2Js address, and send it along via Reticulum. Have it send something like "Received from CALLSIGN: MESSAGE"

You now have a 100% LXMF compatible radio bridge that works with all clients present and future. There are some implementation details (do you have a central database of identities over the Internet? What control codes do you support? Etc.) but in broad strokes you now have a system that works as-is and is human-readable.

And if you want a simple system where people don't need to remember SEND: or the like, you can make a custom/modified app to add/remove these commands to be seamless for the user.

There are other options, such as using fields, but that requires a little coordination with the LXMF standards, so it's likely not a good idea at this time.

As an aside, JS8Call can't support bytes? It's quite inefficient, but can it at least support Base64? (A-Z,a-z,0-9,+,/,=)

[faragher]

Wonderful! Let me know if you need anything.

That seems like an odd limitation. With all the character sets available, even 7 byte ASCII which comes with control codes, it seems a deliberate choice to use something which appears non-standard to me. I'll do more more research.

[faragher]

Ah, they use a variable length encoding to save bandwidth. Makes sense, but does work very poorly for binary data transfer.

[faragher]

Okay, thoughts:

They use a pre-computed Huffman tree. Smart. Here's the first ten options:

" " "01"
"E" "100"
"T" "1101"
"A" "0011"
"O" "11111"
"I" "11100"
"N" "10111"
"S" "10100"
"H" "00011"
"R" "00000"

So, removing space and "E" for the sake of discussion, you can get 8 characters in 4-5 bits. We'll call it 5. You could use two characters to represent 64 options, or three for 512.

Three characters could map to two bytes, or 15 bits to one byte. It's nearly 100% overhead, so . . .

Hex is probably a bad idea as well. 0-9 is 7-8 bits and A-F between 5 and 6, At worst, that means a byte of 00-FF is between 10-16 bits. Again, up to 100% overhead. More compatible, though.

I didn't do the math on Base64, but I assume it can't be better. There are only 44 characters in the table, so the system would be incredibly lossy.

However : there is an option for a densely packed message in the repo. https://bitbucket.org/widefido/js8call/src/24677a00343c2f3cc53f6762466a4b11893094f8/varicode.cpp?at=js8call#varicode.cpp-1735

It's barely mentioned in the documentation, and I didn't walk through the code, but it might be the option you're looking for,

[simplyequipped]

The varicode source files indicate that the huffman coding will be depreciated, and it seems like the s,c dense coding is what is being used currently. I have a working example of various encoding schemes, but they all take several minutes (7 minutes for a simple message in some cases) to send a single encrypted RNS packet.

[faragher]

That may be inescapable due to the speed at which JS8Call operates. Even if they sent one byte per character, a 16 byte address is most of their 22 character frame. a 500 byte Reticulum packet (maximum transmission unit, but may be a partial message) is 500/22 15 second cycles, or 340 seconds. Since the minimum overhead on a Reticulum packet is 19 bytes, or almost a full frame, it's definitely worth converting the messages rather than trying to broadcast a Reticulum packet. LXMF has even more overhead since it contains sender information and various metadata.

[iqw1345]

If I may, and not to derail the conversation from JS8Call, you could go the route of Fldigi. Fldigi does support KISS TCP/IP. If you want a weak signal mode like JS8Call, you could go the Olivia route within Fldigi. As you probably already know, Olivia does rival FT8 and JS8 as a weak signal mode. It's just an idea, but I don't know if it would be a good fit for the QDX though.

As a sidenote, for the countries that allow encryption, it just crossed my mind that you should be able to directly connect fldigi to reticulum and access all the various modes that fldigi has to offer, including the weak signal modes. I played the audio through my speakers and it took a little over 2 minutes to announce using Olivia-16/500 and under 1 minute using Olivia-8/1K.

For the non-ham radio operator, here are some links to better understand Olivia.

https://www.sigidwiki.com/wiki/Olivia
https://en.wikipedia.org/wiki/Olivia_MFSK

Based on this website: http://www.w1hkj.com/modes/olivia.htm

You could achieve up to a staggering 125 baud using Olivia!

If you don't find this relevant, please feel free to ignore, but hope this helps!

[iqw1345]

I have seen a couple of reddit posts saying they've used the QDX on Olivia and had amazing results with it. If you haven't already, it might be worth a shot.

[simplyequipped]

Interesting! Sounds like more experimenting. Thanks for the info.

[simplyequipped]

I stand corrected, Olivia is a MFSK mode. I also noticed that it is mentioned in the QDX product description as a supported Fldigi mode. Thanks for bringing this up.

[ashworthpayne]

Hi

I've just returned from a dip in the internet's Olivia Pool. I thought you all might find this interesting https://github.com/sntfrc/olivia-python

I hope this is useful to someone.

[simplyequipped]

Thanks for sharing this, it certainly looks interesting.

[ashworthpayne]

js8call is interesting but it would be nice if the lower level protocol was separated from the upper layers, giving it more utility. As it is, it's not very robust as a networking tool. It also suffers from being power-hungry. If you look at power-usage over time, the slowest mode is suited to QRP operation (because it will get through with small amounts of tx power) but the duty cycle increases significantly, which means much higher current draw for given message length (it depends on repeated tx to get through). If you're running on batteries you get fewer comm windows when using the lower more robust speeds. The faster modes are far more efficient power-wise but require more tx power to punch through.
I don't have the skills to write a modem. I wish I did. I'd love to have a performant modem for HF that could be adapted to network stacks like RNS.

[simplyequipped]

I agree with everything you've said here. I don't have the skills to write a modem either. I have been periodically working on a project that uses an existing FSK modem: https://github.com/simplyequipped/fskmodem. It's pretty unpolished, but can be tied to RNS.

I am trying to work with 5W or even 100mW, so HF packet is not really up to the task. The main reasons I landed on JS8Call:

• one of the only QRP FSK modems with something like an API
• operating with 100mW transmit power is possible (unlicensed operation in the 22m ISM band in the USA?)
• picks up the slack for imperfect antenna configurations (easier for inexperienced operators)

I did get Sideband-to-JS8Call working in a basic way, but it seems pretty impractical so I have set it aside for now. Instead, I decided to add support for a RNS PipeInterface to pyjs8call. It's certainly not ideal, but it's something until there is a better option.

[ashworthpayne]

While I don't read minds, I suspect we share common cause. Have you considered GMRS? It has certain features that makes it far more resilient in non-permissive environments, while being very easily approachable for non-technical users. Highly performant UHF stations can be operated within the view of even the strictest HOA's simply by modifying OTA TV antennas and off-the-shelf FM radios. With SDR and SSB modems, ranges can be greatly extended. And, a lot of GMRS users are amateurs, meaning they can act as network filters, moving only important information where it needs to go.
If someone were to rewrite the FT8 protocol in the same way JS8Call does, a better networking modem might be possible. I'm kinda impressed by the QDX btw. Thanks for mentioning it.

[simplyequipped]

I suspect that we do as well. GMRS has been on the to-do list for a while, but have not gotten to it yet. You make some good points. I am currently focused on RNode + Sideband for local unlicensed comms. The new voice codec and PTT functionality in Sideband are pretty exciting. Once group messaging is implemented, it will basically be next-level DMR over RNS with virtually zero learning curve.

[simplyequipped]

I have personally moved on from this topic for now, but in case anyone is interested in the proof of concept that came out of it: https://github.com/simplyequipped/pyjs8call/blob/lxmf/pyjs8call/lxmf.py

[markqvist]

Thanks for sharing your work! I'm sure it will be useful for someone else thinking along the same lines!