Need for a generic data transfer profile

[dodoradio]

Currently, transferring data between the watch and paired device requires a dedicated BLE profile. This is a steep learning curve for app developers, as an understanding of BLE and asteroid-btsyncd is needed.

[dodoradio]

It has been suggested that this should act as a dbus interface, so that dbus commands sent to this application get sent to the paired device.

@jrtberlin recommends the use of protbuf as a method of converting data to binary, as it is easy to work with.

Challenges include the 512 byte limit (see #21 ), which would require some work to allow greater volumes of data to be transferred. Possible solutions include an 'address' characteristic that moves the section of data shown in a read characteristic.

[beroset]

Ideally, a successful solution for a generic data transfer mechanism should be:

1. efficient - time, bandwidth, memory and battery are all precious commodities)
2. simple - it should be simple to use and simple to understand. This makes it easier to keep it bug-free.
3. extensible - as a generic means of data transfer, it could be used (and re-used) for many purposes
4. documented - good documentation about how it works and how to use it (and perhaps how not to)
5. secure - although BLE is a local protocol, security considerations should be in scope

Questions:

1. Should we pick a method that will work only for BLE, or should it be applicable to WiFi?
2. Should acknowledgements be selectable?
3. What are some use cases? I can think of a few -- transferring arbitrary files and doing things like synchronizing calendars
4. Should this mechanism replace or extend the existing BLE profile?

[dodoradio]

Regarding use cases, we already have a few:
Transfer of events and calendar data
Transfer of music cover art
Replacing the current weather data system with something simpler and more extensible

[beroset]

Is 6lowpan over BT LE a possibility? https://datatracker.ietf.org/doc/html/rfc7668

[FlorentRevest]

This is a difficult problem space, thanks for looking into it! :)

Currently, transferring data between the watch and paired device requires a dedicated BLE profile. This is a steep learning curve for app developers, as an understanding of BLE and asteroid-btsyncd is needed.

This is both true and not true. Let's say you are a lambda developer and want to add a new syncing feature to a specific app. Indeed it kinda sucks that you need to standardize a BLE profile and implement something in asteroid-btsyncd etc... But on the other hand, the barrier of entry to write a new sync companion app is fairly low. This is one reason why we have sync clients for UBTouch, Windows, SailfishOS etc... Now let's say that we were to implement a complex IP stack on the companion side, it would be a lot more difficult to write a new companion app as well.

That being said, I do share your sentiment. The current BLE profiles suck and we definitely need something more flexible.

It has been suggested that this should act as a dbus interface, so that dbus commands sent to this application get sent to the paired device.

Overall I think it's probably a good idea to expose dbus over BLE but we need to be quite careful about security eg http://illmatics.com/Remote%20Car%20Hacking.pdf page 29

@jrtberlin recommends the use of protbuf as a method of converting data to binary, as it is easy to work with.

Agreed. I've had this conversation with jrt in the past.

Challenges include the 512 byte limit (see #21 ), which would require some work to allow greater volumes of data to be transferred. Possible solutions include an 'address' characteristic that moves the section of data shown in a read characteristic.

Or we'd need a protocol like 6lowpan (which you mention below)

Ideally, a successful solution for a generic data transfer mechanism should be:

1. efficient - time, bandwidth, memory and battery are all precious commodities)

Yeah, just keep in mind that overall we are actually not transferring so much information.

2. simple - it should be simple to use and simple to understand. This makes it easier to keep it bug-free.

And to keep multiple companion apps. But I am afraid that generic and flexible solutions are a bit of a pain.

3. extensible - as a generic means of data transfer, it could be used (and re-used) for many purposes

Same as above.

4. documented - good documentation about how it works and how to use it (and perhaps how not to)

Sure.

5. secure - although BLE is a local protocol, security considerations should be in scope

As said below about D-Bus. It's likely that the more flexible we make it, the less secure we also make it. But on the other hand, asteroid is not a secure operating system. I mean, it comes with super old vendor kernels, it gives you a root shell without password by default on physical access etc... the current BLE protocol implementations are kind of a joke I'm sure that if someone were to fuzz asteroid-btsyncd it would quickly blow up.

Of course I agree that the solution should be secure. I'm just saying that the current standard isn't very high. :)

Questions:

1. Should we pick a method that will work only for BLE, or should it be applicable to WiFi?

This is a common pain point of the current BLE profiles yes. It would be really nice if the same sync profiles would also work on WiFi.

2. Should acknowledgements be selectable?

Not sure what you mean here. If you're talking about different quality of services where in some profiles you want confirmation that the packet got received, I think it's worth making sure that lower parts of the BLE stack don't already cover this.

3. What are some use cases? I can think of a few -- transferring arbitrary files and doing things like synchronizing calendars

Ideally, all current BLE profiles would work on top of this one generic thing. But of course other usecases would be cool too, like a shell over BLE or so...

4. Should this mechanism replace or extend the existing BLE profile?

Idk, on one hand it'd be cool to maintain some sort of backward compatibility with the existing set of companion apps. On the other hand, I'd rather have one clean system rather than lots of different complex systems.

Regarding use cases, we already have a few:
Transfer of events and calendar data
Transfer of music cover art
Replacing the current weather data system with something simpler and more extensible

Yeah.
Also one thing that would be cool for the weather profile would be if the watch could directly trigger an HTTP query and parse the XML. Then the companion apps could be kept simpler and we'd open the door to more direct API calls from asteroid apps.

Is 6lowpan over BT LE a possibility? https://datatracker.ietf.org/doc/html/rfc7668

When I looked into it some years back, I could not figure out a practical way to make this work, but of course this sounds very interesting in theory and I'd love to see it work.

[MagneFire]

It's probably worth noting that there exist two apps that provide a similar functionality to our sync clients: KDE Connect (https://github.com/KDE/kdeconnect-kde) and Gadgetbridge (https://gadgetbridge.org/).
By default kdeconnect does not have Bluetooth enabled, but during compilation it can be enabled.

[beroset]

Also there is GSConnect ( https://github.com/GSConnect/gnome-shell-extension-gsconnect ) for us Gnomes.

[beroset]

I have been looking at the Bluetooth HTTP Proxy Service (HPS) ( https://www.bluetooth.org/docman/handlers/downloaddoc.ashx?doc_id=308344 ) and it seems like it has some very useful properties. First, a service or app on the watch can probably more easily adapt to using either Bluetooth or WiFi this way. Second, it is standard and has been since 2015.

Hmm. Not good for us -- the size of the body of the request or the response is limited to 512 bytes.

[beroset]

Here's a masters thesis that I found helpful in analyzing the different possibilities. https://lup.lub.lu.se/luur/download?func=downloadFile&recordOId=8903860&fileOId=8903861

This synopsis (derived from table 3.1 in that document) describes some aspects of the possibilities:

• GATT REST API
  • gateway implementation only
  • client-initiated communication only
• 6LoWPAN
  • peripheral and gateway implementation
  • client and peripheral initiation possible
• HTTP Proxy Service
  • peripheral and gateway implementation
  • Peripheral initiated communication only

[beroset]

For completeness, the GATT REST API is described here: https://www.bluetooth.com/wp-content/uploads/2019/03/GATT-REST-API_WP_V10r01.pdf