Storing data without GPS

[dennisguse]

OpenTracks now tries to create TrackPoints from BLE sensors (i.e., the ones that provide distance) every second.
It is a really naive implementation and works nicely if GPS is not used.

The current implementation has a preference of sensor-based TrackPoints as those will be triggered more often.
This will very likely result in ignoring useful GPS locations (which are expansive to get from an energy perspective).

For this, we need some more magic (aka don't try to store sensor data every second).

Fixed #500.

• decide on how to implement (change type or just use lat/lng null)
• create a new type for BLE sensor-based trackpoints: SENSORPOINT
• implement OSMDashboard change: OpenTracks API: ignore SENSORPOINTs as those do not contain lat/lng. OSMDashboard#125
• Modify ExportImport test (create senor-based Trackpoints in addition)
• Check behavior for GPX exports/ imports (as TrackPoints without location cannot be exported)
• Export/Import KML/KMZ
• Upgrade OSMDashboard protocol version OR wait for new OSMDashboard release: https://github.com/OpenTracksApp/OSMDashboard/releases/tag/v2.13
• GPX must export cumulative data from SENSORPOINTs (i.e., distance and altitude gain/loss)
• Add this feature to the README
• Rebase onto Moving time#1046 #1047
• Wait for release of Added profiles for custom layouts #1012 (at least a 3 days after availability on F-Droid)

[dennisguse]

One simple option:

• store sensor-based trackpoints if recording distance interval is fulfilled (relative to the previous stored [any] TrackPoint aka lastValidTrackPoint).
• store GPS-based trackpoints if recording distance interval is fulfilled (relative to the previously stored GPS-based TrackPoint)

[dennisguse]

Alternative with longer storage intervals was mentioned here: #500 (comment)

[dennisguse]

@pstorch This PR affects OSMDashboard.
At the moment, we are creating TrackPoints of type TRACKPOINT that do not contain coordinates.
OSMDashboard handles this case using 0/0.

An alternative would be to introduce a new type like SENSOR (i.e., no coordinates), so that TRACKPOINT has always a coordinates.
Which option do you like more?

[pstorch]

What should the OSMDashboard do with those trackpoints? Ignore? I assume that trackpoints with coordinates 0/0 will be displayed at the favourite NULL position in the ocean. I've not tested it.

[dennisguse]

That's what happens and how I discovered that we need to talk about this :)
I guess ignore those TrackPoints as they contain no useful data (for now).
Data might be interesting if we would implement something like line color by speed.

Anyhow, it is a breaking change so we need to upgrade the protocol version.

[dennisguse]

@ChristophWurst Did a test run today and it works :)
With some to be expected flaws 🌞

[pstorch]

Is it an option to filter out those beforehand on OT side? I mean trackpoints without coordinates are not that useful on a map.
If not filter out 0/0 would be easy on the Dashboard side but a little bit a hack. Some apps do it like that, but 0/0 is still a valid coordinate. Not very useful though.
Better to have a flag in the data to know that kind of trackpoints. Make it explicit.

[dennisguse]

@pstorch I introduced a new TrackPoint type (SENSORPOINT, 2), but that requires a change in OSMDashboard:
Current implementation to determine pause would not be working anymore:

        this.pause = type != null ? type != 0 : latitude == PAUSE_LATITUDE;

I see two options:

• modify the where to ignore Type=2; I propose allow list rather than disallow list
  ` try (final Cursor cursor = resolver.query(data, projection, TrackPoint._ID + "> ? AND TrackPoint.TYPE IN (-2, -1, 0, 1)", new String[]{Long.toString(lastTrackPointId)}, null)) {```
• ignore Type=2 during actual processing

Anyhow, the new type is a breaking change (for the current OSMDashboard release), so we need to upgrade the API version.

[pstorch]

As long as we cannot do anything useful on the map with those trackpoints I would prefer the first option to add it to the where clause, then they are not even transferred.

[dennisguse]

This code is ready for review and testing.

One problem.
Until now the GPX export would be "broken": SENSORPOINT TrackPoints are not exported (as they do not contain GPS coordinates).
However, SENSORPOINTs contain cumulative data (i.e., distance and altitude gain/loss), which then will not be exported.
As a quick workaround, I implemented the TrackStatsExtension #652.
Two solutions:

• remove cumulative data from GPX (i.e., opentracks:distance, opentracks:gain, and opentracks:loss) as it is only using TRACKPOINT data
• compute SENSORPOINT data and add it to the next TRACKPOINT.
  (probably the better solution, but it complicates the GPXTrackExporter).

@pstorch Due to the introduction of SENSORPOINT, a new release of OSMDashboard is required to ignore those.
I would also increase the protocol version to be on the safe side. What do you think?

@rgmf Let's merge+release the profile changes first as it provides more value :)

[pstorch]

@pstorch Due to the introduction of SENSORPOINT, a new release of OSMDashboard is required to ignore those.
I would also increase the protocol version to be on the safe side. What do you think?

I don't think an increase in protocol version is necessary. The type field is already available. There is "only" a new value. The query works with the older OpenTracks.

[pstorch]

Anything else to change in OSMDashboard? I think I should release a new version soon, because of the sensor delay bug.

[dennisguse]

@pstorch To the best of my knowledge OSMDashboard is prepared for this PR.

[dennisguse]

@pstorch one thought about the statistics for the share pic functionality of OSMDashboard: SENSORPOINTs contain cumulative sensor data (i.e., distance); is this a problem for OSMDashboard if those are ignored? [don't know how the stats are computed...

[pstorch]

@dennisguse OSMDashboard reads the statistic provided by OpenTracks and displays them as is, no computation. So the question is how OpenTracks calculates them 😉

[dennisguse]

@pstorch Perfect :)
Just woke up and had this thought :D

[rgmf]

It works like a charm. Fantastic feature!!!

I've read the code and I've tried to understand. The first thing very well, the second one not as well 🙄 Maybe, I've should been read it with more attention 🤣

Seriously now, great job ;)

[dennisguse]

@rgmf Sounds great!
I still would like to wait for #1012 as downgrading from this is not possible (we would need two releases to ship the db downgrade code etc). Releasing only the layout functionality, allows people to be able to downgrade (if there is some trouble with something). After some waiting time, we can release this then :)

[dennisguse]

Found and fixed a regression: for creating a marker, the lastStoredTrackPointWithLocation should be used as markers require a valid location.