Downloading along a GPX track

[greuters]

Hello, thanks a lot for this great app @westnordost and others!

We're currently on a long cycling trip and started adding data to OSM, especially the surface and quality of streets which is interesting to us (and often unclear here in Mexico). As we are riding on a tandem, we started to conveniently edit on the way and I hacked together a quick solution to download tiles along our planned route when we have WIFI in our overnight locations. See the (quick and dirty, but working) implementation here: greuters/StreetComplete:gpx-download
To download, there is a new button "Download data along gpx" next to the existing "Download here" button which lets you select a GPX track and starts downloading tiles to cover it.

Do you think such a feature might be interesting to more users and be worth it to invest some more time in a proper solution, probably also overlaying the track on the map (only the last imported one)? I can imagine people going for a hike would use it, cyclists on tandem are probably too much of a niche.

If you like, I can clean up the code a bit and would of course be open for hints how to better do it. In related discussions (
#1757, #3898) I already saw that you think placing it in the hamburger menu is too prominent, would there be a more appropriate place?

I'll be available during the next two weeks, afterwards probably only with sporadic internet access, but there is no hurry for me.

Best,
simon

[westnordost]

Interesting!

Be sure to use the (surface) overlay for such things, this is exactly the use case it was made for! :-)

Glad your solution works for you! I previously outlined in this post #473 (comment) how I imagine a final implementation could look like (I understand your implementation would fall short when the points in the GPX tracks are too far apart).

On the topic of whether this would be accepted as a PR, you are right that I previously stated that placing it in the hamburger menu would be too prominent... a more appropriate place... to be honest, none comes to mind, other than hiding it behind a TBD "expert mode" or something.

On that note, @Helium314 maintains a fork/extension to StreetComplete that basically intends to be pretty much that. And, it also already implements a "download along GPX", though, as far as I know, it only takes the bounding box of the whole GPX track, so this implementation could be enhanced with a better algorithm.

[westnordost]

you are right that I previously stated that placing it in the hamburger menu would be too prominent... a more appropriate place... to be honest, none comes to mind, other than hiding it behind a TBD "expert mode" or something.

Thinking more about this, maybe a requirement for it to be included in mainline StreetComplete would be that you could actually draw the track along which you want to download within the app itself. In other words, there would be no import of a GPX, instead, a view would open (similar to the cut-way or move-node view) in which you could paint onto the map which cells should be downloaded. Could be fun. Does this make sense? Or would anyone planning a (bicycle) trip use a different app to generate a GPX anyway?

[westnordost]

Alternatively, maybe it is possible to register StreetComplete as an app that can open GPX. (Much like StreetComplete is registered to open Geo URIs.) So, when sharing a GPX, it can be shared with StreetComplete and the user has the option to download along the track then.

With this, the functionality would be there without requiring any additional buttons in the menu and thus, inclusing in mainline StreetComplete would be possible.

[Helium314]

Alternatively, maybe it is possible to register StreetComplete as an app that can open GPX

For opening GPX files this should work with

            <intent-filter>
                <action android:name="android.intent.action.VIEW" />
                <category android:name="android.intent.category.DEFAULT" />
                <category android:name="android.intent.category.BROWSABLE" />
                <data android:mimeType="application/gpx" android:scheme="content" android:host="*" android:pathPattern=".*\\.gpx" />
            </intent-filter>

Not sure about the mime type though, maybe should be application/octet-stream.

[greuters]

Nice, thank you for sharing your ideas!

I think registering an intent should be straightforward and I'll invest a little time in exploring that option. Planning in SC is a bit over the top I guess, also because good routing (e.g. BRouter) is included in many apps and probably most users already employ an app they are used to which is much more appropriate (better outdoor map styles, automatic routing, route editing, point management). I will also address the issue of interpolating intermediate waypoints if the track is too sparse (currently my solution would fallback to download the minimal bounding box including the two points - potentially way too much area).
=> this should give a minimal approach which can be considered for inclusion in SC, only major drawback being inefficient tile downloading, as each tile is downloaded separately.

Regarding integration in SCEE the major obstacle for me will be an efficient download strategy as mentioned in SCEE #471. I'll look into that but this will probably take me some time.

[mnalis]

I agree that it is best to stick to StreetComplete supporting already existing .gpx files; the intent idea is nice one, but not as important as being able to download along various types of gpx files.

As for "planning the route in SC", there is IMHO way too much that goes into that (searching for locations / amenities both internally and externally, different routing for cars and various types of bicycles, elevation data, traffic estimation, etc) to be able to put it into StreetComplete (on mobile, OsmAnd does allow for much but not all of that, and it is already hugely complex). Also even if StreetComplete did decide to become "route planner" (which I really don't think would be a good fit), the external .gpx file support would still be needed (e.g. people acquire external .gpx files from other users who traveled there before, or from official websites who created routes, traveling agencies, QR codes etc).

[greuters]

A more mature implementation is available at greuters:StreetComplete:gpx-download-by-intent now.

I tried to address most points from above, so this version:

• has no visible UI changes apart from two new dialogs which are opened when importing a track, and the track being shown on the map after import
• import is triggered by different intent-filters; I tested export / share from Locus Map and OsmAnd, as well as opening a GPX track from Samsung File Manager and Signal
• the algorithm to determine download bounding boxes is quite a bit more involved now; it guarantees a selectable sweep distance along the track is covered (interpolating track points if necessary) and merges adjacent bounding boxes if not "too much" additional area has to be downloaded because of the merge
• processing of the track (before scheduling downloads) can be aborted any time
• before scheduling downloads, the number of downloads and downloaded area are displayed and confirmation is asked
• downloads are scheduled via DownloadController, which handles caching quite well (importing the same track again after abort, overlapping downloads)
• if the import would download an area > 25 * ApplicationConstants.MAX_DOWNLOADABLE_AREA_IN_SQKM, it is automatically aborted and the user informed

The test tracks I played around with (100 - 200km) result in around 50 - 150 downloads between 50 - 200 sqkm and download volumes / times are still OK for me even in more populated areas (around 15min max e.g. around Zurich). What would you consider sane limits? Should the max area be restricted by some user setting?

I think the PR is ready for a first review and test, especially the UI part might have some pitfalls that I didn't see due to lack of Android experience..

[westnordost]

Wow, from a quick glance, this draft PR looks really well made! Actually, you can create a PR and mark it as draft if you are still working on it but have some questions or want to have feedback on some things.

96 downloads is massive, of course. I wouldn't be surprised if that hit some rate limits of the OSM API. If such a limit is hit, it should be handled gracefully - we do not want that it will result in an endless loop of retrying and retrying, for example.

Now, for such huge downloads, it would be really useful if there was a cancel button (and/or pause/resume-button) in the download-notification but I guess this would be a general feature and not anything that is specifically related to this draft feature.

The display of the GPX track: How is it persisted? How to remove it again?

The first dialog is a bit odd, both from a UI perspective and from whether it makes sense to show it at all:

1. The checkbox seems misplaced: First you select in which distance you want to download, then you select whether you want to download at all? For the progress bar, I don't know what it is for... is it the progress for calculating which tiles to download? Does it take that long?
2. I dare say that the selection between 10 and 100 meters distance is probably almost moot, as the tiles in which it is downloaded will be somewhat larger anyway, no? Did you test what is the difference in terms of elements downloaded and total time to download it it makes?

[westnordost]

I don't know for sure, but I would think that the number of downloads is substantial to driving up the overall download time and the time could be reduced by making less downloads but a larger area. I have some ideas for an algorithm, but you wrote you already implemented something like this, so I will shut my mouth until I get a proper look at your code.

[greuters]

Ok, thanks for the feedback. I opened it as draft PR now.

UI

• Rearranging the checkbox makes total sense, I'll do that. Maybe it would even make sense to hide / disable the selector when it is unchecked?
• The progress bar is there for processing the file and calculating which tiles to download. In DEV mode (connected with debugger) this process could take up to one minute, so I added progress indication. Might not be necessary any more now, on a decent phone from 2021 processing usually takes less than 10s with the current implementation and no debugger. Do you have any preference on this?

download distance
The distance determines the minimal area covered around the track, or in other words, how closely the track can sneak past a tile without having to download it (0 would mean to only download the tiles that are actually touched, though the current implementation would not support it). So even though tiles are (depending on your location, afaik) usually bigger, this makes for quite a difference.

Example results on a 130km test track:

• (tested with unlimited settings, 30s needed for processing the track) Download within 1m -> 148 downloads, 71km2
• Download within 10m -> 144 downloads, 75km2
• Download within 100m -> 100 downloads, 106km2
• Download within 250m -> 62 downloads, 185km2
• Download within 500m -> too big to handle, i.e. > 300km2
  Download times scale with both number of downloads and area. Usually 100m was the best compromise for my tracks, though I did not do thorough testing on this.

persistence
This is an open question for me. At the moment, the track is simply lost when you close the app, or overwritten when you import another one. I think that might be ok, but if you would want it persisted this might be done similar to LocationAwareMapFragment.onSaveInstanceState. We might run into limits with this approach though, given your comment of max 5000 storable trackpoints.

rate limits
I have no clue how to handle that ATM - doesn't the DownloadController do this kind of stuff? I didn't care for yet ;)

[Helium314]

Did you check whether background map tiles are properly downloaded?
After doing many successive downloads, I sometimes notice missing tiles (usually only when it's too late and I'm at the place without internet connection). The cache is definitiely large enough, not hitting the limit.

[westnordost]

Link to PR #5369

Let's continue in that ticket.