Be smarter about identifying gaps due to subway travel

[felipevh]

No description provided.

[Nate-Wessel]

Just providing some description here:
The way we're currently detecting unknown time is very naive, only a few lines of if statements if I recall. We need to think through a more robust algorithm that accounts for gaps from subway trips, cold starts, etc.

[felipevh]

I'm not sure where to start with this. In trace.py/make_known_subsets we're currently breaking on gaps of longer than an hour and 1 km. Not sure what unknown time should look like. @Nate-Wessel

[Nate-Wessel]

I've modified the code so that any gap (after cleaning) of larger than 1km is now unknown. This removes pretty much all subway trips, but I figure that the problem of getting subways back in as subways is easier than that of finding other unknown times while those remained. We are now often misidentifying parts of genuine travel as unknown though.

[felipevh]

using osmnx package, so we have an additional python package dependency

[Nate-Wessel]

Hang on - you are using that now? We need to discuss a plan for this before proceeding.

[Nate-Wessel]

OK, here's my plan for moving forward with this:
We need a new module to interact with OSM data, either downloaded in advance and stored locally or queried dynamically from the API. As a first step, it will probably be easiest to use subway station locations as points where signals may appear/disappear. Down the road though I'd like to replace these points with a network of tunnels (of any sort, not just subway, so including PATH etc.) though I think figuring out the topology to make use of this data correctly may take some serious effort. The logic should be internal to the module, so that there's just a method that tells us whether a gap in GPS signal is likely error or related to e.g. underground travel, at least based on what the GIS data says about the area.

Really, this can sort of be our OSM GIS module, possibly handling map matching down the road too. For now though, the only interaction with this module will be in the stage of identifying unknown time.

So:
Short term: make a module that contains methods for accessing subway point locations
Long term: expand/refine that to assess any possible cause of GPS outage such as tunnels, airports, etc.

[Nate-Wessel]

User A has four subway trips to test on and user C has one. All five exhibit the currently expected pattern of travel with a span of unknown time in the middle.

[Nate-Wessel]

We're now retaining any segments (that otherwise would have been gaps) that both start and end within 200m of a subway station node. This fixes three of the 4 previous errors for user A but does not appear to change anything for C.