GTFS-Realtime stop time updates might not be updated correct

[johannilsson]

We’ve just added a new source for realtime stop time updates. For this source we don’t have the possibility to supply updates for all stops for a trip. Instead we only update the stop times that we have data for.

A trip update looks something like this

trip update
  trip id 49345
  stop time update
    departure delay 60
    stop id 740016449 

With no realtime data applied we get this from GET /index/trips/SE:49345/stoptimes for a stop along the trip.

{
  stopId: "SE:740015747",
  scheduledArrival: 59820,
  scheduledDeparture: 59820,
  realtimeArrival: 59820,
  realtimeDeparture: 59820,
  arrivalDelay: 0,
  departureDelay: 0,
  timepoint: true,
  realtime: false,
  realtimeState: "SCHEDULED",
  serviceDay: 0,
  headsign: "Malmö Stenkällan"
}

OTP can find the trip with a plan request.

With realtime applied, GET /index/trips/SE:49345/stoptimes gives this for a stop along the trip.

{
  stopId: "SE:740015747",
  scheduledArrival: 59820,
  scheduledDeparture: 59820,
  realtimeArrival: -1,
  realtimeDeparture: -1,
  arrivalDelay: -59821,
  departureDelay: -59821,
  timepoint: true,
  realtime: true,
  realtimeState: "UDPATED",
  serviceDay: 0,
  headsign: "Malmö Stenkällan"
}

For some reason realtimeArrival and realtimeDeparture is set to -1 and arrivalDelay and departureDelay is set to a negative value.

For stop 740016449 (provided in the stop update) departureDelay is set to 60 as expected.

This also results in OTP not finding the trip with a plan request when the same parameters as previously is used.

I believe the reason for this is that we in Timetable.createUpdatedTripTimes update arrival and departure times for stop times with TripTime.UNAVAILBLE that has the value -1 when a delay is not set in TripUpdate for the stop.

If I modify TripTimes.getArrivalTime and TripTimes.getDepartureTime to check for TripTime.UNAVAILBLE and for those cases instead return the scheduled time. The endpoint for stop times returns the following.

{
  stopId: "SE:740015747",
  scheduledArrival: 59820,
  scheduledDeparture: 59820,
  realtimeArrival: 59820,
  realtimeDeparture: 59820,
  arrivalDelay: 0,
  departureDelay: 0,
  timepoint: true,
  realtime: true,
  realtimeState: "UDPATED",
  serviceDay: 0,
  headsign: "Malmö Stenkällan"
}

And OTP can once again find the trip.

Anyone else seen this behaviour before or am I missing something here?

I'll update this issue with an PR with the changes I've made so you can see the changes in context too.

[johannilsson]

Instead of changing the behaviour of getArrivalTime and getDepartureTime I changed how we apply trip updates in the PR. This has the same effect and will also work for cancelled trips.

[abyrd]

The -1 values are certainly not intended to make it through to the outside world. Much of the realtime logic was originally designed for dealing with Dutch KV data that has different semantics to GTFS-RT.

Then after switching to GTFS-RT we had a discussion about modularization of the arrival prediction process. You really only have one piece of source data, which is the current position of the vehicle or its delay at the last checkpoint/stop it passed through. All the other delay information is predicted or propagated from that one piece of delay info. We considered it ideal for the prediction to be handled by a module external to the trip planner (which could call upon large databases of past performance or specialized algorithms as needed), and came up with the idea of "strict" realtime feeds which would always provide updates for every stop on a trip, leaving no guesswork at all to the router.

I still think this is a good idea, but the idea was never fully implemented or adopted, so we have to accept some prediction happening within the router. In the case of OTP this is just propagation of the delay down the route, perhaps accounting for timepoints where the vehicle will wait to get back on schedule.

As long as we are making forward predictions/propagations, it makes sense to predict backward as well. The approach of just using the scheduled time seems fine, except that it could result in negative travel times if the vehicle is running ahead of schedule (which is allowed in some places e.g. in the Netherlands where only departure times at timepoints are guaranteed).

The other downside I see to using the scheduled times is that we may have accurate delay figures from a previously applied update, which we are then wiping out and replacing with scheduled information.

@johannilsson what's your thinking on this, or am I misinterpreting your situation?

[johannilsson]

Thanks for looking into this @abyrd.

Because of the forward propagation in place I was surprised that previous stops was removed from routing. I didn’t expect that behaviour and that was what I wanted to address.

In the end I used the same approach as if I would have added the stops to the feed with ScheduleRelationship.NO_DATA that sets the delay to 0.

This might not work in all cases, but I still think it’s better than the stops being removed from routing.

A problem I see with this approach is the same as when providing NO_DATA for stops and that is that the delay is set to 0 but the stops is still marked as realtime. This will have the same results as a SCHEDULED stop time update with the delay of 0. These will all show up in clients as “On time”.

A better approach I suppose would be to handle the NO_DATA case. Then we could mark the delay for these stops as unknown instead of assuming they’re on time and clients can take decisions based on this. Same approach would then be used for the previous stops.

We’re producing realtime feeds on a national level from upstream sources and is just trying to reduce the data we send over wire to a bare minimum. I agree that the ideal would be to provide data for the full trip, but according to the spec this shouldn’t be needed if I understand it correct.

[barbeau]

As long as we are making forward predictions/propagations, it makes sense to predict backward as well.

Upstream propagation violates the GTFS-rt spec, if I'm interpreting what you're saying correctly - see google/transit#18.

The other downside I see to using the scheduled times is that we may have accurate delay figures from a previously applied update, which we are then wiping out and replacing with scheduled information.

See the same PR above - assuming you're consuming a GTFS-rt FULL_DATASET feed (vs. DIFFERENTIAL), wiping out any previous state is actually correct behavior - you shouldn't hold over updates from a previous feed snapshot. In the absence of a stop_time_update for a particular stop in a given feed snapshot, you have to assume there is no data for that stop. If there are upstream predictions, then these get propagated downstream to this stop. If there are no upstream predictions, then your only option is to use schedule information.

The above is how we've implemented GTFS-rt per stop prediction support in OneBusAway, which matches the GTFS-rt spec.

[johannilsson]

@abyrd Have you given more thoughts to this?

Given that backward / upstream propagation violates the gtfs-rt spec that does not seem to be an option. Thanks for pointing that out @barbeau, too bad your patch isn't visible on the documentation site yet the updated example makes it more clear how it's intended to work.

[abyrd]

@johannilsson it sounds like you've been using this modified code in practice and it's working for you. @clinct is this change going to cause any problems for the NL deployment? If not I may try to get this into OTP 1.0.

[clinct]

@abyrd It's quite a long thread, you are referring to PR #2297 ?

[samantsunil]

How can we show the real-time vehicle positions on the map having the config set for vehicle-position-updater in router-config.json file? didn't find the source or any repository related to this implementation. Thanks in advance !

[johannilsson]

@abyrd Yes, have had this running for while and it has resolved the original issues I was seeing. Eventually I would like to have a better way to handle the NO_DATA cases but this has resolved the main issues for now.

[abyrd]

@clinct sorry yes, I meant PR #2297. @barbeau and @johannilsson made a good case for handling messages the way this PR does. I'd like to merge it but since Plannerstack is a major user of realtime data, I wanted to check that it won't break anything in NL.

[clinct]

Ok, I also just checked it with @skinkie, and this should not have impact for the Dutch context, of course still needs to be tested then.

Timeline-wise, I'm thinking that it might be smarter to pull this in for 1.1 instead, just to be sure.

[abyrd]

I don't want to take any risks just before the release, so I'll merge these realtime related PRs just after, as part of OTP 1.1.0-SNAPSHOT.

[abyrd]

Looking at this again... I think we need to make some decisions regarding the consumption of "non-strict" GTFS-RT (which has missing times on trip updates) and on simple downstream propagation, which is apparently required by the spec. @t2gran I'm curious about the Norway case - I imagine you're using SIRI but is it being mapped to the same internal data structures as GTFS-RT? Is propagation of delays performed by an external module or within OTP?

[lassetyr]

The SIRI implementation in OTP creates updated TripTimes, but does not use the GTFS-RT datastructures.

In the SIRI standard there are several ways to provide realtime data, we have implemented support for two of them:
In SIRI ET (EstimatedTimetable) all stops for a journey is provided, and the delays are expected to be propagated outside OTP. (This is useful when the vehicle is able to drive faster than planned, and the estimates can reflect this.)

In SIRI VM (VehicleMonitoring), on the other hand, the data provided is only the current delay (as in "now"), and the next stop. To avoid non-increasing trip-times, this delay is added to the stop provided and propagated to all the following stops of that journey.

[barbeau]

@lassetyr For SIRI VM implementation, is the delay propagated across trips within the same block (in GTFS terminology), or does the propagation end at the end of the trip?

[lassetyr]

The propagation ends at the end of the trip.

[abyrd]

Thanks for the clarification @lassetyr. So, in the initial GTFS-RT implementation we handle GTFS-RT like SIRI ET, expecting updates for all stops. As @barbeau has pointed out for true compliance with the GTFS-RT spec we should propagate delays down the line, and it makes sense to me that in that case propagation would continue to other trips in the same block. It's just not something that would be used in RT pipelines that perform full-trip predictions outside OTP, which is the case where it was originally implemented in the Netherlands.

[skinkie]

@abyrd propagate -> smooth out?

[barbeau]

it makes sense to me that in that case propagation would continue to other trips in the same block.

I agree with this, and this is OneBusAway's behavior. It's currently not specified in GTFS-rt whether propagation should happen across trips in the same block (CUTR-at-USF/gtfs-realtime-validator#90), but I plan to open a proposal soon to specify that consumers should propagate across trips in the same block.

[skinkie]

Does GTFS-RT allow to update blocks on scheduled trips?

[barbeau]

@skinkie If you mean the ability to change which trips are contained in a specific block, no, not currently. Our current plan is to encapsulate these types of network changes into a proposed new GTFS-rt Service Changes feed. Draft is here if you're interested - http://bit.ly/gtfs-rt-service-changes. We're actively looking for consumers/producers for this and have a few on-deck.

[abyrd]

On 11 Sep 2018, at 23:53, Stefan de Konink ***@***.***> wrote: @abyrd <https://github.com/abyrd> propagate -> smooth out?

Propagate, meaning handle the case analogous to SIRI VM, where you know how late the vehicle is but don’t receive a full estimated timetable based on that lateness. For example, if delay is 5 minutes, you proceed down the line and “propagate” the delay, adding 5 minutes to every arrival and departure time. It’s just trivial fallback “prediction” where the vehicle is expected to remain exactly as late as it already is. While I think there’s agreement that we want to keep sophisticated prediction in external modules that produce full estimated timetables, it seems reasonable to have this simplest possible fallback within OTP.

[skinkie]

If you want propagation across blocks. Even the most most simple propagation should include the waitpoints and turning times.

[abyrd]

If delay propagation is implemented, I actually don't think we should go to extra effort to include such nuances. Recall that this is just a fallback to meet the GTFS-RT spec.

Transport systems that share data on timing points and turnaround points etc. are probably sophisticated enough to provide full "estimated timetables" from the operator side. Naive delay propagation is a fallback for situations where you have poor data (from legacy systems).

As such it shouldn't attempt to be too smart - at the point where people are actually adapting to timing points where the bus waits, they should be encouraged to run a separate ET prediction system.

If @barbeau wants to include propagation across blocks, that could be added in the spirit of OSS developers solving whatever needs they have locally, but I would rather not go out of the way to perfect this simple "prediction" system. That should be a separate project producing full ET output.

[barbeau]

I don't think we should extend GTFS-rt propagation across multiple blocks (or, at least, that's not what I'm currently proposing - we could explore those more advanced concepts in future proposals, e.g., CUTR-at-USF/gtfs-realtime-validator#91).

My proposal covers the simple case that @abyrd explained of taking a delay and applying it to all stops downstream of a given stop until the end of the block (i.e., across multiple trips within the same block if needed). Propagation does take into account the scheduled arrival_times and departure_times within a block.

Here's a diagram I created to explain what propagation looks with and without propagation delays across trips in the same block:

So, in this case, there is a scheduled 10 minute layover (scheduled arrival_time of trip_A at transit center is 10 minutes before scheduled departure_time of trip_B), so given a delay of 20 minutes on trip_A, you'd see a delay of 10 minutes on trip_B. This is OneBusAway's behavior.

But again, this is just applying the given delay directly against the arrival_time and departure_times in stop_times.txt.

[t2gran]

We have planned to implement something similar for Guaranteed Interchanges. It is very similar to blocks, but it can also propagate between transport modes/operators. If a ferry is late the bus wait. The bus and the ferry may not exist in the same RT system; hence the propagation must be handled in a place which have the RT information from both.

I will keep this Issue in mind and try to solve both cases when we get to it.

[barbeau]

@abyrd @skinkie @t2gran I've just opened a proposal for the GTFS-realtime spec to formalize how delays are propagated across trips in the same block - google/transit#110. Feedback is welcome!

[ddecampos]

@johannilsson
It may not have much to do, but due to the limited information about it, I choose to ask, how do I configure the updater to recognize the vehicle Positions

I would be infinitely grateful if you help me

regards

[johannilsson]

@danieldc2809 This issue does not apply to vehicle positions unfortunately.

[ddecampos]

@johannilsson
OK, so could you help me with something relevant?

This is the answer I have with the real time applied in /index/trips/1:28999-1/stoptimes

{ "stopId": "1:202403", "stopIndex": 0, "stopCount": 59, "scheduledArrival": 23400, "scheduledDeparture": 23400, "realtimeArrival": 23400, "realtimeDeparture": 23400, "arrivalDelay": 0, "departureDelay": 0, "timepoint": true, "realtime": false, "realtimeState": "SCHEDULED", "serviceDay": 0, "blockId": "188SI0005", "headsign": "a Av. De los Constituyentes y Av. Gral Paz" }

but the "realtime" is false

What am I doing wrong, or what am I not doing?

[ddecampos]

I'm seeing that real time is not applying

why?

[t2gran]

@danieldc2809 please use the user/dev mailing list for this kind of questions.
https://groups.google.com/forum/#!forum/opentripplanner-users

[ddecampos]

@t2gran
I've done it before, but thanks

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days