Add a MQTT GTFS-RT trip update updater #2516

hannesj · 2017-12-07T19:03:04Z

The GTFS-RT standard doesn't limit the transport method for the protobufs for the realtime messages. We are in the process of implementing a MQTT-based transport and have created the required GTFS-RT updater for that based on the Apache Paho library.

haphut · 2017-12-07T19:22:24Z

src/main/java/org/opentripplanner/updater/stoptime/MqttGtfsRealtimeUpdater.java

+                    LOG.debug("Connected");
+                    client.subscribe(topic, qos);
+                } catch (MqttException e) {
+                    LOG.warn("Could not subsctibe to: " + topic);


"subscribe"

barbeau · 2017-12-07T19:50:26Z

@hannesj Somewhat off topic (apologies) - would you be willing to share the URL of your MQTT feed when available so we can add support for this transport method in the GTFS-realtime validator?

I've opened an issue here for topic if you'd like to continue the discussion there - CUTR-at-USF/gtfs-realtime-validator#317.

hannesj · 2017-12-07T21:24:45Z

@barbeau sure! Just a correction, the GTFS-RT feed is still encoded as protobufs, they are just transported as MQTT messages, instead of polling from a server or using websocket as the transport.

barbeau · 2017-12-07T21:45:59Z

@hannesj Ah, gotcha, that makes sense (you can tell I haven't worked with MQTT before ;) ). But the MQTT messages contain full messages with header.incrementality: FULL_DATASET, right? Not header.incrementality: DIFFERENTIAL?

barslett · 2017-12-07T22:03:26Z

I told some of you today about our ITxPT standardisation for messages between vehicles and backend. the message transport is also here based on MQTT and protobuf. Let's see if we should use MQTT/protobuf for our SIRI updaters as well, @t2gran

hannesj · 2017-12-07T22:42:18Z

@barbeau we would be differential messages in order to better preserve the history in OTP. Also the size difference would be quite considerable, as we push updated predictions with up to 1/s frequency

barbeau · 2017-12-08T14:19:19Z

@hannesj Interesting - GTFS-rt says the following about differential messages:

currently, this mode is unsupported and behavior is unspecified for feeds that use this mode. There are discussions on the GTFS Realtime mailing list around fully specifying the behavior of DIFFERENTIAL mode and the documentation will be updated when those discussions are finalized.

Just curious if you have additional design documentation to specify some of this behavior? If so and it works well in practice it may be worth trying to get some of this officially adopted into the spec. I'm looking at this from the GTFS-realtime validation point of view too. Right now there are no validation rules for differential feeds because of the above. If we're seeing these in production in the wild it would be good to try and formalize some of the common behavior being used.

abyrd · 2017-12-08T16:22:42Z

The specification may say that it's unsupported, but that text has been around for a long time, and significant practical work has been done toward demonstrating how this mode could/should work. As part of a Dutch transportation ministry project in 2013, OpenTripPlanner was modified to consume a continuous stream of GTFS-RT messages in differential mode over websockets. The transport protocol was chosen over our own initial choice (ZeroMQ) to conform to existing practice in One Bus Away and following advice from other developers who had experimented with the same thing.

The Plannerstack platform is providing OTP-based routing on a complete open data set for the Netherlands, with updates at least once a minute from every vehicle in the country. This is all being done with differential GTFS-RT over Websockets interfacing with One Bus Away modules.

We have well-established semantics and implementations for these ideas that have been around for several years. If there is still ambiguity about this we can certainly document the way this existing system interprets the spec.

abyrd · 2017-12-08T16:27:05Z

It's actually pretty simple. To summarize, our differential GTFS-RT uses trip-level granularity, and the effects of successive messages about the same trip do not accumulate.

An update about a trip is always cleanly applied to the base trip from the original static feed, with no consideration for any previously received message. The differential effect comes from accumulating changes to different trips across the whole data set.

abyrd · 2017-12-08T16:29:11Z

If people in Norway and Finland are using these same rules (which is probably the case if we're all using OTP just with different transport layers) then we can create a joint document promoting this interpretation of differential mode.

barbeau · 2017-12-08T18:38:11Z

Thanks @abyrd! Yes, I'd love to get this behavior documented and adopted into the spec. I've opened an issue at google/transit#84 where we can start tracking documentation related to differential messages, with the goal of creating a proposal to get the documented behavior into the GTFS-realtime spec. Anyone, please feel free to comment more there.

abyrd

This PR itself looks good to me, the main remaining action is to make this widely-accepted definition of differential GTFS-RT official, and @barbeau has already created an issue for that elsewhere.

barbeau · 2018-09-11T14:03:06Z

FYI - GTFS-rt issue for formalizing header.incrementality: DIFFERENTIAL is google/transit#84. I'll add a comment there referencing this PR.

hannesj · 2018-09-11T15:03:29Z

If anyone wants to test the implementation using HSL real-time data, it is available with the following updater configuration. This is in conjunction with this GTFS file

    {
      "type": "MQTT-gtfs-rt-updater",
      "url": "tcp://mqtt.hsl.fi",
      "topic": "gtfsrt/v1/fi/hsl/tu",
      "feedId": "HSL",
      "fuzzyTripMatching": true
    }

abyrd · 2018-09-11T15:39:10Z

Thanks for the config @hannesj, we should indeed test this out before merging just as a matter of good procedure.

barbeau · 2018-09-11T16:18:30Z

Thanks @hannesj! Could you share URLs to your feed (and any documentation) as well as links to the user-facing deployment of OTP that consumes this feed in the GTFS-realtime issue google/transit#84?

hannesj · 2018-09-11T17:09:40Z

The GTFS-RT-feed is published on mqtt.hsl.fi, using the topic gtfsrt/v1/hsl/tu, as specified by the config. The architecture and service documentation unfortunately still refer to the old APIs, but will be updated soon. The public facing ui is located at https://www.reittiopas.fi/, and the OTP API is available at https://api.digitransit.fi/routing/v1/routers/hsl/.

abyrd · 2019-01-23T10:34:54Z

@barbeau did you ever get a chance to test this PR? I'm going to change the base to the new dev-1.x, maybe we can get this merged sometime soon.

abyrd · 2019-01-23T10:42:28Z

Oh, looks like it no longer compiles when updated to dev-1.x. But it might be a quick fix.

barbeau · 2019-01-23T14:48:23Z

@abyrd said:

@barbeau did you ever get a chance to test this PR? I'm going to change the base to the new dev-1.x, maybe we can get this merged sometime soon.

Unfortunately not yet. DIFFERENTIAL GTFS-rt has current taken a backseat compared to other priorities at the moment, but it's definitely something I'd like to return to. Yes, any changes that would get the PR back into a runnable state would be great!

hannesj · 2019-02-10T09:54:19Z

@abyrd @barbeau This is now updated similarly how it was done in 544e1b1

Also the URL for the HSL feed has changed to:

    {
      "id": "hsl-trip-updates",
      "type": "MQTT-gtfs-rt-updater",
      "url": "tcp://mqtt.cinfra.fi",
      "topic": "gtfsrt/v1/fi/hsl/tu",
      "feedId": "HSL",
      "fuzzyTripMatching": true
    },

optionsome · 2019-02-12T10:21:30Z

src/main/java/org/opentripplanner/updater/stoptime/MqttGtfsRealtimeUpdater.java

+ * <pre>
+ * bessersmith.type = mqtt-gtfs-rt-updater
+ * bessersmith.feedId = hsl
+ * bessersmith.url = ssl://mqtt.hsl.fi:443


@hannesj pointed out that the URL is currently different. However, this old URL given here should be production ready soon again. If I'm not mistaken, the new port for MQTT will be 8883 (as recently updated in the documentation here)

hannesj · 2021-09-28T10:23:38Z

Implemented for OTP2 in #3094

haphut reviewed Dec 7, 2017

View reviewed changes

hannesj force-pushed the mqtt-gtfs-rt branch from a306434 to bc2a645 Compare December 7, 2017 19:25

barbeau mentioned this pull request Dec 7, 2017

Support non-HTTP protocols for GTFS-realtime feeds CUTR-at-USF/gtfs-realtime-validator#317

Open

barbeau mentioned this pull request Dec 8, 2017

Define "header.incrementality: DIFFERENTIAL" consumer/producer behavior google/transit#84

Open

hannesj force-pushed the mqtt-gtfs-rt branch from bc2a645 to 699e88f Compare December 9, 2017 22:44

abyrd requested a review from a team September 11, 2018 10:52

abyrd previously approved these changes Sep 11, 2018

View reviewed changes

abyrd changed the base branch from master to dev-1.x January 23, 2019 10:35

Add a MQTT GTFS-RT trip update updater

695de81

hannesj dismissed abyrd’s stale review via 695de81 February 10, 2019 09:53

hannesj force-pushed the mqtt-gtfs-rt branch from 301b3cf to 695de81 Compare February 10, 2019 09:53

optionsome reviewed Feb 12, 2019

View reviewed changes

optionsome mentioned this pull request Feb 19, 2019

Outdated references to graph.properties #2725

Closed

hannesj mentioned this pull request Jun 9, 2020

Add a MQTT GTFS-RT trip update updater for OTP 2 #3094

Merged

8 tasks

hannesj closed this Sep 28, 2021

isabelle-dr mentioned this pull request Jan 8, 2022

Support non-HTTP protocols for GTFS-realtime feeds MobilityData/gtfs-realtime-validator#58

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a MQTT GTFS-RT trip update updater #2516

Add a MQTT GTFS-RT trip update updater #2516

hannesj commented Dec 7, 2017

haphut Dec 7, 2017

barbeau commented Dec 7, 2017

hannesj commented Dec 7, 2017

barbeau commented Dec 7, 2017

barslett commented Dec 7, 2017

hannesj commented Dec 7, 2017 •

edited

Loading

barbeau commented Dec 8, 2017

abyrd commented Dec 8, 2017

abyrd commented Dec 8, 2017

abyrd commented Dec 8, 2017

barbeau commented Dec 8, 2017 •

edited

Loading

abyrd left a comment

barbeau commented Sep 11, 2018

hannesj commented Sep 11, 2018

abyrd commented Sep 11, 2018

barbeau commented Sep 11, 2018

hannesj commented Sep 11, 2018

abyrd commented Jan 23, 2019

abyrd commented Jan 23, 2019

barbeau commented Jan 23, 2019

hannesj commented Feb 10, 2019 •

edited

Loading

optionsome Feb 12, 2019

hannesj commented Sep 28, 2021

Add a MQTT GTFS-RT trip update updater #2516

Add a MQTT GTFS-RT trip update updater #2516

Conversation

hannesj commented Dec 7, 2017

haphut Dec 7, 2017

Choose a reason for hiding this comment

barbeau commented Dec 7, 2017

hannesj commented Dec 7, 2017

barbeau commented Dec 7, 2017

barslett commented Dec 7, 2017

hannesj commented Dec 7, 2017 • edited Loading

barbeau commented Dec 8, 2017

abyrd commented Dec 8, 2017

abyrd commented Dec 8, 2017

abyrd commented Dec 8, 2017

barbeau commented Dec 8, 2017 • edited Loading

abyrd left a comment

Choose a reason for hiding this comment

barbeau commented Sep 11, 2018

hannesj commented Sep 11, 2018

abyrd commented Sep 11, 2018

barbeau commented Sep 11, 2018

hannesj commented Sep 11, 2018

abyrd commented Jan 23, 2019

abyrd commented Jan 23, 2019

barbeau commented Jan 23, 2019

hannesj commented Feb 10, 2019 • edited Loading

optionsome Feb 12, 2019

Choose a reason for hiding this comment

hannesj commented Sep 28, 2021

hannesj commented Dec 7, 2017 •

edited

Loading

barbeau commented Dec 8, 2017 •

edited

Loading

hannesj commented Feb 10, 2019 •

edited

Loading