Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance issues with trip_distance_exceeds_shape_distance #1611

Closed
emmambd opened this issue Nov 27, 2023 · 1 comment · Fixed by #1676
Closed

Performance issues with trip_distance_exceeds_shape_distance #1611

emmambd opened this issue Nov 27, 2023 · 1 comment · Fixed by #1676
Assignees
@cka-y
Labels
bug Something isn't working (crash, a rule has a problem) parsing errors Related to improving validator parsing and clearer parsing error messages. status: Needs discussion We need a discussion on requirements before calling this issue ready

Comments

@emmambd
Copy link
Contributor

emmambd commented Nov 27, 2023 •
edited
Loading

Describe the bug

Upon generating analytics for the 4.2 release, we found that several datasets failed to run, and we assume that's because of the new trip_distance_exceeds_shape_distance error.

These datasets failed with the message The configured memory limit was reached:

  1. us-oregon-trimet-portland-streetcar-gtfs-247
  2. us-washington-sound-transit-metro-transit-city-of-seattle-king-county-metro-gtfs-267
  3. us-illinois-chicago-transit-authority-cta-gtfs-389
  4. us-new-jersey-new-jersey-transit-nj-transit-gtfs-508
  5. at-wien-wiener-lokalbahnen-wlb-gtfs-648
  6. be-vlaams-gewest-de-lijn-gtfs-684
  7. ca-alberta-edmonton-transit-system-gtfs-714
  8. ca-ontario-toronto-transit-commission-gtfs-732
  9. ca-quebec-reseau-de-transport-de-la-capitale-gtfs-757
  10. de-berlin-verkehrsverbund-berlin-brandenburg-gtfs-782
  11. Ie-dublin-dublin-bus-gtfs-947
  12. fr-auvergne-rhone-alpes-cars-region-auvergne-rhone-alpes-transisere-gtfs-985
  13. es-madrid-cercanias-madrid-gtfs-993
  14. nl-unknown-allgo-keolis-gtfs-1077
  15. ee-unknown-abuss-ou-gtfs-1095
  16. gb-unknown-transport-for-greater-manchester-arriva-in-the-north-west-gtfs-1103
  17. fi-unknown-porvoon-museorautatie-gtfs-1102
  18. be-unknown-societe-regionale-wallonne-du-transport-gtfs-1212
  19. it-lombardia-agenzia-mobilita-ambiente-territorio-gtfs-1231
  20. gr-attiki-athens-urban-transport-organisation-organismos-astikon-sugkoinonion-oasa-gtfs-1228
  21. ru-sankt-peterburg-peterburgskii-metropoliten-petersburg-metro-gtfs-1186
  22. tw-unknown-taichung-gtfs-1277
  23. dk-unknown-rejseplanen-gtfs-1292
  24. gb-unknown-chiltern-railways-gtfs-1311
  25. pt-lisboa-carris-metropolitana-gtfs-1873

These datasets failed with the message Timeout of 1800 seconds exceeded:

  1. us-unknown-amtrak-gtfs-11
  2. tn-unknown-uabs-banlieue-sahel-gtfs-1016

This dataset failed with the message connection broken:

  1. us-minnesota-metro-transit-metro-transit-met-council-maple-grove-plymouth-southwest-transit-airport-mac-university-of-minnesota-catch-the-link-gtfs-205

We should further investigate and identify possible performance improvements to this notice. Relates to #1589

Steps/Code to Reproduce

  • Run failed dataset through validator
  • Feed times out with no error message

Expected Results

  • Feeds run successfully through the validator.

Actual Results

Feed fails to parse.

Screenshots

No response

Files used

No response

Validator version

4.2

Operating system

MacOS

Java version

No response

Additional notes

No response
@emmambd emmambd added bug Something isn't working (crash, a rule has a problem) status: Needs triage Applied to all new issues labels Nov 27, 2023
@github-project-automation github-project-automation bot moved this to Requires investigation in Bug triage Nov 27, 2023
@emmambd emmambd moved this to MobilityData Backlog (bugs & maintenance) in Schedule Validator Q2 backlog Nov 27, 2023
@emmambd emmambd added this to the Now milestone Jan 9, 2024
@emmambd emmambd added status: Needs discussion We need a discussion on requirements before calling this issue ready and removed status: Needs triage Applied to all new issues labels Jan 11, 2024
@emmambd emmambd removed this from the Now milestone Feb 12, 2024
@emmambd emmambd added the parsing errors Related to improving validator parsing and clearer parsing error messages. label Feb 12, 2024
@emmambd emmambd added this to the Now milestone Feb 14, 2024
@emmambd
Copy link
Contributor Author

emmambd commented Feb 14, 2024

In the memory reduction design & analysis document, it's mentioned that there's a significant amount of memory usage from the TripAndShapeDistanceValidator. This should be further investigated.

@emmambd emmambd assigned qcdyx and cka-y and unassigned qcdyx Feb 15, 2024
@emmambd emmambd linked a pull request Feb 21, 2024 that will close this issue
feat: reformat trip and shape dist validator #1676
Merged
5 tasks
@cka-y cka-y mentioned this issue Mar 6, 2024
Merged
5 tasks
@emmambd emmambd mentioned this issue Mar 8, 2024
Closed
@github-project-automation github-project-automation bot moved this from MobilityData Backlog (bugs & maintenance) to Done in Schedule Validator Q2 backlog Mar 11, 2024
@github-project-automation github-project-automation bot moved this from Requires investigation to Done in Bug triage Mar 11, 2024
@emmambd emmambd removed this from the Now milestone Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cka-y cka-y

Labels
bug Something isn't working (crash, a rule has a problem) parsing errors Related to improving validator parsing and clearer parsing error messages. status: Needs discussion We need a discussion on requirements before calling this issue ready
Projects
Bug triage
Status: Done
Schedule Validator Q2 backlog
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: reformat trip and shape dist validator MobilityData/gtfs-validator
3 participants
@qcdyx @cka-y @emmambd