osrm-extract: Assertion failed: Value too big for packed storage.

[pbeyssac]

Hello,
I have been using OSRM for years.
Since April, I encounter a problem when recomputing the .osrm files. Updating to 5.24.0 doesn't fix the issue.

"Assertion failed: ((value <= T{(1ULL << Bits) - 1})&&("Value too big for packed storage.")), function push_back, file /wrkdirs/usr/ports/www/osrm-backend/work/osrm-backend-5.24.0/include/util/packed_vector.hpp, line 420."

Note: these are .pbf files I generate myself, extracted from the OSM "planet" file, as the full planet file is too big for OSRM to handle. I use these to run https://signal.eu.org/osm/, a rail routing site. Nothing has changed in my workflow since the bug appeared, so I assume this is due to some change in the OSM data which OSRM doesn't like.

Here is the full output of osrm-extract:

[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /home/pb/osm/profiles/train.lua
[info] Input file: planetrail-latest.osm.pbf
[info] Profile: train.lua
[info] Threads: 4
[info] Parsing in progress..
[info] input file generated by pbet
[info] timestamp: 2021-05-15T23:59:12Z
[info] Using profile api version 4
[info] Found no turn restriction tags
[info] Parse relations ...
[info] Parse ways and nodes ...
[info] Using profile api version 4
[info] Using profile api version 4
[info] Using profile api version 4
[info] Parsing finished after 38.4909 seconds
[info] Raw input contains 30285718 nodes, 3100461 ways, and 0 relations, 0 restrictions
[info] Collecting way information on 0 restrictions...ok, after 0.001517s
[info] Collecting way information on 0 maneuver overrides...ok, after 0.00152s
[info] Sorting used nodes ... ok, after 0.479693s
[info] Erasing duplicate nodes ... ok, after 0.051717s
[info] Sorting all nodes ... ok, after 0.018501s
[info] Building node id map ... ok, after 0.056819s
[info] Confirming/Writing used nodes ... ok, after 1.81551s
[info] Writing barrier nodes ... ok, after 0s
[info] Writing traffic light nodes ... ok, after 0s
[info] Processed 24006180 nodes
[info] Sorting edges by start ... ok, after 1.41454s
[info] Setting start coords ... ok, after 1.70416s
[info] Sorting edges by target ... ok, after 1.17998s
[info] Computing edge weights ... ok, after 5.23102s
[info] Sorting edges by renumbered start ... ok, after 1.4469s
[info] Writing used edges ... ok, after 0.491357s -- Processed 26317565 edges
[info] Writing way meta-data ... ok, after 0.00479s -- Metadata contains << 2746449 entries.
[info] Collecting node information on 0 maneuver overrides...ok, after 0s
[info] Collecting node information on 0 restrictions...ok, after 0s
[info] writing street name index ... ok, after 0.00211s
[info] extraction finished after 52.7095s
[info] Generating edge-expanded graph representation
Assertion failed: ((value <= T{(1ULL << Bits) - 1})&&("Value too big for packed storage.")), function push_back, file /wrkdirs/usr/ports/www/osrm-backend/work/osrm-backend-5.24.0/include/util/packed_vector.hpp, line 420.
zsh: abort (core dumped) /usr/local/bin/osrm-extract -p ${PROFILE} planetrail-latest.osm.pbf

Same in GDB with a stack trace at the end.

signal% gdb /usr/local/bin/osrm-extract
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
(gdb) run -p /home/pb/osm/profiles/train.lua planetrail-latest.osm.pbf
Starting program: /usr/local/bin/osrm-extract -p /home/pb/osm/profiles/train.lua planetrail-latest.osm.pbf
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...[New LWP 101185]
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...[New Thread 8013c1000 (LWP 101185/osrm-extract)]
[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /home/pb/osm/profiles/train.lua
[info] Input file: planetrail-latest.osm.pbf
[info] Profile: train.lua
[info] Threads: 4
[info] Parsing in progress..
[info] input file generated by pbet
[info] timestamp: 2021-05-15T23:59:12Z
[New Thread 8013c2900 (LWP 102921/osrm-extract)]
[New Thread 8013c2e00 (LWP 102924/osrm-extract)]
[info] Using profile api version 4
[info] Found no turn restriction tags
[info] Parse relations ...
[info] Parse ways and nodes ...
[info] Using profile api version 4
[info] Using profile api version 4
[info] Using profile api version 4
[info] Parsing finished after 38.9953 seconds
[info] Raw input contains 30285718 nodes, 3100461 ways, and 0 relations, 0 restrictions
[info] Collecting way information on 0 restrictions...ok, after 0.001539s
[info] Collecting way information on 0 maneuver overrides...ok, after 0.001521s
[info] Sorting used nodes ... ok, after 0.467184s
[info] Erasing duplicate nodes ... ok, after 0.051886s
[info] Sorting all nodes ... ok, after 0.020826s
[info] Building node id map ... ok, after 0.058094s
[info] Confirming/Writing used nodes ... ok, after 1.77668s
[info] Writing barrier nodes ... ok, after 0s
[info] Writing traffic light nodes ... ok, after 0s
[info] Processed 24006180 nodes
[info] Sorting edges by start ... ok, after 1.31083s
[info] Setting start coords ... ok, after 1.69022s
[info] Sorting edges by target ... ok, after 1.23703s
[info] Computing edge weights ... ok, after 6.18987s
[info] Sorting edges by renumbered start ... ok, after 1.31152s
[info] Writing used edges ... ok, after 0.521725s -- Processed 26317568 edges
[info] Writing way meta-data ... ok, after 0.003692s -- Metadata contains << 2746449 entries.
[info] Collecting node information on 0 maneuver overrides...ok, after 0s
[info] Collecting node information on 0 restrictions...ok, after 0s
[info] writing street name index ... ok, after 0.004924s
[info] extraction finished after 53.9224s
[info] Generating edge-expanded graph representation
Assertion failed: ((value <= T{(1ULL << Bits) - 1})&&("Value too big for packed storage.")), function push_back, file /wrkdirs/usr/ports/www/osrm-backend/work/osrm-backend-5.24.0/include/util/packed_vector.hpp, line 420.

Program received signal SIGABRT, Aborted.
[Switching to Thread 8013c1000 (LWP 101185/osrm-extract)]
0x0000000800b5bb6a in thr_kill () from /lib/libc.so.7
(gdb) where
#0 0x0000000800b5bb6a in thr_kill () from /lib/libc.so.7
#1 0x0000000800b59fc4 in raise () from /lib/libc.so.7
#2 0x0000000800ad01b9 in abort () from /lib/libc.so.7
#3 0x0000000800b4a1d1 in __assert () from /lib/libc.so.7
#4 0x00000000004f401e in tbb::task::~task ()
#5 0x00000000004f3557 in tbb::task::~task ()
#6 0x00000000004ef48d in tbb::task::~task ()
#7 0x00000000004edc2f in tbb::task::~task ()
#8 0x00000000004eda2c in tbb::task::~task ()
#9 0x0000000000442c2a in std::__1::basic_filebuf<char, std::__1::char_traits >::overflow ()
#10 0x00000000002c6305 in ?? ()
#11 0x00000000002b89a5 in ?? ()
#12 0x00000000002b72b0 in ?? ()
#13 0x000000080055a000 in ?? ()
#14 0x0000000000000000 in ?? ()

Thanks for your help...
Pierre

[jcoupey]

What is the highest node id in your pbf file? Asking because you may be hitting the same overflow problem described and fixed in #6020.

[pbeyssac]

Clearly this must be it since the first node over 2^33 in my file is 8589936531, created 2021-04-03T15:58:55Z. The last file that worked didn't include it on April 3rd. osrm-extract started crashing as soon as the node was present, from the April 4th run.
Thanks a million!
I will upgrade from sources or wait for the next release.

[pbeyssac]

Recompiled from sources, confirmed working \o/. Thanks.