Cannot process planet-latest.osm.pbf

[krk]

Planet cannot be processed with the error message after 30 hours: "free(): invalid size". This is the third try.

What would you suggest to be able to successfully process the planet file?

Configuration: 224 GB RAM 16 cores and 600 GB free disk space.

Welcome to Ubuntu 15.10 (GNU/Linux 4.2.0-19-generic x86_64)

osrm-extract planet-latest.osm.pbf
[info] Input file: planet-latest.osm.pbf
[info] Profile: profile.lua
[info] Threads: 16
[info] Using script profile.lua
[STXXL-MSG] STXXL v1.4.1 (prerelease/Debug)
[STXXL-MSG] Disk '/mnt/data2/stxxl' is allocated, space: 381469 MiB, I/O implementation: syscall queue=0 devid=0
[info] Parsing in progress..
[info] input file generated by planet-dump-ng 1.1.2
[info] timestamp: 2015-11-30T01:59:42Z
[info] Using turn restrictions
[info] Found 3 exceptions to turn restrictions:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parsing finished after 9110.94 seconds
[info] Raw input contains 3125042135 nodes, 318834828 ways, and 3809663 relations, and 0 unknown entities
[extractor] Sorting used nodes        ... ok, after 1273.75s
[extractor] Erasing duplicate nodes   ... ok, after 1063.97s
[extractor] Sorting all nodes         ... ok, after 16569.4s
[extractor] Building node id map      ... ok, after 9434.62s
[extractor] setting number of nodes   ... ok
[extractor] Confirming/Writing used nodes     ... ok, after 4913.73s
[info] Processed 563024709 nodes
[extractor] Sorting edges by start    ... ok, after 7103.38s
[extractor] Setting start coords      ... ok, after 13883.9s
[extractor] Sorting edges by target   ... ok, after 8440.45s
[extractor] Computing edge weights    ... ok, after 17050.3s
[extractor] Sorting edges by renumbered start ... ^[[Bok, after 9208.92s
[extractor] Writing used edges       ... ok, after 3358.41s
[extractor] setting number of edges   ... ok
[info] Processed 594690182 edges
[extractor] Sorting used ways         ... ok, after 641.269s
[extractor] Sorting 454013 restriction. by from... ok, after 3.57648s
[extractor] Fixing restriction starts ... ok, after 153.038s
[extractor] Sorting restrictions. by to  ... ok, after 1.95642s
[extractor] Fixing restriction ends   ... ok, after 148.438s
[info] usable restrictions: 425218
[extractor] writing street name index ... ok, after 37.0033s
[info] extraction finished after 107242s
*** Error in `osrm-extract': free(): invalid size: 0x00007fc72e8d9f50 ***

[TheMarex]

Which version are you using? Latest develop?

[krk]

then latest, commit c381834

osrm-extract -v gives v4.8.1

[daniel-j-h]

For the record: source location at that commit.

This is a good start to gather more information:

• compile osrm-backend with the address sanitizer: CXXFLAGS=-fsanitize=address (probably -g -fno-omit-frame-pointer, in order to get proper stack traces, too)
• enable symbolized output: export ASAN_SYMBOLIZER_PATH=$(which llvm-symbolizer)
• run osrm-extract as usual on your dataset
• wait for the crash, copy symbolized output and paste here

[krk]

I had updated to commit 9a332d2 before running. In this commit osrm-extract finished successfully, with the following AddressSanitizer info:

==22735==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 6168 byte(s) in 3 object(s) allocated from:
#0 0x7f4b97114a32 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x99a32)
#1 0x7f4b95ed6994 (/usr/lib/libtbb.so.2+0x22994)

Direct leak of 162 byte(s) in 9 object(s) allocated from:
#0 0x7f4b970dd76f in strdup (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x6276f)
#1 0x7f4b95ec9276 (/usr/lib/libtbb.so.2+0x15276)

SUMMARY: AddressSanitizer: 6330 byte(s) leaked in 12 allocation(s).

[TheMarex]

@krk does this work now for you?

[krk]

It was working until I had to restart the server. Now executing osrm-routed gives the following warning.

[warn] Shared memory settings conflict with path settings.

[TheMarex]

@krk how do you start the server?

[krk]

without any parameters, while in the folder that contains processed files.

[daniel-j-h]

From the error message I see that you are specifying both 1/ the base path to an .osrm file and 2/ the shared memory option:

osrm-backend/include/util/routed_options.hpp

Lines 253 to 256 in f682476

	else if (use_shared_memory && option_variables.count("base"))
	{
	SimpleLogger().Write(logWARNING) << "Shared memory settings conflict with path settings.";
	}

Maybe you have a ini configuration file where these parameters are coming from.

Please try without the shared memory option.

[MoKob]

@krk any news on this? If your problem is fixed, feel free to close here.

[krk]

There was no ini file, I wasn't supplying any shared memory options, problem was not fixed.

Closing this issue, will open a new one when I try a later version or a stable version.

Thanks for your quick responses.