signal SIGSEGV, Segmentation fault in osrm-prepare #388

UmpPCPL · 2012-08-27T21:04:47Z

Linux Ubuntu 12.
Many successful OSRM builds.

Now I have SIGSEGV in osrm-prepare. (27.08.2012 22:30 clone from github)

Input file
http://ump.torch.net.pl/osm/2012-08-26/UMP-PL-CCBySA_20120826.osm.bz2

Output:

ump@devbuntu:~/work/Auto-PRD5$ gdb ./osrm-extract
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
http://bugs.launchpad.net/gdb-linaro/...
Reading symbols from /mnt/storage/ump/Auto-PRD5/osrm-extract...(no debugging symbols found)...done.
(gdb) r DATA/UMP-PL-CCBySA_20120826.osm.bz2
Starting program: /mnt/storage/ump/Auto-PRD5/osrm-extract DATA/UMP-PL-CCBySA_20120826.osm.bz2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
[STXXL-MSG] STXXL v1.3.1 (release)
[STXXL-ERRMSG] Warning: no config file found.
[STXXL-ERRMSG] Using default disk configuration.
[STXXL-MSG] 1 disks are allocated, total space: 1000 MiB
[info extractor.cpp:86] extracting data from input file DATA/UMP-PL-CCBySA_20120826.osm.bz2
[info extractor.cpp:115] Loading speed profiles
[info extractor.cpp:117] Found the following speed profiles:
[0]car
[1]bike
[info extractor.cpp:125] Using profile "car"
[info extractor.cpp:159] adding parking_aisle to accessRestrictedService
[info extractor.cpp:168] adding destination to accessRestrictionKeys
[info extractor.cpp:168] adding private to accessRestrictionKeys
[info extractor.cpp:168] adding delivery to accessRestrictionKeys
[info extractor.cpp:168] adding permissive to accessRestrictionKeys
[info extractor.cpp:177] adding no to accessForbiddenKeys
[info extractor.cpp:177] adding private to accessForbiddenKeys
[info extractor.cpp:177] adding agricultural to accessForbiddenKeys
[info extractor.cpp:177] adding forestery to accessForbiddenKeys
[info extractor.cpp:186] adding track to accessForbiddenDefault
[info extractor.cpp:206] Using 2 GB of RAM for buffers
[warn DataStructures/XMLParser.h:35] Parsing plain .osm/.osm.bz2 is deprecated. Switch to .pbf
[New Thread 0x9a0bfb40 (LWP 15886)]
[extractor] parsing finished after 159.516 seconds
[extractor] Sorting used nodes ... ok, after 0.672561s
[extractor] Erasing duplicate nodes ... ok, after 0.205156s
[extractor] Sorting all nodes ... ok, after 1.83004s
[extractor] Sorting used ways ... ok, after 0.207546s
[extractor] Sorting restrctns. by from... ok, after 0.269592s
[extractor] Fixing restriction starts ... ok, after 0.334752s
[extractor] Sorting restrctns. by to ... ok, after 0.032901s
[extractor] Fixing restriction ends ... ok, after 0.0321639s
[info extractor.cpp:332] usable restrictions: 2669
[extractor] Confirming/Writing used nodes ... ok, after 1.5987s
[extractor] setting number of nodes ... ok
[extractor] Sorting edges by start ... ok, after 3.24775s
[extractor] Setting start coords ... ok, after 2.35303s
[extractor] Sorting edges by target ... ok, after 3.75195s
[extractor] Setting target coords ... ok, after 6.30218s
[extractor] setting number of edges ... ok
[extractor] writing street name index ... ok, after 0.00642395s
[info extractor.cpp:518] Processed 16243.5 nodes/sec and 17642.8 edges/sec
[info extractor.cpp:522] [extractor] finished.

Run:
./osrm-prepare DATA/UMP-PL-CCBySA_20120826.osrm DATA/UMP-PL-CCBySA_20120826.osrm.restrictions
[Thread 0x9a0bfb40 (LWP 15886) exited]
Inferior 1 (process 15882) exited normally quit

ump@devbuntu:~/work/Auto-PRD5$ gdb ./osrm-prepare
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
http://bugs.launchpad.net/gdb-linaro/...
Reading symbols from /mnt/storage/ump/Auto-PRD5/osrm-prepare...(no debugging symbols found)...done.
(gdb) r DATA/UMP-PL-CCBySA_20120826.osrm DATA/UMP-PL-CCBySA_20120826.osrm.restrictions
Starting program: /mnt/storage/ump/Auto-PRD5/osrm-prepare DATA/UMP-PL-CCBySA_20120826.osrm DATA/UMP-PL-CCBySA_20120826.osrm.restrictions
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
[info createHierarchy.cpp:83] Loading SRTM from/to /opt/storage/srtm/Eurasia
[info createHierarchy.cpp:86] Using restrictions from file: DATA/UMP-PL-CCBySA_20120826.osrm.restrictions
[info Util/GraphLoader.h:189] Graph loaded ok and has 3187553 edges
[info createHierarchy.cpp:114] 2669 restrictions, 377 bollard nodes, 0 traffic lights
[info createHierarchy.cpp:126] Generating edge-expanded graph representation
[info Contractor/EdgeBasedGraphFactory.cpp:208] Identifying small components
. 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
[info Contractor/EdgeBasedGraphFactory.cpp:262] identified: 1986 many components
. 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
[info Contractor/EdgeBasedGraphFactory.cpp:360] Node-based graph contains 14297241 edges
[info Contractor/EdgeBasedGraphFactory.cpp:362] Edge-based graph skipped 2626 turns, defined by 2669 restrictions.
[info Contractor/EdgeBasedGraphFactory.cpp:363] Generated 6172484 edge based nodes
[info createHierarchy.cpp:144] writing node map ...
[info createHierarchy.cpp:153] writing info on original edges
[info createHierarchy.cpp:173] building grid ...
[STXXL-MSG] STXXL v1.3.1 (release)
[STXXL-ERRMSG] Warning: no config file found.
[STXXL-ERRMSG] Using default disk configuration.
[STXXL-MSG] 1 disks are allocated, total space: 1000 MiB
. 10% . 20% [New Thread 0xb74e8b40 (LWP 16233)]
. 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
[info DataStructures/NNGrid.h:109] sorting grid data consisting of 7364937 edges...
[info DataStructures/NNGrid.h:112] finished sorting after 6.76166s
writing data .... 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
softbased

Program received signal SIGSEGV, Segmentation fault.
0x08097a42 in CRC32::SoftwareBasedCRC32(char*, unsigned int, unsigned int) ()
(gdb)

DennisOSRM · 2012-08-29T19:38:48Z

Do you have a stack trace? The gdb command is bt

UmpPCPL · 2012-08-29T20:30:20Z

(gdb) bt
#0 0x08097a42 in CRC32::SoftwareBasedCRC32(char_, unsigned int, unsigned int)
()
#1 0x08097a0c in CRC32::operator()(char_, unsigned int) ()
#2 0x08051442 in main ()
(gdb)

I can compile with debug info if it helps.

I have now two osrm build cloned at 22.08 and 27.08.
And two data set dated 22.08 and 26.08.
22.08 build processes successfully data from 22.08 and 26.08
27.08 consistently crushes on 22.08 and 26.08 data.

Is it something more I can provide to help you ?

Michal

UmpPCPL · 2012-08-29T20:49:03Z

With debug info compiled in:

Program received signal SIGSEGV, Segmentation fault.
0x080c1795 in boost::crc_optimal<32u, 517762881u, 0u, 0u, true, true>::process_block (
this=0xbfffd3dc, bytes_begin=0x8b159d8, bytes_end=0x12fe8548)
at /usr/include/boost/crc.hpp:962
962 unsigned char const byte_index = helper_type::index( rem_, *p );
(gdb) bt
#0 0x080c1795 in boost::crc_optimal<32u, 517762881u, 0u, 0u, true, true>::process_block (
this=0xbfffd3dc, bytes_begin=0x8b159d8, bytes_end=0x12fe8548)
at /usr/include/boost/crc.hpp:962
#1 0x080c1694 in boost::crc_optimal<32u, 517762881u, 0u, 0u, true, true>::process_bytes (
this=0xbfffd3dc, buffer=0x8b159d8, byte_count=172829552)
at /usr/include/boost/crc.hpp:981
#2 0x080c13f4 in CRC32::SoftwareBasedCRC32 (this=0xbfffd484, str=0x8b159d8 "",
len=172829552, crc=0) at Algorithms/CRC32.cpp:29
#3 0x080c15de in CRC32::operator() (this=0xbfffd484, str=0x8b159d8 "", len=172829552)
at Algorithms/CRC32.cpp:83
#4 0x0805a367 in main (argc=3, argv=0xbffff774) at createHierarchy.cpp:178
(gdb)

DennisOSRM · 2012-08-31T08:09:48Z

I cannot reproduce this bug with the latest master branch and your file from above. Does the error still occur on your end.

Interestingly, the bug happens in boost code to compute a hash value.

UmpPCPL · 2012-08-31T08:56:41Z

Now works :-8. Close please. Probably bug in boost.

UmpPCPL · 2012-09-01T19:24:14Z

Hi Dennis
I still have the same problem but have more information now.
Problem is connected with 32 bit OS.

I have installed 64 bit ubuntu and there it works fine
but with 32 bit always fails.

What is interesting the same error I have at machine where SSE is available.

Now it is not in boost code but near your assembler code.

This is trace from VM where SSE was enabled.

[info DataStructures/NNGrid.h:109] sorting grid data consisting of 7375584 edges...
[info DataStructures/NNGrid.h:112] finished sorting after 35.0053s
writing data .... 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
sse based

Program received signal SIGSEGV, Segmentation fault.
0x080c1426 in CRC32::SSEBasedCRC32 (this=0xbfffd0a4, str=0xb6612008 "",
len=173112884, crc=1148545018) at Algorithms/CRC32.cpp:44
44 );
(gdb) bt
#0 0x080c1426 in CRC32::SSEBasedCRC32 (this=0xbfffd0a4, str=0xb6612008 "",
len=173112884, crc=1148545018) at Algorithms/CRC32.cpp:44
#1 0x080c15de in CRC32::operator() (this=0xbfffd0a4, str=0xb6612008 "",
len=173112884) at Algorithms/CRC32.cpp:83
#2 0x0805a367 in main (argc=3, argv=0xbffff394) at createHierarchy.cpp:176
(gdb)

DennisOSRM · 2012-09-01T19:41:47Z

Thanks for investigating. It totally makes sense since the assembler
instructions that are used may not be available on a 32 Bit system.

On 01.09.2012 21:24, UmpPCPL wrote:

Hi Dennis
I still have the same problem but have more information now.
Problem is connected with 32 bit OS.

I have installed 64 bit ubuntu and there it works fine but with 32 bit
always fails.

What is interesting the same error I have at machine where SSE is available.

Now it is not in boost code but near your assembler code.

This is trace from VM where SSE was enabled.

[info DataStructures/NNGrid.h:109] sorting grid data consisting of
7375584 edges...
[info DataStructures/NNGrid.h:112] finished sorting after 35.0053s
writing data .... 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
sse based

Program received signal SIGSEGV, Segmentation fault.
0x080c1426 in CRC32::SSEBasedCRC32 (this=0xbfffd0a4, str=0xb6612008 "",
len=173112884, crc=1148545018) at Algorithms/CRC32.cpp:44
44 );
(gdb) bt
#0 0x080c1426 in CRC32::SSEBasedCRC32 (this=0xbfffd0a4, str=0xb6612008 "",
len=173112884, crc=1148545018) at Algorithms/CRC32.cpp:44
#1 #1 0x080c15de in
CRC32::operator() (this=0xbfffd0a4, str=0xb6612008 "",
len=173112884) at Algorithms/CRC32.cpp:83
#2 #2 0x0805a367 in
main (argc=3, argv=0xbffff394) at createHierarchy.cpp:176
(gdb)

—
Reply to this email directly or view it on GitHub
#388 (comment).

UmpPCPL · 2012-09-01T20:17:16Z

Problem is not lack of instructions.
The same crash i observe with and without SSE.
In my opinion something is wrong in datasize calculation after that commit

"Using DeallocatingVector class instead of doing vector-swap-tricks"
417fcde

it changed type of nodeBasedEdgeList in CreateHierarchy.cpp line 164

- std::vectorEdgeBasedGraphFactory::EdgeBasedNode nodeBasedEdgeList;
- DeallocatingVectorEdgeBasedGraphFactory::EdgeBasedNode nodeBasedEdgeList;

form this point 32bit version is broken,

DennisOSRM · 2012-09-01T20:25:33Z

Cool investigating! I will look for a fix when I arrive in Japan for
State Of The Map.

On 01.09.2012 22:17, UmpPCPL wrote:

Problem is not lack of instructions.
The same crash i observe with and without SSE.
In my opinion something is wrong in datasize calculation after that commit

"Using DeallocatingVector class instead of doing vector-swap-tricks"
417fcde

it changed type of nodeBasedEdgeList in CreateHierarchy.cpp line 164

std::vectorEdgeBasedGraphFactory::EdgeBasedNode nodeBasedEdgeList;

DeallocatingVectorEdgeBasedGraphFactory::EdgeBasedNode
nodeBasedEdgeList;

form this point 32bit version is broken,

—
Reply to this email directly or view it on GitHub
#388 (comment).

AntonZag · 2012-09-17T08:54:27Z

Hello guys!
I spend whole weekend trying to get this working on
MS VisualStudio and found that error in DeallocatingVector has nothing to do with
32 or 64 bits. It has following error:

Postfix increment and decrement operators are meant to return OLD value
and only then increment and decrement themselves. And in code now
new position is returned and iterator preserves old position:

    inline DeallocatingVectorIterator operator++(int) { //postfix
        DeallocatingVectorIteratorState _myState(mState);
        _myState.mIndex++; _myState.setPointerForIndex();
        return DeallocatingVectorIterator(_myState);
    }
    inline DeallocatingVectorIterator operator --(int) { //postfix
        if(DeallocateC) assert(false);
        DeallocatingVectorIteratorState _myState(mState);
        _myState.mIndex--; _myState.setPointerForIndex();
        return DeallocatingVectorIterator(_myState);
    }

That breaks C library's std::sort algorithm. (Maybe 64-bit GCC has
completely different sort implementation without postcrements). In order to fix it, change
to following code:

    inline DeallocatingVectorIterator operator++(int) { //postfix
        DeallocatingVectorIteratorState _myState(mState);
        mState.mIndex++; mState.setPointerForIndex();
        return DeallocatingVectorIterator(_myState);
    }
    inline DeallocatingVectorIterator operator --(int) { //postfix
        if(DeallocateC) assert(false);
        DeallocatingVectorIteratorState _myState(mState);
        mState.mIndex--; mState.setPointerForIndex();
        return DeallocatingVectorIterator(_myState);
    }

Hope that does help. If not, I can check if I modified anything else.
Best regards,
Anton Z.

emiltin · 2012-09-17T09:00:15Z

interesting. i experience segfaults in osrm-prepare on mac when running the cucumber tests. i wonder if this could be related.

UmpPCPL · 2012-09-17T09:20:56Z

AntonZag. Did you have the same stack trace like mine ?

AntonZag · 2012-09-17T09:36:07Z

No, that is on 32-bit windows, and I also got that CRC32 error later, which is explained in Issue #414.
That got through past CRC32 calculation.
Now got some error on freeing memory on 200Mb+ .osrm file.
I intentionally lowered 'bucket' size in DeallocaingVector and because of that,
probability of non-contiguous memory allocation and different errors due to this increased greatly.

Small data can be processed with little memory error crash probability, but we need stable code, right? :)

UmpPCPL · 2012-09-17T09:44:04Z

Anton
You are right it is problem with non-contiguous memory allocation.
After your findings it will be easy fix for Dennis.

DennisOSRM · 2012-09-17T13:40:56Z

Fix will be available shortly and it should also help with this issue.

potentially a few other.

sauerworld · 2012-09-17T14:56:38Z

Updating to master fixed this for me.

DennisOSRM · 2012-09-17T14:58:09Z

Thanks for the feedback.

DennisOSRM · 2012-09-18T15:38:41Z

Thanks for pointing out the problems with the postfix operators. Indeed, GCC seems implement sorting around these operators, but clang does not. Working on a fix now.

UmpPCPL closed this as completed Aug 31, 2012

UmpPCPL reopened this Sep 1, 2012

DennisOSRM pushed a commit that referenced this issue Sep 17, 2012

Iterator based CRC32 computation. Fixes issues #388, #414 and

ae8dfcb

potentially a few other.

DennisOSRM closed this as completed Sep 17, 2012

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

signal SIGSEGV, Segmentation fault in osrm-prepare #388

signal SIGSEGV, Segmentation fault in osrm-prepare #388

UmpPCPL commented Aug 27, 2012

DennisOSRM commented Aug 29, 2012

UmpPCPL commented Aug 29, 2012

UmpPCPL commented Aug 29, 2012

DennisOSRM commented Aug 31, 2012

UmpPCPL commented Aug 31, 2012

UmpPCPL commented Sep 1, 2012

DennisOSRM commented Sep 1, 2012

UmpPCPL commented Sep 1, 2012

DennisOSRM commented Sep 1, 2012

AntonZag commented Sep 17, 2012

emiltin commented Sep 17, 2012

UmpPCPL commented Sep 17, 2012

AntonZag commented Sep 17, 2012

UmpPCPL commented Sep 17, 2012

DennisOSRM commented Sep 17, 2012

sauerworld commented Sep 17, 2012

DennisOSRM commented Sep 17, 2012

DennisOSRM commented Sep 18, 2012

signal SIGSEGV, Segmentation fault in osrm-prepare #388

signal SIGSEGV, Segmentation fault in osrm-prepare #388

Comments

UmpPCPL commented Aug 27, 2012

DennisOSRM commented Aug 29, 2012

UmpPCPL commented Aug 29, 2012

UmpPCPL commented Aug 29, 2012

DennisOSRM commented Aug 31, 2012

UmpPCPL commented Aug 31, 2012

UmpPCPL commented Sep 1, 2012

DennisOSRM commented Sep 1, 2012

UmpPCPL commented Sep 1, 2012

DennisOSRM commented Sep 1, 2012

AntonZag commented Sep 17, 2012

emiltin commented Sep 17, 2012

UmpPCPL commented Sep 17, 2012

AntonZag commented Sep 17, 2012

UmpPCPL commented Sep 17, 2012

DennisOSRM commented Sep 17, 2012

sauerworld commented Sep 17, 2012

DennisOSRM commented Sep 17, 2012

DennisOSRM commented Sep 18, 2012