Reduce memory in stxxl

[oxidase]

Issue

Issue #77 introduced a regression in memory consumption in InternalExtractorEdge. This PR aimed to reduce memory consumption by using float type for the value and avoid the variant type. The float type can be used with assumption that initial edge weights and duration values from profiles have 24 significant bits (about 7 decimal digits). This is pretty safe assumption, because these values are used as 31 significant bits integers and summing up without precision loss 128 maximum possible values will lead to signed integer overflow. To route for over 128 edges without risk of the integer overflow edge weights should have less than 24 significant bits.
The variant type introduces 8 bytes overhead by std::size_t type_index and results to sizeof(mapbox::util::variant<osrm::extractor::detail::ValueByEdge, osrm::extractor::detail::ValueByMeter>)
= 16 bytes per WeightData or 32 bytes per edge. Because there are only two types ValueByEdge that is always non-negative and ValueByMeter that is always strictly positive the sign bit can be used to encode the value type in a single value without values domains overlap.

Structure sizes:

• before merging Allow abitrary edge weights #2399 was

sizeof(osrm::extractor::InternalExtractorEdge::WeightData) = 16
sizeof(osrm::extractor::NodeBasedEdgeWithOSM) = 40
sizeof(osrm::extractor::InternalExtractorEdge) = 64
ptype osrm::extractor::InternalExtractorEdge
type = struct osrm::extractor::InternalExtractorEdge {
    osrm::extractor::NodeBasedEdgeWithOSM result;
    osrm::extractor::InternalExtractorEdge::WeightData weight_data;
    osrm::util::Coordinate source_coordinate;

• after merging Allow abitrary edge weights #2399

sizeof(osrm::extractor::InternalExtractorEdge::WeightData) = 16
sizeof(osrm::extractor::NodeBasedEdgeWithOSM) = 48
sizeof(osrm::extractor::InternalExtractorEdge) = 88
ptype osrm::extractor::InternalExtractorEdge
type = struct osrm::extractor::InternalExtractorEdge {
    osrm::extractor::NodeBasedEdgeWithOSM result;
    WeightData weight_data;
    WeightData duration_data;
    osrm::util::Coordinate source_coordinate;

• with the PR

sizeof(osrm::extractor::InternalExtractorEdge::WeightData) = 4
sizeof(osrm::extractor::NodeBasedEdgeWithOSM) = 44
sizeof(osrm::extractor::InternalExtractorEdge) = 60
ptype osrm::extractor::InternalExtractorEdge
type = struct osrm::extractor::InternalExtractorEdge {
    osrm::extractor::NodeBasedEdgeWithOSM result;
    WeightData weight_data;
    WeightData duration_data;
    osrm::util::Coordinate source_coordinate;

So PR saves 28 bytes per edge wrt to the current master (>11GB) and 4 bytes wrt master before #2399 merge (1.6GB) for raw 392418286 input ways.

Tasklist

• review
• adjust for comments

Requirements / Relations

The PR is rebased on top of #3629, so only e6c884d commit is relevant for review.

[TheMarex]

Needs a rebase on master now that #3632 is merged otherwise this looks good to me.

[TheMarex]

Elegant solution. Precision should be more than enough.

[oxidase]

rebased. with the change i see only two lines [STXXL-ERRMSG] External memory block allocation error: 2145386496 bytes requested, 1163919360 bytes free. Trying to extend the external memory space...
on the planet size and extraction + contraction time is 14 hours