Performance improvements: Faster Overpass queries #1514

westnordost · 2019-08-01T18:44:32Z

Some Overpass queries are very slow and clog up the download process by pitching the app in a long queue, waiting to replenish its Overpass download quota. If the queries were executed faster, the app would reach the quota later.

So:

Find out which quests take long and why. First step here is to create a little program in the tests/ directory that iterates through the quest type list and outputs the compiled overpass query for each for some bbox to console. See Initial quest scan may not found any quests before running out of query quota #1457 (comment). The output of this program would also be interesting for @rugk who is maintaining the quests wiki page and links to the used overpass queries for each quest type.
Executing each query on some location should give an idea which quest types are problematic.
Implement optimizations based on these observations. In the predecessor ticket to this one, a few suggestions have already been made:
1. make possible the usage of regex in keys in the TQL syntax and use it where applicable
2. express nested ors in OQL with if: <condition> blocks if they don't contain regex
3. Far the biggest impact though has some fiddling of evaluation strategies for Overpass Queries. At the moment, it is unknown what the best strategy would be here. See Initial quest scan may not found any quests before running out of query quota #1457 (comment) and /cc @mmd-osm

The text was updated successfully, but these errors were encountered:

westnordost · 2019-08-02T21:07:33Z

@mmd-osm Regarding putting the name later: You said in #1457 (comment)

Different filter criteria are not executed in the exact sequence given in the query, in particular they are grouped by the type of filter first. Different types are: key only, exact key value, key w/ negated value, key w/ value regex, key w/ negated value regex, key regex w/ value regex. There’s a heuristic in place which aims at cutting down the number of objects as early as possible.

If the heuristic is there to cut down the number of objects as early as possible, why does it not work in the cases mentioned above? Is it because the grouping by filter type comes first and only then within these groups, the filters are sorted by that heuristic?

If I understand correctly, this query would not be very optimized because it pulls in first all elements with no name, even though pulling in only elements with superspecialkey would be the better strategy.

way[superspecialkey ~ "right|okay"][!name];

Did I understand this correctly?
If yes, isn't this then a topic that should better be handled upstream (at Overpass)?

mmd-osm · 2019-08-02T21:44:45Z

Is it because the grouping by filter type comes first and only then within these groups, the filters are sorted by that heuristic?

No, what I meant by the heuristic is a certain sequence in which those different groups are being evaluated. Although a bit dated and somewhat high level, Roland gave a presentation at FOSSGIS2013 on how the evaluation works, starting on slide 23: https://www.fossgis.de/konferenz/2013/programm/attachments/428_workshop.odp / https://www.fossgis.de/konferenz/2013/programm/events/520.de.html

Some of the filters take size of the bounding box into account, or switch evaluation strategy on the fly, if one approach produces too much data during evaluation. I'm not aware of a concise description on all the details (and there are many more, which I didn't mention yet), except for the source code.

If I understand correctly, this query would not be very optimized because it pulls in first all elements with no name, even though pulling in only elements with superspecialkey would be the better strategy.

The major issue here is that there's no statistical information available on local or even global tag distribution and frequency, i.e. it is not known up front that superspecialkey is quite rare, and name is very frequent.

By forcing a different evaluation sequence, I'm bringing in my own knowledge about tag frequency - which should be fairly universal in case of a name tag. On the other hand, this makes it hard to generalize this optimization approach, as you would need to keep some statistical information around to decide if this is worthwhile or even counterproductive.

westnordost · 2019-08-02T22:10:52Z

The major issue here is that there's no statistical information available on local or even global tag distribution and frequency, i.e. it is not known up front that superspecialkey is quite rare, and name is very frequent.

But that could be done, in the future, right? A server-local taginfo of the like. After all, the data is all already there and overpass itself has the best means to acquire this data (quickly).

rugk · 2019-08-03T08:59:56Z

Uhm, did I miss something or how did you extract the compiled overpass queries now?
In the OP you stated:

First step here is to create a little program in the tests/ directory that iterates through the quest type list and outputs the compiled overpass query for each for some bbox to console.

… but then you just came up with the results (out of nowhere). I also checked the tests dir and could not find anything… 🤔

westnordost · 2019-08-03T09:07:41Z

I wrote it, but did not commit it yet, work in progress. If you intend to copy&paste the queries to the wiki page, better wait until I am through with the optimizations. Minus imports, it is less than 30 lines of code.

westnordost · 2019-08-04T22:06:23Z

It's quite a brain teaser to find an algorithm to convert any boolean expression to overpass syntax. Though I may have cracked it now. Here is a teaser of an automatically generated query:

AddBusStopName

Tag Filters

nodes with
((public_transport = platform and ~bus|trolleybus|tram ~ yes)
or
(highway = bus_stop and public_transport != stop_position))
and !name and noname != yes

[bbox:{{bbox}}];
node[!name][noname != yes] -> .n1;
node.n1[public_transport = platform][~'^bus|trolleybus|tram$' ~ '^yes$'] -> .n2;
node.n1[highway = bus_stop][public_transport != stop_position] -> .n3;
(.n2; .n3;) -> .n1;
out meta geom;

mmd-osm · 2019-08-05T04:59:58Z

Both queries look surprisingly similar 😎

A quick comment,though: You really need to double check if moving the second line (producing .n1) towards the end helps avoiding excessive memory consumption, see #1514 (comment)

Example:

[bbox:50.04559282551578,19.9237060546875,50.070472406781924,19.967780113220215];
node[!name][noname != yes] -> .n1;
node.n1[public_transport = platform][~'^bus|trolleybus|tram$' ~ '^yes$'] -> .n2;
node.n1[highway = bus_stop][public_transport != stop_position] -> .n3;
.n1 out count;
.n2 out count;
.n3 out count;

.n1 has 172850 nodes in this case... this will not really fly. node[!name][noname != yes] is a very weak filter, and will include almost all nodes in a bounding box, even all those nodes that don't have any tags at all(!). The idea should be to move weak filters to the end, when the strong filters already removed most of the data.

vs:

[bbox:50.04559282551578,19.9237060546875,50.070472406781924,19.967780113220215];

(node[public_transport = platform][~'^bus|trolleybus|tram$' ~ '^yes$'];
 node[highway = bus_stop][public_transport != stop_position];
 )->.n2;
node.n2[!name][noname != yes] -> .n1;
.n1 out count;
.n2 out count;

westnordost · 2019-08-05T19:14:02Z

Alrighty, this is the final syntax:

AddRoadName

[bbox:{{bbox}}];
way[highway~"^(primary|secondary|tertiary|unclassified|residential|living_street|pedestrian)$"][!name][!ref][noname != yes][!junction][area != yes]->.unnamed;
(
  way.unnamed['access' !~ '^private|no$'];
  way.unnamed['foot']['foot' !~ '^private|no$'];
); out meta geom;

[bbox:{{bbox}}];
way[highway~"^(primary|secondary|tertiary|unclassified|residential|living_street|pedestrian)$"][!name][!ref][noname != yes][!junction][area != yes] -> .without_names;
way[highway~"^(primary|secondary|tertiary|unclassified|residential|living_street|pedestrian)$"][name] -> .with_names;
way.with_names(around.without_names: 30.0 );
out body geom;

AddPlaceName

[bbox:{{bbox}}];
nwr[shop][shop !~ '^no|vacant$'] -> .e2;
nwr[craft] -> .e3;
nwr[office] -> .e4;
nwr[tourism = information][information = office] -> .e5;
nwr[amenity ~ '^restaurant|cafe|ice_cream|fast_food|bar|pub|biergarten|food_court|nightclub|cinema|theatre|planetarium|arts_centre|studio|events_venue|conference_centre|exhibition_centre|music_venue|townhall|prison|courthouse|embassy|police|fire_station|ranger_station|bank|bureau_de_change|money_transfer|post_office|library|marketplace|internet_cafe|community_centre|social_facility|nursing_home|childcare|retirement_home|social_centre|youth_centre|car_wash|car_rental|boat_rental|fuel|ferry_terminal|dentist|doctors|clinic|pharmacy|hospital|place_of_worship|monastery|kindergarten|school|college|university|research_institute|driving_school|dive_centre|language_school|music_school|casino|brothel|gambling|love_hotel|stripclub|animal_boarding|animal_shelter|animal_breeding|veterinary$'] -> .e6;
nwr[tourism ~ '^attraction|zoo|aquarium|theme_park|gallery|museum|hotel|guest_house|motel|hostel|alpine_hut|apartment|resort|camp_site|caravan_site|chalet$'] -> .e7;
nwr[leisure ~ '^nature_reserve|sports_centre|fitness_centre|dance|golf_course|water_park|miniature_golf|stadium|marina|bowling_alley|amusement_arcade|adult_gaming_centre|tanning_salon|horse_riding$'] -> .e8;
(.e2; .e3; .e4; .e5; .e6; .e7; .e8;) -> .e1;
nwr.e1[!name][!brand][noname != yes];
out meta geom;

AddOneway

AddBusStopName

Tag Filters

nodes with
((public_transport = platform and ~bus|trolleybus|tram ~ yes)
or
(highway = bus_stop and public_transport != stop_position))
and !name and noname != yes

[bbox:{{bbox}}];
node[public_transport = platform][~'^bus|trolleybus|tram$' ~ '^yes$'] -> .n2;
node[highway = bus_stop][public_transport != stop_position] -> .n3;
(.n2; .n3;) -> .n1;
node.n1[!name][noname != yes];
out meta geom;

AddIsBuildingUnderground

Tag Filters

ways, relations with building and !location and layer~-[0-9]+

[bbox:{{bbox}}];
way[building][!location][layer ~ '^-[0-9]+$'] -> .w1;
rel[building][!location][layer ~ '^-[0-9]+$'] -> .r1;
(.w1; .r1;);
out meta geom;

AddHousenumber

(
  way['building'~'^(house|residential|apartments|detached|terrace|dormitory|semi|semidetached_house|farm|school|civic|college|university|public|hospital|kindergarten|train_station|hotel|retail|commercial)$'][location!=underground][ruins!=yes][!'addr:housenumber'][!'addr:housename'][!'addr:conscriptionnumber'][!'addr:streetnumber'][!noaddress]({{bbox}});
  rel['building'~'^(house|residential|apartments|detached|terrace|dormitory|semi|semidetached_house|farm|school|civic|college|university|public|hospital|kindergarten|train_station|hotel|retail|commercial)$'][location!=underground][ruins!=yes][!'addr:housenumber'][!'addr:housename'][!'addr:conscriptionnumber'][!'addr:streetnumber'][!noaddress]({{bbox}});
) -> .buildings;
.buildings > -> .building_nodes;
node.building_nodes[~'^addr:(housenumber|housename|conscriptionnumber|streetnumber)$'~'.']; < -> .buildings_with_addr_nodes;
(.buildings; - .buildings_with_addr_nodes;);
out meta geom;

MarkCompletedHighwayConstruction

[bbox:{{bbox}}];way[highway = construction](if:!is_date(t['opening_date']) || date(t['opening_date']) < date('2019-09-01T00:00:00Z')) -> .construction_with_unknown_state;
(
  way[highway=construction](newer: '2019-08-18T00:00:00Z');
  relation[highway=construction](newer: '2019-08-18T00:00:00Z');
) -> .recently_edited_construction;
(.construction_with_unknown_state; - .recently_edited_construction;) -> .roads_for_review;
.roads_for_review out meta geom;

AddReligionToPlaceOfWorship

Tag Filters

nodes, ways, relations with amenity=place_of_worship and !religion and name

[bbox:{{bbox}}];
nwr[amenity = place_of_worship][!religion][name];
out meta geom;

AddParkingAccess

Tag Filters

nodes, ways, relations with amenity=parking and (!access or access=unknown)

[bbox:{{bbox}}];
nwr[amenity = parking] -> .e1;
nwr.e1[!access] -> .e2;
nwr.e1[access = unknown] -> .e3;
(.e2; .e3;);
out meta geom;

AddRecyclingType

Tag Filters

nodes, ways, relations with amenity = recycling and !recycling_type

[bbox:{{bbox}}];
nwr[amenity = recycling][!recycling_type];
out meta geom;

AddSport

Tag Filters

nodes, ways with leisure=pitch and
(!sport or sport ~ team_handball|hockey|skating|football )
and (access !~ private|no)

[bbox:{{bbox}}];
node[leisure = pitch] -> .n1;
node.n1[!sport] -> .n2;
node.n1[sport ~ '^team_handball|hockey|skating|football$'] -> .n3;
(.n2; .n3;) -> .n1;
node.n1[access !~ '^private|no$'] -> .n1;
way[leisure = pitch] -> .w1;
way.w1[!sport] -> .w2;
way.w1[sport ~ '^team_handball|hockey|skating|football$'] -> .w3;
(.w2; .w3;) -> .w1;
way.w1[access !~ '^private|no$'] -> .w1;
(.n1; .w1;);
out meta geom;

AddRoadSurface

Tag Filters

ways with highway ~ primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified|residential|living_street|pedestrian|track|road and !surface
and (access !~ private|no or (foot and foot !~ private|no))

[bbox:{{bbox}}];
way[highway ~ '^primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified|residential|living_street|pedestrian|track|road$'][!surface] -> .w1;
way.w1[access !~ '^private|no$'] -> .w2;
way.w1[foot][foot !~ '^private|no$'] -> .w3;
(.w2; .w3;);
out meta geom;

AddMaxSpeed

Tag Filters

ways with highway ~ motorway|motorway_link|trunk|trunk_link|primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified|residential
 and !maxspeed and !maxspeed:forward and !maxspeed:backward
 and !source:maxspeed and !zone:maxspeed and !maxspeed:type and !zone:traffic
 and surface !~ unpaved|compacted|gravel|fine_gravel|pebblestone|grass_paver|ground|earth|dirt|grass|sand|mud|ice|salt|snow|woodchips
 and motor_vehicle !~ private|no
 and vehicle !~ private|no
 and area != yes
 and (access !~ private|no or (foot and foot !~ private|no))

[bbox:{{bbox}}];
way[highway ~ '^motorway|motorway_link|trunk|trunk_link|primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified|residential$'][!maxspeed][!'maxspeed:forward'][!'maxspeed:backward'][!'source:maxspeed'][!'zone:maxspeed'][!'maxspeed:type'][!'zone:traffic'][surface !~ '^unpaved|compacted|gravel|fine_gravel|pebblestone|grass_paver|ground|earth|dirt|grass|sand|mud|ice|salt|snow|woodchips$'][motor_vehicle !~ '^private|no$'][vehicle !~ '^private|no$'][area != yes] -> .w1;
way.w1[access !~ '^private|no$'] -> .w2;
way.w1[foot][foot !~ '^private|no$'] -> .w3;
(.w2; .w3;);
out meta geom;

AddMaxHeight

[bbox:{{bbox}}];
node[barrier = height_restrictor] -> .n2;
node[amenity = parking_entrance][parking ~ '^underground|multi-storey$'] -> .n3;
(.n2; .n3;) -> .n1;
node.n1[!maxheight][!'maxheight:physical'];

out meta geom;

[bbox:{{bbox}}];
way[highway ~ '^motorway|motorway_link|trunk|trunk_link|primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified|residential|living_street|track|road$'] -> .w2;
way[highway = service][access !~ '^private|no$'][vehicle !~ '^private|no$'] -> .w3;
(.w2; .w3;) -> .w1;
way.w1[covered = yes] -> .w4;
way.w1[tunnel ~ '^yes|building_passage|avalanche_protector$'] -> .w5;
(.w4; .w5;) -> .w1;
way.w1[!maxheight][!'maxheight:physical'];

out meta geom;

AddRailwayCrossingBarrier

[bbox:{{bbox}}];
way[highway][access ~ '^private|no$'];
node(w) -> .private_roads;
way[railway ~ '^tram|abandoned$'];
node(w) -> .excluded_railways;
node[railway = level_crossing][!'crossing:barrier'];
(._; - .private_roads; );
(._; - .excluded_railways; );
out meta geom;

AddPostboxCollectionTimes

Tag Filters

nodes with amenity=post_box and !collection_times
and collection_times:signed != no and access !~ private|no

[bbox:{{bbox}}];
node[amenity = post_box][!collection_times]['collection_times:signed' != no][access !~ '^private|no$'];
out meta geom;

AddOpeningHours

Tag Filters

nodes, ways, relations with
(
 shop and shop !~ no|vacant
 or amenity = bicycle_parking and bicycle_parking = building
 or amenity = parking and parking = multi-storey
 or amenity = recycling and recycling_type = centre
 or tourism = information and information = office
 or amenity ~ restaurant|cafe|ice_cream|fast_food|bar|pub|biergarten|food_court|nightclub|cinema|planetarium|casino|library|townhall|courthouse|embassy|community_centre|youth_centre|bank|bureau_de_change|money_transfer|post_office|marketplace|internet_cafe|car_wash|car_rental|boat_rental|fuel|dentist|doctors|clinic|pharmacy|veterinary
 or tourism ~ zoo|aquarium|theme_park|gallery|museum
 or leisure ~ fitness_centre|dance|golf_course|water_park|miniature_golf|bowling_alley|horse_riding|amusement_arcade|adult_gaming_centre|tanning_salon
 or office ~ insurance|government|travel_agent|tax_advisor|religion|employment_agency
 or craft ~ carpenter|shoemaker|tailor|photographer|dressmaker|electronics_repair|key_cutter|stonemason
)
 and !opening_hours and name and opening_hours:signed != no
 and (access !~ private|no)

[bbox:{{bbox}}];
nwr[shop][shop !~ '^no|vacant$'] -> .e2;
nwr[amenity = bicycle_parking][bicycle_parking = building] -> .e3;
nwr[amenity = parking][parking = 'multi-storey'] -> .e4;
nwr[amenity = recycling][recycling_type = centre] -> .e5;
nwr[tourism = information][information = office] -> .e6;
nwr[amenity ~ '^restaurant|cafe|ice_cream|fast_food|bar|pub|biergarten|food_court|nightclub|cinema|planetarium|casino|library|townhall|courthouse|embassy|community_centre|youth_centre|bank|bureau_de_change|money_transfer|post_office|marketplace|internet_cafe|car_wash|car_rental|boat_rental|fuel|dentist|doctors|clinic|pharmacy|veterinary$'] -> .e7;
nwr[tourism ~ '^zoo|aquarium|theme_park|gallery|museum$'] -> .e8;
nwr[leisure ~ '^fitness_centre|dance|golf_course|water_park|miniature_golf|bowling_alley|horse_riding|amusement_arcade|adult_gaming_centre|tanning_salon$'] -> .e9;
nwr[office ~ '^insurance|government|travel_agent|tax_advisor|religion|employment_agency$'] -> .e10;
nwr[craft ~ '^carpenter|shoemaker|tailor|photographer|dressmaker|electronics_repair|key_cutter|stonemason$'] -> .e11;
(.e2; .e3; .e4; .e5; .e6; .e7; .e8; .e9; .e10; .e11;) -> .e1;
nwr.e1[!opening_hours][name]['opening_hours:signed' != no][access !~ '^private|no$'];
out meta geom;

AddBikeParkingCapacity

Tag Filters

nodes, ways with amenity=bicycle_parking and !capacity and access !~ private|no

[bbox:{{bbox}}];
node[amenity = bicycle_parking][!capacity][access !~ '^private|no$'] -> .n1;
way[amenity = bicycle_parking][!capacity][access !~ '^private|no$'] -> .w1;
(.n1; .w1;);
out meta geom;

AddOrchardProduce

Tag Filters

ways, relations with landuse = orchard and !trees and !produce and !crop

[bbox:{{bbox}}];
way[landuse = orchard][!trees][!produce][!crop] -> .w1;
rel[landuse = orchard][!trees][!produce][!crop] -> .r1;
(.w1; .r1;);
out meta geom;

AddCycleway

[bbox:{{bbox}}];
way[highway ~ '^(primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified)$'][area != yes][motorroad != yes][!cycleway][!'cycleway:left'][!'cycleway:right'][!'cycleway:both'][!'sidewalk:bicycle'][!'sidewalk:both:bicycle'][!'sidewalk:left:bicycle'][!'sidewalk:right:bicycle'][maxspeed !~ '^(20|15|10|8|7|6|5|10 mph|5 mph|walk)$'][surface !~ '^(unpaved|compacted|gravel|fine_gravel|pebblestone|grass_paver|ground|earth|dirt|grass|sand|mud|ice|salt|snow|woodchips)$'][bicycle != no][access !~ '^private|no$'][bicycle != use_sidepath]['bicycle:backward' != use_sidepath]['bicycle:forward' != use_sidepath] -> .streets;
(
  way[highway=cycleway](around.streets: 15);
  way[highway ~ '^(path|footway)$'](around.streets: 15);
) -> .cycleways;
way.streets(around.cycleways: 15) -> .streets_near_cycleways;
(.streets; - .streets_near_cycleways;);
out meta geom;

AddSidewalk

[bbox:{{bbox}}];
way[highway ~ '^(primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified|residential)$'][area != yes][motorroad != yes][!sidewalk][!'sidewalk:left'][!'sidewalk:right'][!'sidewalk:both'][maxspeed !~ '^(8|7|6|5|5 mph|walk)$'][surface !~ '^(unpaved|compacted|gravel|fine_gravel|pebblestone|grass_paver|ground|earth|dirt|grass|sand|mud|ice|salt|snow|woodchips)$'][lit = yes][foot != no][access !~ '^private|no$'][foot != use_sidepath] -> .streets;
way[highway ~ '^(path|footway|cycleway)$'](around.streets: 15) -> .ways;
way.streets(around.ways: 15) -> .streets_near_ways;
(.streets; - .streets_near_ways;);
out meta geom;

AddProhibitedForPedestrians

Tag Filters

ways with (
  ~'sidewalk(:both)?' ~ none|no or
  (sidewalk:left ~ none|no and sidewalk:right ~ none|no)
)
and !foot
and access !~ private|no
and motorroad != yes and highway ~ trunk|trunk_link|primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified and surface ~ paved|asphalt|cobblestone|cobblestone:flattened|sett|concrete|concrete:lanes|concrete:plates|paving_stones|metal|wood|unhewn_cobblestone and ( oneway~yes|-1 or bridge=yes or tunnel=yes or bicycle~no|use_sidepath or lit=yes )

[bbox:{{bbox}}];
way[~'^sidewalk(:both)?$' ~ '^none|no$'] -> .w2;
way['sidewalk:left' ~ '^none|no$']['sidewalk:right' ~ '^none|no$'] -> .w3;
(.w2; .w3;) -> .w1;
way.w1[!foot][access !~ '^private|no$'][motorroad != yes][highway ~ '^trunk|trunk_link|primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified$'][surface ~ '^paved|asphalt|cobblestone|cobblestone:flattened|sett|concrete|concrete:lanes|concrete:plates|paving_stones|metal|wood|unhewn_cobblestone$'] -> .w1;
way.w1[oneway ~ '^yes|-1$'] -> .w4;
way.w1[bridge = yes] -> .w5;
way.w1[tunnel = yes] -> .w6;
way.w1[bicycle ~ '^no|use_sidepath$'] -> .w7;
way.w1[lit = yes] -> .w8;
(.w4; .w5; .w6; .w7; .w8;);
out meta geom;

AddCrossingType

Tag Filters

nodes with highway=crossing and !crossing

[bbox:{{bbox}}];
node[highway = crossing][!crossing];
out meta geom;

AddBuildingLevels

Tag Filters

ways, relations with 
 building ~ house|residential|apartments|detached|terrace|dormitory|semi|semidetached_house|bungalow|school|civic|college|university|public|hospital|kindergarten|transportation|train_station|hotel|retail|commercial|office|warehouse|industrial|manufacture|parking|farm|farm_auxiliary|barn|cabin
 and !building:levels and !height and !building:height
  and !man_made and location!=underground and ruins!=yes

[bbox:{{bbox}}];
way[building ~ '^house|residential|apartments|detached|terrace|dormitory|semi|semidetached_house|bungalow|school|civic|college|university|public|hospital|kindergarten|transportation|train_station|hotel|retail|commercial|office|warehouse|industrial|manufacture|parking|farm|farm_auxiliary|barn|cabin$'][!'building:levels'][!height][!'building:height'][!man_made][location != underground][ruins != yes] -> .w1;
rel[building ~ '^house|residential|apartments|detached|terrace|dormitory|semi|semidetached_house|bungalow|school|civic|college|university|public|hospital|kindergarten|transportation|train_station|hotel|retail|commercial|office|warehouse|industrial|manufacture|parking|farm|farm_auxiliary|barn|cabin$'][!'building:levels'][!height][!'building:height'][!man_made][location != underground][ruins != yes] -> .r1;
(.w1; .r1;);
out meta geom;

AddBusStopShelter

Tag Filters

nodes with 
((public_transport = platform and ~bus|trolleybus|tram ~ yes)
or
(highway = bus_stop and public_transport != stop_position))
and !shelter and !covered

[bbox:{{bbox}}];
node[public_transport = platform][~'^bus|trolleybus|tram$' ~ '^yes$'] -> .n2;
node[highway = bus_stop][public_transport != stop_position] -> .n3;
(.n2; .n3;) -> .n1;
node.n1[!shelter][!covered];
out meta geom;

AddVegetarian

Tag Filters

nodes, ways with amenity ~ restaurant|cafe|fast_food
and name and !diet:vegetarian

[bbox:{{bbox}}];
node[amenity ~ '^restaurant|cafe|fast_food$'][name][!'diet:vegetarian'] -> .n1;
way[amenity ~ '^restaurant|cafe|fast_food$'][name][!'diet:vegetarian'] -> .w1;
(.n1; .w1;);
out meta geom;

AddVegan

Tag Filters

nodes, ways with amenity ~ restaurant|cafe|fast_food
and name and diet:vegetarian ~ yes|only and !diet:vegan

[bbox:{{bbox}}];
node[amenity ~ '^restaurant|cafe|fast_food$'][name]['diet:vegetarian' ~ '^yes|only$'][!'diet:vegan'] -> .n1;
way[amenity ~ '^restaurant|cafe|fast_food$'][name]['diet:vegetarian' ~ '^yes|only$'][!'diet:vegan'] -> .w1;
(.n1; .w1;);
out meta geom;

AddInternetAccess

Tag Filters

nodes, ways, relations with (amenity=library or tourism ~ hotel|guest_house|hostel|motel)
and !internet_access and !wifi and name

[bbox:{{bbox}}];
nwr[amenity = library] -> .e2;
nwr[tourism ~ '^hotel|guest_house|hostel|motel$'] -> .e3;
(.e2; .e3;) -> .e1;
nwr.e1[!internet_access][!wifi][name];
out meta geom;

AddParkingFee

Tag Filters

nodes, ways, relations with
amenity = parking and !fee and !fee:conditional and
access ~ yes|customers|public

[bbox:{{bbox}}];
nwr[amenity = parking][!fee][!'fee:conditional'][access ~ '^yes|customers|public$'];
out meta geom;

AddMotorcycleParkingCapacity

Tag Filters

nodes, ways with amenity=motorcycle_parking and !capacity and access !~ private|no

[bbox:{{bbox}}];
node[amenity = motorcycle_parking][!capacity][access !~ '^private|no$'] -> .n1;
way[amenity = motorcycle_parking][!capacity][access !~ '^private|no$'] -> .w1;
(.n1; .w1;);
out meta geom;

AddPathSurface

Tag Filters

ways with highway ~ path|footway|cycleway|bridleway|steps
and !surface and access !~ private|no

[bbox:{{bbox}}];
way[highway ~ '^path|footway|cycleway|bridleway|steps$'][!surface][access !~ '^private|no$'];
out meta geom;

AddTracktype

Tag Filters

ways with highway=track and !tracktype
and (access !~ private|no or (foot and foot !~ private|no))

[bbox:{{bbox}}];
way[highway = track][!tracktype] -> .w1;
way.w1[access !~ '^private|no$'] -> .w2;
way.w1[foot][foot !~ '^private|no$'] -> .w3;
(.w2; .w3;);
out meta geom;

AddForestLeafType

[bbox:{{bbox}}];(way[landuse=forest][!leaf_type](if: length()<700.0);
relation[landuse=forest][!leaf_type](if: length()<700.0);
way[natural=wood][!leaf_type](if: length()<700.0);
relation[natural=wood][!leaf_type](if: length()<700.0);
);out meta geom;

AddBikeParkingType

Tag Filters

nodes, ways with amenity=bicycle_parking and access !~ private|no and !bicycle_parking

[bbox:{{bbox}}];
node[amenity = bicycle_parking][access !~ '^private|no$'][!bicycle_parking] -> .n1;
way[amenity = bicycle_parking][access !~ '^private|no$'][!bicycle_parking] -> .w1;
(.n1; .w1;);
out meta geom;

AddWheelchairAccessToilets

Tag Filters

nodes, ways with  amenity=toilets and access !~ private|customers and !wheelchair

[bbox:{{bbox}}];
node[amenity = toilets][access !~ '^private|customers$'][!wheelchair] -> .n1;
way[amenity = toilets][access !~ '^private|customers$'][!wheelchair] -> .w1;
(.n1; .w1;);
out meta geom;

AddPlaygroundAccess

Tag Filters

nodes, ways, relations with leisure = playground and (!access or access = unknown)

[bbox:{{bbox}}];
nwr[leisure = playground] -> .e1;
nwr.e1[!access] -> .e2;
nwr.e1[access = unknown] -> .e3;
(.e2; .e3;);
out meta geom;

AddWheelchairAccessBusiness

Tag Filters

nodes, ways, relations with
(
 shop and shop !~ no|vacant
 or amenity = parking and parking = multi-storey
 or amenity = recycling and recycling_type = centre
 or tourism = information and information = office
 or amenity ~ restaurant|cafe|ice_cream|fast_food|bar|pub|biergarten|food_court|nightclub|cinema|library|theatre|arts_centre|casino|conference_centre|bank|bureau_de_change|money_transfer|post_office|internet_cafe|marketplace|police|ranger_station|courthouse|embassy|townhall|community_centre|youth_centre|car_wash|car_rental|fuel|driving_school|doctors|clinic|pharmacy|veterinary|dentist|place_of_worship
 or tourism ~ zoo|aquarium|theme_park|gallery|attraction|viewpoint|museum|hotel|guest_house|hostel|motel|apartment|chalet
 or leisure ~ golf_course|water_park|miniature_golf|dance|bowling_alley|horse_riding|sports_centre|fitness_centre|amusement_arcade|adult_gaming_centre|tanning_salon
 or office ~ insurance|government|travel_agent|tax_advisor|religion|employment_agency|lawyer|estate_agent|political_party|therapist
 or craft ~ carpenter|shoemaker|tailor|photographer|dressmaker|electronics_repair|key_cutter|stonemason|winery
) and !wheelchair and name

[bbox:{{bbox}}];
nwr[shop][shop !~ '^no|vacant$'] -> .e2;
nwr[amenity = parking][parking = 'multi-storey'] -> .e3;
nwr[amenity = recycling][recycling_type = centre] -> .e4;
nwr[tourism = information][information = office] -> .e5;
nwr[amenity ~ '^restaurant|cafe|ice_cream|fast_food|bar|pub|biergarten|food_court|nightclub|cinema|library|theatre|arts_centre|casino|conference_centre|bank|bureau_de_change|money_transfer|post_office|internet_cafe|marketplace|police|ranger_station|courthouse|embassy|townhall|community_centre|youth_centre|car_wash|car_rental|fuel|driving_school|doctors|clinic|pharmacy|veterinary|dentist|place_of_worship$'] -> .e6;
nwr[tourism ~ '^zoo|aquarium|theme_park|gallery|attraction|viewpoint|museum|hotel|guest_house|hostel|motel|apartment|chalet$'] -> .e7;
nwr[leisure ~ '^golf_course|water_park|miniature_golf|dance|bowling_alley|horse_riding|sports_centre|fitness_centre|amusement_arcade|adult_gaming_centre|tanning_salon$'] -> .e8;
nwr[office ~ '^insurance|government|travel_agent|tax_advisor|religion|employment_agency|lawyer|estate_agent|political_party|therapist$'] -> .e9;
nwr[craft ~ '^carpenter|shoemaker|tailor|photographer|dressmaker|electronics_repair|key_cutter|stonemason|winery$'] -> .e10;
(.e2; .e3; .e4; .e5; .e6; .e7; .e8; .e9; .e10;) -> .e1;
nwr.e1[!wheelchair][name];
out meta geom;

AddToiletAvailability

Tag Filters

nodes, ways with
( (shop ~ mall|department_store and name) or highway ~ services|rest_area )
and !toilets

[bbox:{{bbox}}];
node[shop ~ '^mall|department_store$'][name] -> .n2;
node[highway ~ '^services|rest_area$'] -> .n3;
(.n2; .n3;) -> .n1;
node.n1[!toilets] -> .n1;
way[shop ~ '^mall|department_store$'][name] -> .w2;
way[highway ~ '^services|rest_area$'] -> .w3;
(.w2; .w3;) -> .w1;
way.w1[!toilets] -> .w1;
(.n1; .w1;);
out meta geom;

AddFerryAccessPedestrian

Tag Filters

ways, relations with route=ferry and !foot

[bbox:{{bbox}}];
way[route = ferry][!foot] -> .w1;
rel[route = ferry][!foot] -> .r1;
(.w1; .r1;);
out meta geom;

AddFerryAccessMotorVehicle

Tag Filters

ways, relations with route=ferry and !motor_vehicle

[bbox:{{bbox}}];
way[route = ferry][!motor_vehicle] -> .w1;
rel[route = ferry][!motor_vehicle] -> .r1;
(.w1; .r1;);
out meta geom;

AddBuildingType

Tag Filters

ways, relations with building = yes
and !man_made and !historic and !military and !power and location!=underground and ruins!=yes

[bbox:{{bbox}}];
way[building = yes][!man_made][!historic][!military][!power][location != underground][ruins != yes] -> .w1;
rel[building = yes][!man_made][!historic][!military][!power][location != underground][ruins != yes] -> .r1;
(.w1; .r1;);
out meta geom;

AddWayLit

Tag Filters

ways with (
  highway ~ residential|living_street|pedestrian
  or highway ~ primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified|service and (
    sidewalk ~ both|left|right|yes|separate
    or ~source:maxspeed|maxspeed:type|zone:maxspeed|zone:traffic ~ .+:urban
  )
  or highway ~ footway|cycleway|steps
  or highway = path and (foot = designated or bicycle = designated)
)
and !lit
and (access !~ private|no or (foot and foot !~ private|no))

[bbox:{{bbox}}];
way[highway ~ '^residential|living_street|pedestrian$'] -> .w2;
way[highway ~ '^primary|primary_link|secondary|secondary_link|tertiary|tertiary_link|unclassified|service$'] -> .w3;
way.w3[sidewalk ~ '^both|left|right|yes|separate$'] -> .w4;
way.w3[~'^source:maxspeed|maxspeed:type|zone:maxspeed|zone:traffic$' ~ '^.+:urban$'] -> .w5;
(.w4; .w5;) -> .w3;
way[highway ~ '^footway|cycleway|steps$'] -> .w6;
way[highway = path] -> .w7;
way.w7[foot = designated] -> .w8;
way.w7[bicycle = designated] -> .w9;
(.w8; .w9;) -> .w7;
(.w2; .w3; .w6; .w7;) -> .w1;
way.w1[!lit] -> .w1;
way.w1[access !~ '^private|no$'] -> .w10;
way.w1[foot][foot !~ '^private|no$'] -> .w11;
(.w10; .w11;);
out meta geom;

AddToiletsFee

Tag Filters

nodes, ways with amenity = toilets and access !~ private|customers and !fee

[bbox:{{bbox}}];
node[amenity = toilets][access !~ '^private|customers$'][!fee] -> .n1;
way[amenity = toilets][access !~ '^private|customers$'][!fee] -> .w1;
(.n1; .w1;);
out meta geom;

AddBabyChangingTable

Tag Filters

nodes, ways with (((amenity ~ restaurant|cafe|fuel|fast_food or shop ~ mall|department_store)
and name and toilets=yes) or amenity=toilets) and !diaper and !changing_table

[bbox:{{bbox}}];
node[amenity ~ '^restaurant|cafe|fuel|fast_food$'] -> .n3;
node[shop ~ '^mall|department_store$'] -> .n4;
(.n3; .n4;) -> .n2;
node.n2[name][toilets = yes] -> .n2;
node[amenity = toilets] -> .n5;
(.n2; .n5;) -> .n1;
node.n1[!diaper][!changing_table] -> .n1;
way[amenity ~ '^restaurant|cafe|fuel|fast_food$'] -> .w3;
way[shop ~ '^mall|department_store$'] -> .w4;
(.w3; .w4;) -> .w2;
way.w2[name][toilets = yes] -> .w2;
way[amenity = toilets] -> .w5;
(.w2; .w5;) -> .w1;
way.w1[!diaper][!changing_table] -> .w1;
(.n1; .w1;);
out meta geom;

AddBikeParkingCover

Tag Filters

nodes, ways with amenity=bicycle_parking and access !~ private|no and !covered
and bicycle_parking !~ shed|lockers|building

[bbox:{{bbox}}];
node[amenity = bicycle_parking][access !~ '^private|no$'][!covered][bicycle_parking !~ '^shed|lockers|building$'] -> .n1;
way[amenity = bicycle_parking][access !~ '^private|no$'][!covered][bicycle_parking !~ '^shed|lockers|building$'] -> .w1;
(.n1; .w1;);
out meta geom;

AddTrafficSignalsSound

Tag Filters

nodes with highway = crossing and crossing = traffic_signals and !traffic_signals:sound

[bbox:{{bbox}}];
node[highway = crossing][crossing = traffic_signals][!'traffic_signals:sound'];
out meta geom;

AddRoofShape

Tag Filters

ways, relations with roof:levels
and roof:levels!=0 and !roof:shape and !3dr:type and !3dr:roof

[bbox:{{bbox}}];
way['roof:levels']['roof:levels' != 0][!'roof:shape'][!'3dr:type'][!'3dr:roof'] -> .w1;
rel['roof:levels']['roof:levels' != 0][!'roof:shape'][!'3dr:type'][!'3dr:roof'] -> .r1;
(.w1; .r1;);
out meta geom;

AddWheelchairAccessPublicTransport

Tag Filters

nodes, ways, relations with (amenity = bus_station or railway ~ station|subway_entrance)
and !wheelchair

[bbox:{{bbox}}];
nwr[amenity = bus_station] -> .e2;
nwr[railway ~ '^station|subway_entrance$'] -> .e3;
(.e2; .e3;) -> .e1;
nwr.e1[!wheelchair];
out meta geom;

AddWheelchairAccessOutside

Tag Filters

nodes, ways, relations with leisure=dog_park and !wheelchair

[bbox:{{bbox}}];
nwr[leisure = dog_park][!wheelchair];
out meta geom;

AddTactilePavingBusStop

Tag Filters

nodes, ways with
(public_transport = platform or (highway = bus_stop and public_transport != stop_position))
and !tactile_paving

[bbox:{{bbox}}];
node[public_transport = platform] -> .n2;
node[highway = bus_stop][public_transport != stop_position] -> .n3;
(.n2; .n3;) -> .n1;
node.n1[!tactile_paving] -> .n1;
way[public_transport = platform] -> .w2;
way[highway = bus_stop][public_transport != stop_position] -> .w3;
(.w2; .w3;) -> .w1;
way.w1[!tactile_paving] -> .w1;
(.n1; .w1;);
out meta geom;

AddTactilePavingCrosswalk

Tag Filters

nodes with highway = crossing and !tactile_paving and foot != no

[bbox:{{bbox}}];
node[highway = crossing][!tactile_paving][foot != no];
out meta geom;

AddBridgeStructure

Tag Filters

ways with man_made=bridge and !bridge:structure

[bbox:{{bbox}}];
way[man_made = bridge][!'bridge:structure'];
out meta geom;

AddReligionToWaysideShrine

Tag Filters

nodes, ways, relations with historic=wayside_shrine and !religion and (access !~ private|no)

[bbox:{{bbox}}];
nwr[historic = wayside_shrine][!religion][access !~ '^private|no$'];
out meta geom;

AddCyclewaySegregation

Tag Filters

ways with
(
  (highway = path and bicycle = designated and foot = designated)
  or (highway = footway and bicycle = designated)
  or (highway = cycleway and foot ~ designated|yes)
)
and !segregated and area != yes
and surface ~ paved|asphalt|cobblestone|cobblestone:flattened|sett|concrete|concrete:lanes|concrete:plates|paving_stones|metal|wood|unhewn_cobblestone

[bbox:{{bbox}}];
way[highway = path][bicycle = designated][foot = designated] -> .w2;
way[highway = footway][bicycle = designated] -> .w3;
way[highway = cycleway][foot ~ '^designated|yes$'] -> .w4;
(.w2; .w3; .w4;) -> .w1;
way.w1[!segregated][area != yes][surface ~ '^paved|asphalt|cobblestone|cobblestone:flattened|sett|concrete|concrete:lanes|concrete:plates|paving_stones|metal|wood|unhewn_cobblestone$'];
out meta geom;

MarkCompletedBuildingConstruction

[bbox:{{bbox}}];
way[building = construction](if:!is_date(t['opening_date']) || date(t['opening_date']) < date('2019-09-01T00:00:00Z')) -> .construction_with_unknown_state;
(
  way[building=construction](newer: '2019-03-05T00:00:00Z');
  relation[building=construction](newer: '2019-03-05T00:00:00Z');
) -> .recently_edited_construction;
(.construction_with_unknown_state; - .recently_edited_construction;) -> .buildings_under_construction_ways;
relation[building = construction](if:!is_date(t['opening_date']) || date(t['opening_date']) < date('2019-09-01T00:00:00Z')) -> .construction_with_unknown_state;
(
  way[building=construction](newer: '2019-03-05T00:00:00Z');
  relation[building=construction](newer: '2019-03-05T00:00:00Z');
) -> .recently_edited_construction;
(.construction_with_unknown_state; - .recently_edited_construction;) -> .buildings_under_construction_relations;
(.buildings_under_construction_ways; .buildings_under_construction_relations;);
out meta geom;

AddCyclewayPartSurface

Tag Filters

ways with highway ~ path|footway|cycleway|bridleway and surface = paved and bicycle ~ designated|yes and segregated=yes
and !cycleway:surface and !surface:cycleway

[bbox:{{bbox}}];
way[highway ~ '^path|footway|cycleway|bridleway$'][surface = paved][bicycle ~ '^designated|yes$'][segregated = yes][!'cycleway:surface'][!'surface:cycleway'];
out meta geom;

AddFootwayPartSurface

Tag Filters

ways with highway ~ path|footway|cycleway|bridleway and surface = paved and foot ~ designated|yes and segregated=yes
and !footway:surface and !surface:footway

[bbox:{{bbox}}];
way[highway ~ '^path|footway|cycleway|bridleway$'][surface = paved][foot ~ '^designated|yes$'][segregated = yes][!'footway:surface'][!'surface:footway'];
out meta geom;

AddMotorcycleParkingCover

Tag Filters

nodes, ways with amenity = motorcycle_parking
and access !~ private|no and !covered and motorcycle_parking !~ shed|garage_boxes|building

[bbox:{{bbox}}];
node[amenity = motorcycle_parking][access !~ '^private|no$'][!covered][motorcycle_parking !~ '^shed|garage_boxes|building$'] -> .n1;
way[amenity = motorcycle_parking][access !~ '^private|no$'][!covered][motorcycle_parking !~ '^shed|garage_boxes|building$'] -> .w1;
(.n1; .w1;);
out meta geom;

AddFireHydrantType

Tag Filters

nodes with emergency = fire_hydrant and !fire_hydrant:type

[bbox:{{bbox}}];
node[emergency = fire_hydrant][!'fire_hydrant:type'];
out meta geom;

AddParkingType

Tag Filters

nodes, ways, relations with amenity = parking and !parking

[bbox:{{bbox}}];
nwr[amenity = parking][!parking];
out meta geom;

AddPowerPolesMaterial

Tag Filters

nodes with power = pole and !material

[bbox:{{bbox}}];
node[power = pole][!material];
out meta geom;

AddCarWashType

Tag Filters

nodes, ways with amenity = car_wash and !automated and !self_service

[bbox:{{bbox}}];
node[amenity = car_wash][!automated][!self_service] -> .n1;
way[amenity = car_wash][!automated][!self_service] -> .w1;
(.n1; .w1;);
out meta geom;

AddBenchBackrest

Tag Filters

nodes with amenity = bench and !backrest

[bbox:{{bbox}}];
node[amenity = bench][!backrest];
out meta geom;

AddTrafficSignalsButton

Tag Filters

nodes with highway = crossing and crossing = traffic_signals and !button_operated

[bbox:{{bbox}}];
node[highway = crossing][crossing = traffic_signals][!button_operated];
out meta geom;

westnordost · 2019-08-05T21:33:15Z

The following times are from https://overpass.maptime.in/api/ which has little load, so the times are more stable (and probably less than one can expect from the main instance). The bbox used was [bbox:53.5, 9.9, 53.6, 10.0]

Quest Type	Old Time	New Time	😀
AddRoadName	4.0s	3.7s
AddPlaceName	23.4s	4.4s	🏎️
AddOneway
AddBusStopName	23.8s	1.3s	🏎️
AddIsBuildingUnderground	1.3s	1.3s
AddHousenumber	1.8s	1.9s
MarkCompletedHighwayConstruction	1.1s	1.1s
AddReligionToPlaceOfWorship	1.2s	1.2s
AddParkingAccess	1.8s	1.5s
AddRecyclingType	1.2s	1.1s
AddSport	1.6s	1.3s
AddRoadSurface	1.4s	1.3s
AddMaxSpeed	1.4s	1.2s
AddMaxHeight	2.1s	2.4s
AddRailwayCrossingBarrier	1.5s	1.4s
AddPostboxCollectionTimes	1.0s	1.0s
AddOpeningHours	13.4s	5.5s	🚗
AddBikeParkingCapacity	0.9s	0.9s
AddOrchardProduce	1.1s	1.1s
AddCycleway	25.9s	25.8s	🐌
AddSidewalk	31.4s	31.4s	🐌
AddProhibitedForPedestrians	22.5s	1.7s	🏎️
AddCrossingType	0.9s	0.9s
AddBuildingLevels	1.5s	1.5s
AddBusStopShelter	1.7s	1.2s
AddVegetarian	1.2s	1.2s
AddVegan	1.1s	1.1s
AddInternetAccess	1.9s	1.8s
AddParkingFee	1.4s	1.4s
AddMotorcycleParkingCapacity	0.9s	0.9s
AddPathSurface	1.1s	1.1s
AddTracktype	1.4s	1.1s
AddBikeParkingType	1.6s	1.6s
AddWheelChairAccessToilets	0.9s	1.0s
AddPlaygroundAccess	1.3s	1.3s
AddWheelchairAccessBusiness	6.3s	5.6s	🚗
AddToiletAvailability	1.5s	1.4s
AddFerryAccessPedestrian	1.2s	1.2s
AddFerryAccessMotorVehicle	1.3s	1.3s
AddBuildingType	2.1s	2.1s
AddWayLit	7.1s	2.7s	🚗
AddToiletsFee	0.9s	0.9s
AddBabyChangingTable	1.4s	1.8s
AddBikeParkingCover	1.0s	1.0s
AddTrafficSignalsSound	0.9s	0.9s
AddRoofShape	1.4s	1.4s
AddWheelChairAccessPublicTransport	1.7s	1.7s
AddWheelchairAccessOutside	0.9s	0.9s
AddTactilePavingBusStop	1.6s	1.6s
AddTactilePavingCrosswalk	1.0s	1.0s
AddBridgeStructure	0.9s	0.9s
AddReligionToWaysideShrine	0.9s	0.9s
AddCyclewaySegregation	2.4s	1.7s	🚗
MarkCompletedBuildingConstruction	1.7s	1.7s
AddMotorcycleParkingCover	0.8s	0.8s
AddFireHydrantType	0.9s	0.9s
AddParkingType	1.5s	1.5s
AddPowerPolesMaterial	0.9s	0.9s
AddCarWashType	0.9s	0.9s
AddBenchBackrest	0.9s	0.8s
AddTrafficSignalsButton	0.9s	0.9s
Total response time	223s	146s

So, this is in waiting for the response time alone, almost 1.5 minutes, or overall, 53% faster.
To summarize, the following queries are above average (of 2.4s):

Quest Type	Time
AddSidewalk	31.4s
AddCycleway	25.8s
AddWheelcharAccessBusiness	5.6s
AddOpeningHours	5.5s
AddPlaceName	4.4s
AddRoadName	3.7s
AddWayLit	2.7s

By the way, how come that the pull request you mentioned drolbr/Overpass-API#167 has not been merged after more than 4 years? That's a pretty long time.

westnordost · 2019-08-07T11:43:28Z

By the way, do you know who hosts https://overpass.maptime.in/api/ ? I would like to ask, if it is okay to add it to StreetComplete's alternative overpass servers list

mmd-osm · 2019-08-07T12:40:02Z

AFAIK it was set up for OSMCha by the Mapbox team in India. I don't know exactly if the team is stilll around or who currently maintains the box. I think @willemarcel should have more details.

westnordost · 2019-08-07T12:40:14Z

I did the above performance test also on other servers, the results are:

Overpass server	Total response time for downloading all quests
maptime.in	146s
kumi systems	248s
lz4.overpass-api.de	304s
z.overpass-api.de	795s (+ 595s waiting for quota = 1390s)

westnordost · 2019-08-07T12:43:30Z

So even simply switching away from the default overpass instance, of which at least z.overpass-api.de seems to be hopelessly overloaded, will lead to a massive time reduction in download. Something between 3x to 6x faster.
Is it "ok" to use lz4.overpass-api.de as the default server, or may the server name change any time or is otherwise only intended to be used behind the load balancer? z.overpass-api.de seems to be the only server that imposes a quota on the requests.

mmd-osm · 2019-08-07T12:43:35Z

lz4 has seen a massive increase in load during the last two weeks and has higher than expected response times., z.* uses a slower compression, and is expected to be slower. one of the 4 kumi boxes lags behind by 7 years, another one is currently unavailable.

mmd-osm · 2019-08-07T12:45:31Z

overpass-api.de uses DNS round robin rather than a load balancer, i.e. the server has two DNS entries, one pointing to the z.* IP address, the other one to lz4.*. And yes, this might change...

I think both servers have rate limiting in place: http://lz4.overpass-api.de/api/status

westnordost · 2019-08-07T12:59:52Z

I figure they have both rate limiting in place, but the test did not run into any rate limiting, I guess because it is has enough free capacities.

So, can you confirm that lz4 may be used and is also intended to be used directly, without the DNS round robin / load balancer in between? Not querying the overloaded server every ~second time would already increase the speed the quests are downloaded three-fold.

westnordost · 2019-08-07T13:04:01Z

StreetComplete always correctly sets the user agent for every request it makes. Can it be ascertained from munin or other statistics for how many % of requests / for how much load the app is actually responsible for?

mmd-osm · 2019-08-07T15:15:44Z

can you confirm that lz4 may be used and is also intended to be used directly, without the DNS round robin / load balancer in between?

If you manage a list of servers in your app and fall back to other servers if one turns out to be unavailable, that should be ok. Ideally, that would be part of some configuration that won't require users to download a new version of the app, in case some changes are needed.

I can't really help with the question re. production, you need to get in touch with Roland by email.

rugk · 2019-09-01T13:05:53Z

So may I use the queries from #1514 (comment) for updating the wiki now?

westnordost · 2019-09-01T13:55:23Z

Absolutely! I ran the script again in case there were small changes and changed my comment above.

You can also run the program later in case things get changed or new quests added. It is a simple Java program that prints all the things. To run it yourself in Android Studio, just press this button:

mmd-osm · 2019-09-16T18:35:53Z

If you change your script a bit, you can create a wiki page with links to overpass turbo: https://wiki.openstreetmap.org/wiki/User:Mmd/Test_SC

(btw: dev branch runs all queries in 24s)

westnordost · 2019-09-16T20:24:18Z

Cool, nice Page!

@rugk keeps links to all the overpass queries on this page https://wiki.openstreetmap.org/wiki/StreetComplete/Quests

What, you mean, all of them together in 24s??

mmd-osm · 2019-09-16T20:38:10Z

Yeah, I know this page, I thought it might be easier to just generate the content based on the existing quests in the code.

That’s the “Total response time for downloading all quests” as before.

westnordost · 2019-09-16T21:19:44Z

How can the dev instance be 12 times faster than lz4?

mmd-osm · 2019-09-17T05:22:02Z

It runs a different fork/branch with performance fixes.

westnordost · 2019-09-17T07:03:05Z

That's pretty incredible

…

On September 17, 2019 7:22:04 AM GMT+02:00, mmd ***@***.***> wrote: It runs a different codeline with performance fixes.

rugk · 2019-09-17T07:15:27Z

It runs a different fork/branch with performance fixes.

So would not it be useful to contribute these back upstream? 😄

mmd-osm · 2019-09-17T08:04:25Z

Sure, that’s ongoing...

goldfndr · 2019-11-03T22:45:41Z

I've noticed that in the resulting regular expression syntax, some have the ORs in parentheses, most don't. For example, AddRoadName has both "^(primary|secondary|tertiary|unclassified|residential|living_street|pedestrian)$" and '^private|no$', and AddCycleway has them explicitly but AddWheelchairAccessBusiness doesn't have them. Is there a benefit to including the parentheses or detriment to avoiding them?

westnordost · 2019-11-03T22:51:17Z

Hm, I think ^private|no$ may be a problem. Did you find it only in AddRoadName, or in other places as well?

See https://regexper.com/#%5Eprivate%7Cno%24

westnordost · 2019-11-03T23:36:27Z

Right, the overpass query was generated without the parentheses. Should be fixed now.

westnordost · 2019-11-03T23:51:05Z

Anyhow, a big thanks for noticing and notifying me about it! That's quite a big bug!

rugk · 2019-11-04T08:46:10Z

Testing the AddRoadName query I am surprised that Overpass Turbo actually highlights me the streets with names and not without ones…

mmd-osm · 2019-11-04T09:58:49Z

NB: It seems this is entirely unrelated to the Performance optimization in this issue. The original queries already had the same issue. As an example: 6d234a4

goldfndr · 2019-11-05T12:14:52Z

Thanks. Any possibility if you could yet again update the final syntax?

westnordost added the enhancement label Aug 1, 2019

westnordost mentioned this issue Aug 1, 2019

Initial quest scan may not found any quests before running out of query quota #1457

Closed

westnordost self-assigned this Aug 1, 2019

This comment has been minimized.

Sign in to view

westnordost mentioned this issue Aug 5, 2019

Overpass performance #1516

Merged

westnordost mentioned this issue Aug 7, 2019

Question: May overpass.maptime.in/api/ be used by StreetComplete? OSMCha/osmcha-frontend#381

Closed

westnordost referenced this issue Aug 12, 2019

change default overpass url to lz4.overpass-api.de

41190d3

Atrate mentioned this issue Aug 27, 2019

No Questions around #1542

Closed

westnordost closed this as completed in #1516 Aug 28, 2019

Performance improvements: Faster Overpass queries #1514

Performance improvements: Faster Overpass queries #1514

Comments

westnordost commented Aug 1, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

westnordost commented Aug 2, 2019

mmd-osm commented Aug 2, 2019 • edited Loading

westnordost commented Aug 2, 2019

rugk commented Aug 3, 2019

westnordost commented Aug 3, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

westnordost commented Aug 4, 2019

AddBusStopName

mmd-osm commented Aug 5, 2019 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

westnordost commented Aug 5, 2019 • edited Loading

AddRoadName

AddPlaceName

AddOneway

AddBusStopName

AddIsBuildingUnderground

AddHousenumber

MarkCompletedHighwayConstruction

AddReligionToPlaceOfWorship

AddParkingAccess

AddRecyclingType

AddSport

AddRoadSurface

AddMaxSpeed

AddMaxHeight

AddRailwayCrossingBarrier

AddPostboxCollectionTimes

AddOpeningHours

AddBikeParkingCapacity

AddOrchardProduce

AddCycleway

AddSidewalk

AddProhibitedForPedestrians

AddCrossingType

AddBuildingLevels

AddBusStopShelter

AddVegetarian

AddVegan

AddInternetAccess

AddParkingFee

AddMotorcycleParkingCapacity

AddPathSurface

AddTracktype

AddForestLeafType

AddBikeParkingType

AddWheelchairAccessToilets

AddPlaygroundAccess

AddWheelchairAccessBusiness

AddToiletAvailability

AddFerryAccessPedestrian

AddFerryAccessMotorVehicle

AddBuildingType

AddWayLit

AddToiletsFee

AddBabyChangingTable

AddBikeParkingCover

AddTrafficSignalsSound

AddRoofShape

AddWheelchairAccessPublicTransport

AddWheelchairAccessOutside

AddTactilePavingBusStop

mmd-osm commented Aug 2, 2019 •

edited

Loading

mmd-osm commented Aug 5, 2019 •

edited

Loading

westnordost commented Aug 5, 2019 •

edited

Loading

westnordost commented Aug 5, 2019 •

edited

Loading

mmd-osm commented Aug 7, 2019 •

edited

Loading

mmd-osm commented Aug 7, 2019 •

edited

Loading

mmd-osm commented Sep 17, 2019 •

edited

Loading