Cucumber Test Suite

The correct functionality of OSRM is supported by a test suite based on Cucumber (https://cucumber.io/). The tests are run automatically on Travis CI (https://travis-ci.org/Project-OSRM/osrm-backend).

Overview:

• Introduction
• Running tests locally
  • Install dependencies
  • Run tests
• Tests explained
• Writing tests
  • Node maps
  • Lat/Lon tables
  • Routability tables
• Customizing Tests

Introduction

Cucumber is the framework we use for writing and running tests. We use the JavaScript port of Cucumber, distributed as an NPM module. All support code and step definitions for the tests are written in JavaScript, and tests are written using the Gherkin syntax. They run against a compiled version of osrm-backend by using its commandline tools.

The test suite currently does not have complete coverage for all osrm-backend functionality, but we are striving for full test coverage.

Running tests locally

Whenever you are making a change to osrm-backend, you should run the test suite locally to verify that the change does not introduce regressions. If your change introduces new or changed behaviour, you need to adjust the test suite accordingly.

Install dependencies

• Have osrm-backend built locally as explained here
  • You need to build the Node bindings as well; when running cmake include -DBUILD_NODE_BINDINGS=ON
• Install Node.js
  • Tests require Node 4.0+. The nvm project is very helpful for installing and switching between Node versions.
• Install Node dependencies
  • Once Node.js 4.0+ is installed you can install modules via npm. npm install will install all dependencies listed in package.json, then run npm link to link cucumber as an executable.
  • Alternately, you can replace cucumber with node_modules/cucumber/bin/cucumber.js in the commands below.

Run tests

The following profiles each run different subsets of cucumber tests.

cucumber              # default profile, runs all tests that should pass (exclude @todo and @bug)
cucumber -p verify    # same as above, but with minimal output format
cucumber -p bugs      # run all tests covering known bugs (@bug)
cucumber -p todo      # run all tests covering unimplemented features (@todo)
cucumber -p all       # run all tests, including @todo and @bug

You can also run specific tests or groups of tests:

• To run tests with a specific tag, use: cucumber --tags @restrictions
• To exclude tests withs a specific tag, use: cucumber --tags ~@restrictions
• To run a specific scenario: cucumber features/bicycle/restrictions.feature:6
• To run a named scenario: cucumber --name "A single way with two nodes"
• To run all scenario in a specific folder: cucumber features/foot

Note that the default profile excludes tests tagged with @todo and @bug, so if you want to run a test with one of those tags, you need to use the 'all' profile:

cucumber -p all -t @weird_bug_7

Finally, run your tests by calling test:

npm test

Output

Tests are evaluated by parsing the output returned by osrm-routed. Results are shown in the terminal. A passed route will be shown using a green row. A failed row will be shown with two rows: The expected result in a yellow row, followed by the actual result in a grey row.

(Note that there's currently an open cucumber-js issue about implementing formatted diffs)

Log files

During the tests, several log files are used:

• Output from osrm-extract and osrm-contract are redirected to one file located attest/logs/path/to/test.feature/hash/name_of_test.log
• Output from osrm-routed is redirected to test/osrm-routed.log

Tests explained

The general structure of executing one cucumber test scenario is this:

• Create an OSM fixture file
• Run the OSM fixture file through osrm-extract and osrm-contract
• Start osrm-routed with the generated dataset
• Make a request against osrm-routed
• Compare request results with expectations

Let's dive into details:

Files

Tests are located in a sub-directory called features/ and organized into feature files, each containing many scenarios.

Actual test runs are performed inside the folder test/. The folder contain a .stxxl config file using a small 10MB disk.

Mechanics

For each test scenario, an .osm data file is constructed based on the description in the test scenario.

It's then preprocessed with osrm-extract and osrm-contract. The relevant profile script is copied to test/profile.ini before each preprocessing, so the binaries pick it up.

The resulting .osrm files are cached, so that preprocessing is only repeated if either the .osm data, the profile or the binaries have changed. This helps to speed up re-running of tests.

test/server.ini is now rewritten to ensure that the right configuration is used. osrm-routed is then launched, and routes are requested via http calls to localhost:5000, and the response is compared to the expected result. All request are timed out after 10 seconds. Both timeouts and crashes are caught and reported.

To ensure a clean run of every test, all processes named osrm-extract, osrm-contract and osrm-routed are killed before and after each test, using a signal 9, no matter who launched them.

Writing tests

For Cucumber documentation, see https://github.com/cucumber/cucumber/wiki/A-Table-Of-Content.

The OSRM tests currently come in three main flavors:

• Node maps
• Lat/Lon tables
• Routability tables

Node Maps

With node maps you geometrically place nodes on a mini map. Each node is identified by a letter. You then string together nodes to create ways, and optionally add relations. Finally you specify a list of routes to be tested, described by start and end points, and the expected route:

Scenario: Zig Zag
	Given the node map
	  """      
	  a      c
	   \   /
	     b
	  """

	And the ways
	 | nodes |
	 | ab    |
	 | bc    |
	
	When I route I should get
	 | from | to | route    |
	 | a    | c  | ab,bc,bc |
	 | c    | a  | bc,ab,ab |

The final segment of the route appears twice in this response. This is actually desired behaviour. The first part is for a turn onto the segment, the second one is for the arrival operation. The full list of instructions would be depart, new name, arrive. Both the new name and arrive indicate the same segment, hence the replication of the segment in route.

Nodes

Under 'Given the node map', the node map is constructed using nodes. Each node must be specified by a letter a-z. Only one node can exists at each location, and a node can only be present once on the map. Other characters like \ / | - can be used to visually indicate connections,but note that these characters are in fact ignored by the test. To construct ways you need to use an "And the ways" table.

When we generate the .osm file for the test case from the node map, node IDs are assigned, starting at 1, in left-to-right, top-to-bottom order. In the example above, a=1, c=2, b=3.

You can add OSM tags to nodes by adding using a And the nodes table:

	And the nodes
	 | node | barrier |
	 | a    | bollard |
	 | b    | gate    |

The node column identifies the node, while additional columns are added as tags to the corresponding node.

Locations

In addition to the nodes specified with a-z, you can add locations specified with numbers 0-9. Locations can be used as start/end/via points when requesting routes, but are not included in the OSM file, meaning you add/move them without affecting the OSM input data.

For example, this map results in the same OSM input as the example above, but you can test a route from 1 to 2.

	Given the node map
	 | a |   | c |
	 | 1 | b | 2 |

Specifying ways

Under And the ways, you create ways by stringing together nodes in the node map. For example, ab is a way starting at a and ending at b, while abc is a way starting at a, passing b, and ending at c. A ways can have two or more nodes. All nodes of the way must exist in the node map.

You can add OSM tags to ways by adding columns to the And the ways table:

	And the ways
	 | nodes | oneway |
	 | ab    | yes    |
	 | bc    | no     |

Note that Given ... and And ... is interchangeable in cucumber, and you should just use what creates a natural language flow.

Specifying routes to test

An OSM file is constructed based on the given nodes, ways and relations, and then used for testing. Using When I route I should get you specify routes to be tested, each with a start and an end node, and the ways you expect the route to follow:

	When I route I should get
	 | from | to | route    |
	 | a    | c  | ab,bc,bc |
	 | c    | a  | bc,ab,ab |

An alternative syntax uses a single waypoints column to specify the nodes you want to follow:

	When I route I should get
	 | waypoints | route    |
	 | a,c       | ab,bc,bc | # from a, to b
	 | a,b,c     | ab,bc,bc | # from a, via b, to c

If you specify more than two waypoints, via points are used.

Note that you cannot specify the nodes you expect the computed route to follow, only the ways. This is because OSRM does not return nodes in query results, only ways.

Testing Distance and Travel Time

In addition to the route itself, you can test for total length or time (or both) by adding the corresponding columns to the test table:

	When I route I should get
	 | from | to | route | time | distance |
	 | a    | b  | ab,ab | 24s  | 100m     |
	 | c    | d  | cd,cd | 72s  | 200m     |

Time is specified in seconds, while length is specified in meters. (Note that OSRM currently quantized length to the nearest 10m, and time to the nearest second)

OSRM uses spherical coordinates. Distances in node maps deviate from the naive flat geometry when you move away from the equator.

Testing Turn Instructions

You can test the turn instructions by adding a "turns" columns to the test table:

	When I route I should get
	 | from | to | route    | turns               |
	 | a    | c  | ab,bc,bc | depart,left,arrive  |
	 | c    | a  | bc,ab,ab | depart,right,arrive |

The expected turns are specified using words that correspond to the v5 turns and modiffiers that are returned by OSRM:

depart
arrive
new name [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
continue [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
turn [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
merge [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
ramp [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
fork [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
end of road [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
roundabout-exit-[#|undefined]
rotary-exit-[#|undefined]
roundabout turn [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
notification

Fuzzy ranges

Fuzzy ranges for expected numerical values can be specified using either a percentage range, or an absolute +- range:

	When I route I should get
	 | from | to | route       | distance |
	 | a    | d  | abcde,abcde | 300m +-1 |
	 | a    | e  | abcde,abcde | 700m ~1% |

Comment column

You can add a comment column by using the column name #. The column will be ignored, but can be useful if you need to explain individual rows:

	When I route I should get
	 | from | to | route       | time | #                    |
	 | a    | d  | abcde,abcde | 30s  | we're going downhill |
	 | a    | e  | abcde,abcde | 60s  | we're going uphill   |

Lat/Lon Tables

Lat/lon tables are quite similar to node maps, except the nodes are placed using a table with absolute latitude and longitude values:

	Scenario: Longitudinal distances at latitude 45
		Given the node locations
		 | node | lat | lon |
		 | c    | 45   | 80 |
		 | d    | 45   | 0  |

		And the ways
		 | nodes |
		 | cd    |

		When I route I should get
		 | from | to | route | distance      |
		 | c    | d  | cd,cd | 6028844 ~0.5% |

This is especially useful for testing things related to the Mercator projection and huge distances. In this case, node IDs are assigned in the .osm file in top-to-bottom order. In the example above, c=1, d=2.

Response codes

Response codes of OSRM queries can be checked using a code column as

        When I route I should get
          | from | to | code    |
          |    a |  b | NoRoute |

Annotations

If annotations are included in the request,

		Given the query options                                                 
	         | annotations | true |

then they can also be checked for in the response output with the following syntax: a:{annotation_type}

		When I route I should get
		 | from | to | route    | a:nodes | a:duration |
		 | a    | c  | ab,bc,bc | 1:2,2:3 | 10,2.5     |

The fields that can be checked for are the fields that can be requested with the annotations parameter: duration, distance, datasources, nodes, weight. A : delineates annotation values within an annotation object within a step, a , delineates grouped data from the same leg object.

Routability Tables

This flavor of tests is used for checking individual isolated ways, and is often useful for testing access and barrier tags. You can test each direction (forward/backward) separately:

Scenario: Simplest possible oneway
	Given the speedprofile "car"
 	Then routability should be
	 | highway | oneway | forw | backw |
	 | primary |        | x    | x     |
	 | primary | yes    | x    |       |
	 | primary | -1     |      | x     |

Column headers are used to specify tag keys. In the example above, the highway and oneway tags are used.

You can also set tags on a central node of the way, by using columns with titles in the form of node/...

	Scenario: Car - Barriers 
		Then routability should be
		 | highway | node/barrier   | bothw |
		 | primary |                | x     |
		 | primary | bollard        |       |

Columns forw, backw and bothw have special meaning:

• If a forw_ column is present, the way is tested for forward routability
• If a backw_ column is present, the way is tested for backwards routability
• If a _bothw_ column is present, the way is tested for both forwards and backwards routability

A single OSM file is constructed, containing an isolated way for each row. The ways are labelled w1, w2, w3, etc and placed in a line, but with spacing in between:

--w1--    --w2--    --w3--    

Customizing Tests

For all types of tests, nodes are by default spaced on a grid with origin at 1,1 and a grid size of 100m. The grid size can be changed using:

	Given a grid size of 500 meters

The origin can be changed by:

	Given the origin 55.5,12.2

Note that grid spacing uses fixed lat/lon increments (ignoring the latitude), and will therefore be distorted when you move away from the equator. For this reason, nodes far ways from the equator are often better placed using absolute node locations.

By default, tests use the "bicycle" profile in profiles/bicycle.lua, but this can be changed for each scenario by adding:

	Given the speedprofile "car"

Note that you can use a Background: section if you need to specify things for all test scenarios in a .feature file:

	Background:
		Given the speedprofile "car"
		Given a grid size of 500 meters