Skip to content

Latest commit

 

History

History
113 lines (70 loc) · 8.81 KB

TutorialDemoServer.md

File metadata and controls

113 lines (70 loc) · 8.81 KB
Raw

A Quick RESTler Tutorial

Let's try RESTler on a simple example of REST API and service.

The directory restler\demo_server in this repo contains a self-contained example of service you can locally on your machine. See the README.md file in that directory to install and start this demo_server service.

Let's create a fresh directory C:\restler-test where will run RESTler on this example. Let's cd into that dir: cd C:\restler-test

1. Compile

Let's copy the Swagger spec swagger.json from restler\demo_server into C:\restler-test and let's compile it with RESTler:

C:\RESTler\restler\Restler.exe compile --api_spec C:\restler-test\swagger.json

This command creates a new sub-directory Compile where the results of the compilation are saved in several files:

  • grammar.py and grammar.json are the RESTler grammars
  • dict.json is a default dictionary associating parameters types with a set of default values (you can edit this file if you want more or different values)
  • engine_settings.json is a default engine_settings file with options for the RESTler test engine (you can edit this file to turn on other options)
  • config.json is a default compiler configuration file (you can edit this file and re-run the compiler with other options)

For this simple tutorial, we will use the default values automatically generated by the RESTler compiler and move directly to the next step.

2. Test

Let's run RESTler in test mode to see what specification coverage we get with this default RESTler grammar:

C:\RESTler\restler\restler.exe test --grammar_file C:\restler-test\Compile\grammar.py --dictionary_file C:\restler-test\Compile\dict.json --settings C:\restler-test\Compile\engine_settings.json --no_ssl

(For help, run C:\RESTler\restler\restler.exe --help)

The results are saved in a new sub-directory Test. In the sub-directory C:\restler-test\Test\RestlerResults\experiment<...>\logs, we see a file named main.txt which lists one-by-one all the API requests in the Swagger spec and whether the request has been succesfully executed (VALID) or not (INVALID). A VALID request means RESTler was able to execute the request and get a 20x HTTP status code as a response. The total spec coverage is given at the bottom of that file:

Final Swagger spec coverage: 5 / 6

The exact requests executed by RESTler and the service response are all logged in the file network.testing.<...>.txt.

The file speccov.json is a machine-readable version of main.txt, which is easier to parse by other tools (for instance, for regression testing and automatically comparing coverage data).

RESTler has a garbage-collector (gc) that attempts to delete all resources ever created by RESTler during a test run. The garbage-collector logs are in the file garbage_collector.gc.<...>.txt and the raw HTTP/S traffic from the gc is in network.gc.<...>.txt.

In this example, coverage is 5 / 6 and the only INVALID request is

Rendering INVALID
    - restler_static_string: 'GET '
    - restler_static_string: '/'
    - restler_static_string: 'api'
    - restler_static_string: '/'
    - restler_static_string: 'blog'
    - restler_static_string: '/'
    - restler_static_string: 'posts'
    - restler_static_string: '?'
    - restler_static_string: 'per_page='
    + restler_fuzzable_int: ['0', '1']
    - restler_static_string: '&'
    - restler_static_string: 'page='
    + restler_fuzzable_int: ['0', '1']
    - restler_static_string: ' HTTP/1.1\r\n'
    - restler_static_string: 'Accept: application/json\r\n'
    - restler_static_string: 'Host: localhost:8888\r\n'
    - restler_static_string: '\r\n'

By looking at network.testing.<...>.txt, we can see that RESTler attempts to execute this request 4 times, each time with a value either 0 or 1 for the per_page= and page=. It turns out none of these 4 combinations are valid: the per_page= must be 2 minimally, but RESTler was not able to infer this automatically. (One way to fix this is to edit dict.json and add the value 2 in the list for restler_fuzzable_int.)

3. Fuzz-lean

Let's now try to run restler in Fuzz-lean mode.

C:\RESTler\restler\restler.exe fuzz-lean --grammar_file C:\restler-test\Compile\grammar.py --dictionary_file C:\restler-test\Compile\dict.json --settings C:\restler-test\Compile\engine_settings.json --no_ssl

The results are in a new FuzzLean directory and the experiment results can be found in C:\restler-test\FuzzLean\RestlerResults\experiment<...>\. The logs\ directory should contain the same coverage results as the previous Test run, but you should also now see a new bug_buckets directory.

Inside the bug_buckets directory there should be four files:

  • bug_buckets.txt
  • InvalidDynamicObjectChecker_20x_1.txt
  • InvalidDynamicObjectChecker_500_1.txt
  • PayloadBodyChecker_500_1.txt

The bug_buckets.txt file contains a list of each unique sequence that found a bug. Any request that receives a '500' status code as a response will be treated as a bug. Additionally, some checkers also record bugs if an unexpected '20x' status code is received. Bugs are bucketized by only including one bug per unique request sequence even if that same sequence produces more than one bug throughout the fuzzing run. The exception to this is if a new bug was detected due to a different status code, e.g. the first bug was a 500 and the second was a 503.

Each unique sequence in the bug_buckets.txt file also has a corresponding individual log associated with that bug. This log shows the sequence of requests and their responses exactly as they were sent and received from the server. These particular bugs were planted in the demo server for this example.

Looking at the InvalidDynamicObjectChecker_20x_1.txt log you can see that the sequence of requests includes a POST request that creates a blog post and a GET request that attempts to get the post that was just created. Here the Invalid Dynamic Object Checker replaced the blog post id of 5879 with 5879?api-version=2019-01-01, which was accepted by the demo server because the unexpected query string of ?api-version=2019-01 was ignored, which resulted in a 200 response code. Because the Invalid Dynamic Object Checker assumes that any 'invalid dynamic object' will fail to produce a 20x status code, it flags this as a bug.

The next log in the list, InvalidDynamicObjectChecker_500_1.txt comes from the same sequence of requests as the previous Invalid Dynamic Object Checker bug. As mentioned before, because these two bugs were detected from different response codes, they are considered unique bugs.

As you can see in the log, this request received a 500 status code in its response, which is always considered a bug. Looking closely at the GET request you can see that the request contains two question marks after the endpoint, which triggered the 500 error in the demo server.

Finally, the PayloadBodyChecker_500_1.txt file contains a sequence that includes a POST request followed by a PUT request. The body used during this fuzz can be seen in the sequence log, but, for Payload Body Checker bugs only, you can also find the body used to fuzz the request at the top of the log file in the header. In this case, the body was {'body': 'fuzzstring'}.

Right above the body in the header you should also see StructMissing_/id/checksum. This is a Payload Body Checker specific tag that helps identify how the body was fuzzed in order to help identify what could have caused the bug in the service. What this particular tag is telling us is that the body was fuzzed by removing a piece of the body's structure (StructMissing) and the pieces that were missing were 'id' and 'checksum'. Because the bug was planted, we know that this bug was, in fact, triggered by the missing 'id' field in the body.

4. Fuzz

Now let's try to fuzz:

C:\RESTler\restler\restler.exe fuzz --grammar_file C:\restler-test\Compile\grammar.py --dictionary_file C:\restler-test\Compile\dict.json --settings C:\restler-test\Compile\engine_settings.json --no_ssl --time_budget 1

The new time_budget parameter is the length, in hours, to perform the fuzzing run.

The results are in a new Fuzz directory and the experiment results can be found in C:\restler-test\Fuzz\RestlerResults\experiment<...>\.

Like the FuzzLean experiment above, you will see three unique bugs in bug_buckets\bug_buckets.txt as well as their corresponding bug logs. If you were to closely inspect the logs\network.testing.####.1.txt file you may notice that there were several more instances where these same bugs were triggered. Because RESTler bucketizes the bugs based on request type and response code, the duplicate bugs were not reported.

Because the demo server is a very small and uncomplicated API, the one hour fuzz was unable to detect any additional unique bugs that FuzzLean did not already find.