Skip to content

Latest commit

 

History

History
125 lines (96 loc) · 5.63 KB

blackbox.md

File metadata and controls

125 lines (96 loc) · 5.63 KB
Raw

Black-Box Testing

RESTful APIs

Informally, in Black-Box Testing of a RESTful API, we generate test cases without knowing the internal details of the API. Still, we need to know the schema of the API to determine which endpoints can be called, and how. Otherwise, sending random bits on a TCP socket will unlikely result in any meaningful, well formatted HTTP message.

Even with a schema, there is still the issue of generating inputs, like HTTP query parameters and body payloads (e.g., in JSON/XML). This can be done at random, but still within the constraints defined in the schema (e.g., the structure of the body payloads).

To do this kind of testing, there is only the need to have an API up and running, and specify where the schema can be found. For example, https://apis.guru lists many APIs online. Such website provides an API itself to query info on existing APIs. Such small API (only 2 endpoints) can be easily tested by running the following on a command-line:

java -jar core/target/evomaster.jar  --blackBox true --bbSwaggerUrl https://api.apis.guru/v2/openapi.yaml  --outputFormat JAVA_JUNIT_4 --maxTime 30s --ratePerMinute 60

The command is doing the following:

  • java -jar core/target/evomaster.jar: execute the EvoMaster core process. The executable evomaster.jar must be either downloaded or built from source.
  • --blackBox true: by default, EvoMaster does white-box testing. Here, we specify that we do black-box testing instead.
  • --bbSwaggerUrl ...: URL of where the OpenAPI/Swagger schema is. The location of the API will be inferred from this schema (e.g., from host and servers tags). If such info is missing, then the API is assumed to be on same host as the schema. If needed, the API host location can be changed with the optional --bbTargetUrl (which overrides what specified in the schema).
  • --outputFormat JAVA_JUNIT_4: must specify how the tests will be generated, e.g., in Java using JUnit 4 in this case. Note: the language of the generated tests is not necessarily related to the language in which the tested application is implemented.
  • --maxTime 30s: for how long to run the search, i.e., just 30 seconds in this very simple example.
  • --ratePerMinute 60: avoid doing a DoS attack by bombarding the remote service with too many HTTP calls in quick rapid succession. Limit to max 1 per second (i.e., 60 per minute) in this example.

This command will create the following test suite, in which 2 GET calls are executed:

public class EvoMasterTest {

    private static String baseUrlOfSut = "https://api.apis.guru";
    
    @BeforeClass
    public static void initClass() {
        RestAssured.enableLoggingOfRequestAndResponseIfValidationFails();
        RestAssured.useRelaxedHTTPSValidation();
        RestAssured.urlEncodingEnabled = false;
        RestAssured.config = RestAssured.config()
            .jsonConfig(JsonConfig.jsonConfig().numberReturnType(JsonPathConfig.NumberReturnType.DOUBLE))
            .redirect(redirectConfig().followRedirects(false));
    }

    
    @Test
    public void test_0() throws Exception {
        
        given().accept("application/json")
                .get(baseUrlOfSut + "/v2/list.json")
                .then()
                .statusCode(200)
                .assertThat()
                .contentType("application/json")
                .body("size()", numberMatches(1605));
    }
    
    
    @Test
    public void test_1() throws Exception {
        
        given().accept("application/json")
                .get(baseUrlOfSut + "/v2/metrics.json")
                .then()
                .statusCode(200)
                .assertThat()
                .contentType("application/json")
                .body("'numAPIs'", numberMatches(1605.0))
                .body("'numEndpoints'", numberMatches(46276.0))
                .body("'numSpecs'", numberMatches(2869.0));
    }
}

GraphQL APIs

Black-box fuzzing of GraphQL APIs uses the same options as for RESTful APIs. One difference is that --bbTargetUrl is used to specify the entry point of the GraphQL API. Another difference is that we must specify the --problemType to be GRAPHQL, as the default is REST. An example on GitLab's API is:

evomaster.exe  --problemType GRAPHQL --bbTargetUrl https://gitlab.com/api/graphql --blackBox true --outputFormat JAVA_JUNIT_4 --maxTime 30s --ratePerMinute 60

AUTH

Since version 1.3.0, it is possible to specify custom HTTP headers (e.g., to pass auth tokens), using the options from --header0 to --header2 (in case more than one HTTP header is needed).

However, in contrast to white-box mode, in black-box mode there is currently no configuration to automatically setup the fetching of dynamic tokens. For example, if an auth token needs to be fetched from a login endpoint (e.g., a POST on a /login with username and password), then such call has to be done manually (and then the token can be passed to EvoMaster with --header0 option, e.g., --header0 "cookie: <token>"). Such call can be done with other tools like Postman and cURL.

WARNING

Black-box testing is easy to do: first download the tool, and then just specify where the Swagger schema can be found. However, as it does know nothing about the internal details of the tested application, it is unlikely that it will get good results (i.e., in terms of code coverage and detected faults).

The first time you try EvoMaster, use black-box testing to get an idea of what EvoMaster could do for you. However, after an initial trial, we recommend to switch to white-box testing, as it can lead to much, much better results.