Encoded question mark missing from reported query parameter value

[Typraeurion]

Describe the issue
I’m running unit tests which generate random strings for query string parameters. I found that when the parameter value contains a '?' character, the test fails because the reported request is missing this character even though I’ve verified that the URL has properly encoded it as "%3F" (using URIBuilder from Apache httpcore5) and the mock server has received it as shown in the debug log. (The log output from the mock server is included at the bottom.) In this case both re1kbe and UgD1t have values with a leading ‘?’. My test (code shown below) checks the request with a custom method:

    public static void assertQueryEquals(
            Map<String,String> expectedQuery,
            HttpRequest request) throws AssertionError {

        Map<String,String> converted = new TreeMap<>();
        if (request.getQueryStringParameters() != null) {

            Multimap<NottableString, NottableString> actualQuery =
                    request.getQueryStringParameters().getMultimap();

            for (NottableString key : actualQuery.keySet()) {
                String firstValue = null;
                for (NottableString value : actualQuery.get(key)) {
                    firstValue = value.getValue();
                    break;
                }
                converted.put(key.getValue(), firstValue);
            }

        }

        assertEquals(Optional.ofNullable(expectedQuery)
                .orElse(Collections.emptyMap()), converted,
                "Query string parameters");

    }

which seems straightforward enough. In this case, it fails:

Multiple Failures (1 failure)
	org.opentest4j.AssertionFailedError: Query string parameters ==> expected: <{HRH5q1=3'Kd323M9j, UgD1t=?a>i=YvgxoBkgjR, kFWJFNM=|{MJ]h)DtQYE, piufSbS3n=P$Ltw\\ZKBo]SP;jJ, re1kbe=?t6a:6-$TtF}> but was: <{HRH5q1=3'Kd323M9j, UgD1t=a>i=YvgxoBkgjR, kFWJFNM=|{MJ]h)DtQYE, piufSbS3n=P$Ltw\\ZKBo]SP;jJ, re1kbe=t6a:6-$TtF}>
Expected :{HRH5q1=3'Kd323M9j, UgD1t=?a>i=YvgxoBkgjR, kFWJFNM=|{MJ]h)DtQYE, piufSbS3n=P$Ltw\\ZKBo]SP;jJ, re1kbe=?t6a:6-$TtF}
Actual   :{HRH5q1=3'Kd323M9j, UgD1t=a>i=YvgxoBkgjR, kFWJFNM=|{MJ]h)DtQYE, piufSbS3n=P$Ltw\\ZKBo]SP;jJ, re1kbe=t6a:6-$TtF}

The ‘?’ are missing from the “actual” values of the parameters, as returned by the recorded request in the mock server.

What you are trying to do
Testing to verify that my code correctly encodes query string parameters with any arbitrary characters.

MockServer version
5.15.0

To Reproduce
Steps to reproduce the issue:

1. How you are running MockServer (i.e maven plugin, docker, etc)
   In the test class:

    private static ClientAndServer mockServer;

    @BeforeAll
    public static void startServer() {

        logger.debug("Starting the mock HTTP server");
        mockServer = ClientAndServer.startClientAndServer();

    }

    @BeforeEach
    public void resetServerMock() {
        mockServer.reset();
    }

    @AfterAll
    public static void stopServer() {

        logger.debug("Stopping the mock HTTP server");
        mockServer.stop(true);

    }

2. Code you used to create expectations

    /**
     * General method for running tests against the first form of makeRequest.
     * This uses the given {@code method}, {@code query}, {@code headers},
     * {@code postData}, and {@code useJson} flag, and makes up a {@code path}.
     * It then verifies that the given parameters were used in the REST API call.
     *
     * @param method the HTTP method to use for this request
     * @param query optional query string parameters to include in the URL
     * @param headers optional headers to include in the request
     * @param postData optional body for the request (POST or PUT only)
     * @param useJson whether the request body should be sent as JSON
     * ({@code true}) or application/x-form-urlencoded ({@code false}).
     */
    private void runBasicRequestTest(HttpMethod method,
                                    Map<String, String> query,
                                    Map<String, String> headers,
                                    Map<String, String> postData,
                                    boolean useJson) {

        MockAPIDataSource dataSource = new MockAPIDataSource();
        dataSource.setBaseUrl(getBaseMockURL());
        dataSource.setCredentials(RandomStringUtils.randomAlphabetic(7, 14),
                RandomStringUtils.randomPrint(8, 20));

        String apiPath = String.format("/%s/%s",
                RandomStringUtils.randomAlphabetic(5, 10),
                RandomStringUtils.randomAlphabetic(8, 16));

        String expectedResponse = (method == HttpMethod.HEAD) ? null
                : RandomStringUtils.randomPrint(20, 80);
        HttpResponse response = response(expectedResponse)
                .withHeader(HttpHeaders.CONTENT_TYPE,
                        MediaType.TEXT_PLAIN.toString());

        mockServer.when(request().withMethod(method.name()))
                        .respond(response);

        String actualResponse = dataSource.makeRequest(method, apiPath,
                query, headers, postData, useJson);

        mockServer.verify(request(apiPath).withMethod(method.name()));
        HttpRequest[] requests = mockServer.retrieveRecordedRequests(null);
        assertEquals(1, requests.length, "Number of recorded requests");

        assertAll(
                () -> assertEquals(apiPath, requests[0].getPath().toString(), "Path"),
                () -> assertQueryEquals(query, requests[0]),
                () -> {
                    if (postData == null)
                        return;
                    assertContainsHeader(
                            HttpHeaders.CONTENT_TYPE,
                            useJson ? MediaType.APPLICATION_JSON.toString()
                                    : MediaType.APPLICATION_FORM_URLENCODED.toString(),
                            requests[0]);
                },
                () -> assertContainsHeader(
                        HttpHeaders.AUTHORIZATION,
                        expectedAuthorization(dataSource),
                        requests[0]),
                () -> assertIncludesHeaders(headers, requests[0]),
                () -> assertPostDataEquals(postData,
                        requests[0].getBodyAsString(), useJson),
                () -> assertEquals(expectedResponse, actualResponse,
                        "Response string")
        );

    }
…
    @Test
    public void testGetRequestWithQuery() {

        Map<String,String> query = new TreeMap<>();
        int targetSize = R.nextInt(3, 7);
        while (query.size() < targetSize) {
            query.put(RandomStringUtils.randomAlphanumeric(5, 10),
                    RandomStringUtils.randomPrint(10, 21));
        }

        runBasicRequestTest(HttpMethod.GET, query, null, null, false);

    }

(Due to the random nature of the query parameters, the test needs to be repeated until a ‘?’ shows up somewhere.)

3. What error you saw

org.opentest4j.MultipleFailuresError: Multiple Failures (1 failure)
	org.opentest4j.AssertionFailedError: Query string parameters ==> expected: <{HRH5q1=3'Kd323M9j, UgD1t=?a>i=YvgxoBkgjR, kFWJFNM=|{MJ]h)DtQYE, piufSbS3n=P$Ltw\\ZKBo]SP;jJ, re1kbe=?t6a:6-$TtF}> but was: <{HRH5q1=3'Kd323M9j, UgD1t=a>i=YvgxoBkgjR, kFWJFNM=|{MJ]h)DtQYE, piufSbS3n=P$Ltw\\ZKBo]SP;jJ, re1kbe=t6a:6-$TtF}>

Expected behaviour
(Test passes)

MockServer Log

17:00:54.818 [MockServer-EventLog0] INFO org.mockserver.log.MockServerEventLog -- 62609 resetting all expectations and request logs
17:00:55.609 [MockServer-EventLog0] INFO org.mockserver.log.MockServerEventLog -- 62609 creating expectation:
  {
    "httpRequest" : {
      "method" : "GET"
    },
    "httpResponse" : {
      "statusCode" : 200,
      "reasonPhrase" : "OK",
      "headers" : {
        "Content-Type" : [ "text/plain" ]
      },
      "body" : "OdKkQU'1VYwr9hsL`AW69`5}jEty)_uY:+k0"
    },
    "id" : "f9b2796f-1f04-4fc9-9332-45053207b615",
    "priority" : 0,
    "timeToLive" : {
      "unlimited" : true
    },
    "times" : {
      "unlimited" : true
    }
  }
 with id:
  f9b2796f-1f04-4fc9-9332-45053207b615
17:10:22.227 [MockServer-EventLog0] INFO org.mockserver.log.MockServerEventLog -- 62609 received request:
  {
    "method" : "GET",
    "path" : "/xEpwx/lnflDhYIe",
    "queryStringParameters" : {
      "re1kbe" : [ "?t6a:6-$TtF" ],
      "piufSbS3n" : [ "P$Ltw\\\\ZKBo]SP;jJ" ],
      "kFWJFNM" : [ "|{MJ]h)DtQYE" ],
      "UgD1t" : [ "?a>i=YvgxoBkgjR" ],
      "HRH5q1" : [ "3'Kd323M9j" ]
    },
    "headers" : {
      "content-length" : [ "0" ],
      "User-Agent" : [ "Java/17.0.11" ],
      "Host" : [ "localhost:62609" ],
      "Connection" : [ "keep-alive" ],
      "Authorization" : [ "Basic ZlVxZXdHbUI6QWEodXhWNWFvaEVOZEVHVlhqfQ==" ],
      "Accept" : [ "text/plain, */*" ]
    },
    "keepAlive" : true,
    "secure" : false,
    "protocol" : "HTTP_1_1",
    "localAddress" : "127.0.0.1:62609",
    "remoteAddress" : "127.0.0.1:62651"
  }
17:10:22.228 [MockServer-EventLog0] INFO org.mockserver.log.MockServerEventLog -- 62609 request:
  {
    "method" : "GET",
    "path" : "/xEpwx/lnflDhYIe",
    "queryStringParameters" : {
      "re1kbe" : [ "?t6a:6-$TtF" ],
      "piufSbS3n" : [ "P$Ltw\\\\ZKBo]SP;jJ" ],
      "kFWJFNM" : [ "|{MJ]h)DtQYE" ],
      "UgD1t" : [ "?a>i=YvgxoBkgjR" ],
      "HRH5q1" : [ "3'Kd323M9j" ]
    },
    "headers" : {
      "content-length" : [ "0" ],
      "User-Agent" : [ "Java/17.0.11" ],
      "Host" : [ "localhost:62609" ],
      "Connection" : [ "keep-alive" ],
      "Authorization" : [ "Basic ZlVxZXdHbUI6QWEodXhWNWFvaEVOZEVHVlhqfQ==" ],
      "Accept" : [ "text/plain, */*" ]
    },
    "keepAlive" : true,
    "secure" : false,
    "protocol" : "HTTP_1_1",
    "localAddress" : "127.0.0.1:62609",
    "remoteAddress" : "127.0.0.1:62651"
  }
 matched expectation:
  {
    "httpRequest" : {
      "method" : "GET"
    },
    "httpResponse" : {
      "statusCode" : 200,
      "reasonPhrase" : "OK",
      "headers" : {
        "Content-Type" : [ "text/plain" ]
      },
      "body" : "OdKkQU'1VYwr9hsL`AW69`5}jEty)_uY:+k0"
    },
    "id" : "f9b2796f-1f04-4fc9-9332-45053207b615",
    "priority" : 0,
    "timeToLive" : {
      "unlimited" : true
    },
    "times" : {
      "unlimited" : true
    }
  }
17:10:22.236 [MockServer-EventLog0] INFO org.mockserver.log.MockServerEventLog -- 62609 returning response:
  {
    "statusCode" : 200,
    "reasonPhrase" : "OK",
    "headers" : {
      "Content-Type" : [ "text/plain" ]
    },
    "body" : "OdKkQU'1VYwr9hsL`AW69`5}jEty)_uY:+k0"
  }
 for request:
  {
    "method" : "GET",
    "path" : "/xEpwx/lnflDhYIe",
    "queryStringParameters" : {
      "re1kbe" : [ "?t6a:6-$TtF" ],
      "piufSbS3n" : [ "P$Ltw\\\\ZKBo]SP;jJ" ],
      "kFWJFNM" : [ "|{MJ]h)DtQYE" ],
      "UgD1t" : [ "?a>i=YvgxoBkgjR" ],
      "HRH5q1" : [ "3'Kd323M9j" ]
    },
    "headers" : {
      "content-length" : [ "0" ],
      "User-Agent" : [ "Java/17.0.11" ],
      "Host" : [ "localhost:62609" ],
      "Connection" : [ "keep-alive" ],
      "Authorization" : [ "Basic ZlVxZXdHbUI6QWEodXhWNWFvaEVOZEVHVlhqfQ==" ],
      "Accept" : [ "text/plain, */*" ]
    },
    "keepAlive" : true,
    "secure" : false,
    "protocol" : "HTTP_1_1",
    "localAddress" : "127.0.0.1:62609",
    "remoteAddress" : "127.0.0.1:62651"
  }
 for action:
  {
    "statusCode" : 200,
    "reasonPhrase" : "OK",
    "headers" : {
      "Content-Type" : [ "text/plain" ]
    },
    "body" : "OdKkQU'1VYwr9hsL`AW69`5}jEty)_uY:+k0"
  }
 from expectation:
  f9b2796f-1f04-4fc9-9332-45053207b615
17:10:54.210 [MockServer-EventLog0] INFO org.mockserver.log.MockServerEventLog -- 62609 verifying sequence that match:
  {
    "httpRequests" : [ {
      "method" : "GET",
      "path" : "/xEpwx/lnflDhYIe"
    } ]
  }
17:10:54.223 [MockServer-EventLog0] INFO org.mockserver.log.MockServerEventLog -- request sequence found:
  [{
    "method" : "GET",
    "path" : "/xEpwx/lnflDhYIe"
  }]
17:10:54.854 [MockServer-EventLog0] INFO org.mockserver.log.MockServerEventLog -- retrieved requests in json that match:
  { }

[Typraeurion]

I’ve just encountered another failure which shows the same issue with a leading exclamation point (!) character. You can force the bug to appear by prepending "!" + (or "?" +) to the random value string in the test method.

Expected :{GwjWg3R=[!)!.slU]in|Url(>2], J3WXeb=[!TJu)"m"Y#dc], JZ460KOc=[!91`#I0plxcFYZO|6], dvsSt5g=[!qzOP)GJ7cb], pqKdMh0C=[!:}$$Umw3^]bX:BjfR@], u3K4Z2g=[!hd7hn=4X0`|U]}
Actual   :{GwjWg3R=[)!.slU]in|Url(>2], JZ460KOc=[91`#I0plxcFYZO|6], u3K4Z2g=[hd7hn=4X0`|U], dvsSt5g=[qzOP)GJ7cb], J3WXeb=[TJu)"m"Y#dc], pqKdMh0C=[:}$$Umw3^]bX:BjfR@]}