Low Level Rest Client shows successful response bad request with empty response

[kevcodez]

Using the AWS Elasticsearch Service with the latest possible version (6.0.1) deployed.

When doing basic requests like an index check using HEAD /<indexname> I am getting a response with status code 400 and an empty response body. The request works fine with a basic HTTP Client.

I was unable to produce this locally. This seems to be an issue in the low level REST Client.

Elasticsearch version 6.0.1

@Bean
open fun elasticsearchLowClient(): RestClient {
    return RestClient.builder(
        HttpHost(elasticsearchHost, elasticsearchPort, elasticsearchProtocol)
    ).build()
}

val indexExistsResponse = elasticsearchLowClient.performRequest("HEAD", index, mapOf()) // <- 400 empty response

if (indexExistsResponse.statusLine.statusCode == 404) {
    elasticsearchLowClient.performRequest(
            "PUT", index, hashMapOf(),
            StringEntity("...valid", ContentType.APPLICATION_JSON)
    )

}

[jasontedor]

We have a test case that tests that HEAD requests for an existing index operate correctly.

This is not to say that there is not a bug here, but it is to say that without a correlating log message or a reproduction, we are unlikely to make much progress on this issue. Do you have anything here? What happens if you send the request without GET instead of HEAD?

[jasontedor]

FYI @elastic/es-core-infra.

[kevcodez]

@jasontedor Same goes for GET - 400 bad request with empty response. Executing both GET and HEAD via curl against the ES instance did just fine.

Switchting to the basic apache http client with the same request URL resulted in a 200 OK rather than 400. This was just getting more strange, as I could not reproduce this locally.

I will try to get some logs out of the AWS service by tomorrow, I cannot access the logs today.

[jasontedor]

Do you have a proxy between the client and the instance? There might be a problem with that here; I did just now test the client through a proxy and I had no troubles, but maybe your proxy configuration is different.

This worked fine for me:

public class Head {

    public static void main(String[] args) throws IOException {
        final HttpHost host = new HttpHost("<redacted>", 9243, "https");
        final Header header = new BasicHeader("Authorization", "Basic <redacted>");
        try (RestClient client = RestClient.builder(host).setDefaultHeaders(new Header[] { header }).build()) {
            final Response response = client.performRequest("HEAD", "/metricbeat-6.2.1-2018.03.07", Collections.emptyMap());
            System.out.println(response.getStatusLine());
        }
    }

}

where the address I used here is a proxy.

Output:

09:44:56 [jason:~/src/jasontedor/head] $ ./gradlew run

> Task :run 
HTTP/1.1 200 OK


BUILD SUCCESSFUL in 1s
2 actionable tasks: 2 executed

[jasontedor]

No additional feedback, closing.

[gskoropada]

@kevcodez I've just came across the same issue and figured out that the problem was a missing / in the URI.
It worked fine with a local Elasticsearch docker container, but got 400s from AWS.
When I added that / it worked just fine.

[aug24]

I'll add to this for posterity, as it's taken me ~15 hours to work this one out:

When making a new high level ElasticSearch java client, and making requests via the underlying low level client, to an AWS ELB/ALB, failure to put the leading slash on the URI will result in a 400 from the ELB/ALB, EVEN THOUGH direct connections (eg your test database) will work just fine.

The line causing the problem is (I think)
https://github.com/elastic/elasticsearch/blob/master/client/rest/src/main/java/org/elasticsearch/client/RestClient.java#L806

Which ought to be replaced with a check to see if the path begins with "/" and prepends it if not.

[edit]

Just found out that this was discussed here: #30119 (comment)

The official 'fix' is to use the 'pathPrefix' option to always prepend with "/". The 'benefit' of this approach is that users can make invalid requests if they really really want to!

The remaining problem is that your error will simply be a 400 from AWS because the request is not valid HTTP, so if you are here reading this, then I hope I have improved your day!

[rorticus0502]

@kevcodez I've just came across the same issue and figured out that the problem was a missing / in the URI. It worked fine with a local Elasticsearch docker container, but got 400s from AWS. When I added that / it worked just fine.

I am getting 400 from AWS when running an updateByQuery, but the exact same borequest works when run against local elastic running in docker. I have just added a forward slash to the uri and it still works locally, so I am going to try this now in AWS env. If it works then a big thank you to you.

[rorticus0502]

I'll add to this for posterity, as it's taken me ~15 hours to work this one out:

When making a new high level ElasticSearch java client, and making requests via the underlying low level client, to an AWS ELB/ALB, failure to put the leading slash on the URI will result in a 400 from the ELB/ALB, EVEN THOUGH direct connections (eg your test database) will work just fine.

The line causing the problem is (I think) https://github.com/elastic/elasticsearch/blob/master/client/rest/src/main/java/org/elasticsearch/client/RestClient.java#L806

Which ought to be replaced with a check to see if the path begins with "/" and prepends it if not.

[edit]

Just found out that this was discussed here: #30119 (comment)

The official 'fix' is to use the 'pathPrefix' option to always prepend with "/". The 'benefit' of this approach is that users can make invalid requests if they really really want to!

The remaining problem is that your error will simply be a 400 from AWS because the request is not valid HTTP, so if you are here reading this, then I hope I have improved your day!

yes, thank you