regression on Manticore 0.8.0 due port being part of host

[kares]

... with Manticore 0.8.0 the following configuration started failing:

  hosts => ["localhost:9243"]
  ssl => true

with

Manticore::ClientProtocolException: URI does not specify a valid host name: https://localhost:9243:9200/
                        initialize at /opt/local/rvm/gems/jruby-9.2.20.0@ls/gems/manticore-0.8.0-java/lib/manticore/response.rb:36
                              call at /opt/local/rvm/gems/jruby-9.2.20.0@ls/gems/manticore-0.8.0-java/lib/manticore/response.rb:79
                         call_once at /opt/local/rvm/gems/jruby-9.2.20.0@ls/gems/manticore-0.8.0-java/lib/manticore/response.rb:274
                              code at /opt/local/rvm/gems/jruby-9.2.20.0@ls/gems/manticore-0.8.0-java/lib/manticore/response.rb:158

caused by:

 Java::OrgApacheHttpClient::ClientProtocolException:
   URI does not specify a valid host name: https://localhost:9243:9200/
   org.apache.http.impl.client.CloseableHttpClient.determineTarget(org/apache/http/impl/client/CloseableHttpClient.java:95)

---

While the issue seems to be caused by the Manticore 0.8 upgrade it's questionable whether host detection in the client/plugin is correct in this case. The 'normalized' options from Elasticsearch::Client (passed down to Manticore) end up as:

@options={:hosts=>[{:host=>"localhost:9243", :scheme=>"https", :protocol=>"https", :port=>9200}]

... clearly elasticsearch-transport filled in a default 9200 port assuming a raw host was specified and this ends up being passed down to the underlying HttpClient library built as scheme://host:port ...

This behavior is not new in elasticsearch gem (tested versions 7.17.0, 7.16, 7.15 and 7.14.1)

---

NOTE: a work-around is to change the original Logstash configuration to hosts => ["https://localhost:9243"] (no need for the ssl setting).

[andsel]

In registration of the plug it could check the form of hosts. If they are without http or https and specify a port, like in localhost:1234 should it print a warning message? Or fail the config? But failing, in this case, is breaking change.

[nicenemo]

I see this problem too with Logstash 7.17.0. It is also inconsistent with the Elasticsearch output plugin of 7.17.
for the output plugin to work with https URLs they should be prefixed with https://. That does not seem to work for the Elasticsearch filter plugin.

We have the following in our logstash pipeline to partially prevent posting duplicates to a data stream.

    elasticsearch {
      ssl => true
      hosts => ["our_es_cloud_end_point:9243"]
      api_key => "our_key_id:not_going_to_tell_you"
      index => ".ds-logs-some-data-*"
      query_template => "/opt/data/query-templates/document-exists.json"
      add_tag => [ "exists" ]
    }
    if "exists" in [tags] {
      drop { }
    }

I now have to disable this filter.

The Elasticsearch input plugin still works without https:// prefix. With https:// prefix it does not.

[kares]

@nicenemo sorry for the inconvenience this is causing, we managed to push an update #156

please bin/logstash-plugin update logstash-filter-elasticsearch (to 3.11.1) and let us know if you stumble upon anything else, after the upgrade your:

    elasticsearch {
      ssl => true
      hosts => ["our_es_cloud_end_point:9243"]
      # ...
    }

... should be working again

---

The Elasticsearch input plugin still works without https:// prefix. With https:// prefix it does not.

Guess you mean it does not work without the prefix just like the filter, I initially thought it would work but did not try.
We'll follow up with the input, tracking the issue at logstash-plugins/logstash-input-elasticsearch#166