Support TLS with username and password

[wiwats2019]

hi all,

I have to used external elasticsearch (5 node) for storage data in back end.
I enable https with searchguard plug in and try to run with below command

docker run -e SPAN_STORAGE_TYPE=elasticsearch  jaegertracing/jaeger-collector --log-level=debug --es.server-urls https://ip:9200 --es.username=jaeger --es.password=jaeger --es.tls.ca= /etc/ssl/certs/cacert.pem

and i got error

{"level":"fatal","ts":1549876373.3432007,"caller":"collector/main.go:103","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: Head https://ip:9200: Tunnel Forbidden: no Elasticsearch node available","errorVerbose":"no Elasticsearch node available\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.init\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:88\ngithub.com/jaegertracing/jaeger/pkg/es.init\n\t<autogenerated>:1\ngithub.com/jaegertracing/jaeger/plugin/storage/es.init\n\t<autogenerated>:1\ngithub.com/jaegertracing/jaeger/plugin/storage.init\n\t<autogenerated>:1\ngithub.com/jaegertracing/jaeger/cmd/env.init\n\t<autogenerated>:1\nmain.init\n\t<autogenerated>:1\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:189\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333\nhealth check timeout: Head https://ip:9200: Tunnel Forbidden\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).startupHealthcheck\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:1116\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.NewClient\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:244\ngithub.com/jaegertracing/jaeger/pkg/es/config.(*Configuration).NewClient\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/es/config/config.go:95\ngithub.com/jaegertracing/jaeger/plugin/storage/es.(*Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/factory.go:78\ngithub.com/jaegertracing/jaeger/plugin/storage.(*Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:90\nmain.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:102\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:203\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333\nfailed to create primary Elasticsearch client\ngithub.com/jaegertracing/jaeger/plugin/storage/es.(*Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/factory.go:80\ngithub.com/jaegertracing/jaeger/plugin/storage.(*Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:90\nmain.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:102\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:203\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:103\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:203\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201"}

so i test run with command

docker run -e SPAN_STORAGE_TYPE=elasticsearch  jaegertracing/jaeger-collector --log-level=debug --es.server-urls https://ip:9200 

and got same error too
could jeager team help me how to pass the parameter for https to connecto es( for solve this issue)

elasticsearch = v6.4.0
searchguard= search-guard-6-6.4.0-23.0

[pavolloffay]

This highly depends on your searchguard configuration. Make sure that the user has privileges to access/create/search.. jaeger indices

[wiwats2019]

The user can all access and create index and i try to use supper user and get the same error too

[pavolloffay]

@wiwats2019 try using tagged jaeger version e.g 1.9. The latest tag had a bug where all storages - including archive required proper configuration. We made the archive optional.

[wiwats2019]

@pavolloffay after you comment i pull the image and try to use jaeger version 1.9 and get the error below

command

docker run \

-p14269:14269
-p14268:14268
-p14267:14267
-p14250:14250
-p9411:9411
-e SPAN_STORAGE_TYPE=elasticsearch
-v /home/appadm/src_images/jaeger/cert/cacert-esd-eam.pem:/certs/cacert-esd-eam.pem
--name jaeger-collector
--rm jaegertracing/jaeger-collector:1.9
--es.server-urls=https://ip:9200
--es.index-prefix=production
--es.username=jaeger
--es.password=jaeger
--es.tls.ca=/certs/cacert-esd-eam.pem

and get the error "no Elasticsearch node available"

{
"level": "info",
"ts": 1550462246.7197754,
"caller": "healthcheck/handler.go:99",
"msg": "Health Check server started",
"http-port": 14269,
"status": "unavailable"
}
{
"level": "fatal",
"ts": 1550462251.7932832,
"caller": "collector/main.go:103",
"msg": "Failed to init storage factory",
"error": "health check timeout: no Elasticsearch node available",
"errorVerbose": "no Elasticsearch node available\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.init\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:88\ngithub.com/jaegertracing/jaeger/pkg/es.init\n\t:1\ngithub.com/jaegertracing/jaeger/plugin/storage/es.init\n\t:1\ngithub.com/jaegertracing/jaeger/plugin/storage.init\n\t:1\ngithub.com/jaegertracing/jaeger/cmd/env.init\n\t:1\nmain.init\n\t:1\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:189\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333\nhealth check timeout\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.(*Client).startupHealthcheck\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:1114\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.NewClient\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:244\ngithub.com/jaegertracing/jaeger/pkg/es/config.(*Configuration).NewClient\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/es/config/config.go:89\ngithub.com/jaegertracing/jaeger/plugin/storage/es.(*Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/factory.go:69\ngithub.com/jaegertracing/jaeger/plugin/storage.(*Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:90\nmain.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:102\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:203\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333",
"stacktrace": "main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:103\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:203\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201"
}

but when i not used https the jaeger is work so please help to suggest me

[appadm@monts02 ~]$ docker run \

-p14269:14269
-p14268:14268
-p14267:14267
-p14250:14250
-p9411:9411
-e SPAN_STORAGE_TYPE=elasticsearch
--name jaeger-collector
--rm jaegertracing/jaeger-collector:1.9
--es.server-urls=http://ip:19200
--es.index-prefix=production

{"level":"info","ts":1550461885.6275349,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
{"level":"info","ts":1550461885.6429968,"caller":"static/strategy_store.go:79","msg":"No sampling strategies provided, using defaults"}
{"level":"info","ts":1550461885.6458397,"caller":"collector/main.go:139","msg":"Starting jaeger-collector TChannel server","port":14267}
{"level":"info","ts":1550461885.645977,"caller":"grpcserver/grpc_server.go:64","msg":"Starting jaeger-collector gRPC server","grpc-port":"14250"}
{"level":"info","ts":1550461885.6460555,"caller":"collector/main.go:153","msg":"Registering metrics handler with HTTP server","route":"/metrics"}
{"level":"info","ts":1550461885.6461244,"caller":"collector/main.go:162","msg":"Starting jaeger-collector HTTP server","http-port":14268}
{"level":"info","ts":1550461885.6461449,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}

[pavolloffay]

I think we do not support username+pass with CA cert at the moment.