gocql throwing 'unable to connect to initial hosts: tls: DialWithDialer timed out'

[nutharsh]

Hello,
I am unable to connect to Cassandra flavor of cosmos db. I am trying to run docker all in one image to remote cosmos db cassandra API from my local mac machine.

Appreciate any help in resolving below issue.

I am able to connect and create schema via cqlsh. But, docker image is unable to pass through ssl stage it seems.

On docker image run command, it throws below error,

{"level":"fatal","ts":1555153542.5605443,"caller":"query/main.go:87","msg":"Failed to init storage factory","error":"gocql: unable to create session: control: unable to connect to initial hosts: tls: DialWithDialer timed out","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/main.go:87\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/main.go:147\nruntime.main\n\t/home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/proc.go:200"}

cmd used to run and connect to remote cassandra

docker run --rm
--name jaeger
-e SPAN_STORAGE_TYPE=cassandra
-e CASSANDRA_SERVERS=abcd.cassandra.cosmos.azure.com
-e CASSANDRA_PORT=10350
-e CASSANDRA_PASSWORD=pwd
-e CASSANDRA_USERNAME=user
-e CASSANDRA_TLS=false
-e CASSANDRA_KEYSPACE=jaeger_v1_test
-e CASSANDRA_CONNECTIONS_PER_HOST=1
-e CASSANDRA_TLS_VERIFY_HOST=false
jaegertracing/all-in-one:latest

[yurishkuro]

I'd try ruling out networking issues by running native binary instead of container. Can any other local apps access Cosmos?

[yurishkuro]

It looks like either a connectivity issue or perhaps the protocol mismatch. Does cqlsh work?

[vprithvi]

@nutharsh I'd also recommend turning on gocql debug logs to capture more data.
Unfortunately, it's a bit involved. You'd have to create a build with the gocql_debug tag by passing -tags="gocql_debug to go build.

[sagaranand015]

@yurishkuro @vprithvi : I, too ran into the same issue while making the collector component talk to an internal Cassandra Cluster.
Please note the following observations:

1. I'm able to connect to the same list of Cassandra IPs using another go program with gocql. (See below please).
2. For debugging, I changed the cluster connection config and hardcoded the same thing as in the attached go program. It still threw the same error. (https://github.com/sagaranand015/jaeger-debug/blob/master/pkg/cassandra/config/config.go)

=== Working go program ===

package main

import (
	"fmt"
	"log"
	"time"

	"github.com/gocql/gocql"
)

func main() {
	fmt.Println("=========== CLUSTER SETTINGS ===========")

	cluster := gocql.NewCluster("<IP Address list>")  // shared cluster config
	cluster.Consistency = gocql.Quorum
	cluster.ProtoVersion = 4
	cluster.ConnectTimeout = time.Second * 10
	
	cluster.Authenticator = gocql.PasswordAuthenticator{Username: "my_username", Password: "my_password"}

	cluster.SslOpts = &gocql.SslOptions {
		CaPath: "<path to cer/pem file>",
	}

	fmt.Println("=========== PRINTING CLUSTER ========")
	fmt.Println(cluster)
	fmt.Println("=========== DONE PRINTING CLUSTER ========")

	session, err := cluster.CreateSession()
	if err != nil {
		log.Println(err)
		return
	}
	defer session.Close()

	// create table
	err = session.Query("CREATE TABLE IF NOT EXISTS cass_abhi_go.sleep_study (name text, study_date date, sleep_time_hours float, PRIMARY KEY (name, study_date));").Exec()
	if err != nil {
		log.Println(err)
		return
	}

}

I've tried to do the same thing in Jaeger config code, but unable to connect to initial hosts: tls: DialWithDialer timed out seems to persists.

I did try passing the gocql_debug tag to the build command as follows, but no additional information was thrown in the logs.

go build -o collector -tags="gocql_debug" main.go

Any pointers here would be extremely helpful.
Thanks a ton!

[yurishkuro]

@sagaranand015

• it may be a different issue from the top of this ticket because that one had TLS off
• if you have a working program, is it also working when running from a Docker container? And is it different from the configuration sequence in Jaeger?

[sagaranand015]

@yurishkuro : Tried with some more debugging and looks like this happens when the connectTimeout parameter in Cassandra cluster config is less than a particular threshold. Was able to successfully connect to Cassandra by hardcoding a larger connectTimeout in cluster config.

Do you think this parameter can be driven from the flags?
Can I create an issue and raise a PR with proposed changes?

[yurishkuro]

Sounds reasonable. Don't need a separate issue, just a PR.

[sagaranand015]

@yurishkuro : Raised the PR #1647 as per above discussion. Please review. Thanks

[pawarvijay]

Solution -> i used docer compose from this location and every thing worked AWESOME
https://github.com/jaegertracing/jaeger/tree/master/docker-compose