Bug Report: vttestserver-latest Connection refused on grpc port #17396

danjwilks · 2024-12-16T22:55:19Z

Overview of the Issue

The gRPC port on for the latest version of the vttestserver (vttestserver:mysql80) container is not accessible, while it was accessible in the previous versions vttestserver:v21.0.0-mysql80 & vttestserver:v20.0.3-mysql80

Reproduction Steps

In our repro below, we set PORT=33574 and try to establish a TCP connection to the container on port 33575 which should be the grpc port as per documentation. For vttestserver-latest, we see connect to 172.17.0.4 port 33575 (tcp) failed: Connection refused whereas for vttestserver-21.0.0 and vttestserver-20.0.3 we see Connection to 172.17.0.3 33575 port [tcp/*] succeeded!. Below you can find the repro script followed by its output in the logs fragment section.

#!/usr/bin/env bash

set -eo pipefail

docker_run_args=(
    '-e' 'PORT=33574'
    '-e' 'KEYSPACES=test,unsharded'
    '-e' 'NUM_SHARDS=2,1'
    '-e' 'MYSQL_MAX_CONNECTIONS=70000'
    '-e' 'MYSQL_BIND_HOST=0.0.0.0'
    '--health-cmd="mysqladmin ping -h127.0.0.1 -P33577"'
    '--health-interval=5s'
    '--health-timeout=2s'
    '--health-retries=5'
    '--rm'
)

mysql_port=33577
grpc_port=33575

version_20_container_name=vttestserver-20.0.3
version_21_container_name=vttestserver-21.0.0
latest_container_name=vttestserver-latest

trap "docker kill ${version_20_container_name} >/dev/null ; docker kill ${version_21_container_name} >/dev/null ; docker kill ${latest_container_name} >/dev/null" EXIT

# Version 20.0.3
echo '|------ Version 20.0.3 ---------------------------------------------------------------------------------------------------|'

echo 'Pre-pulling image to keep logs clean'
docker pull vitess/vttestserver:v20.0.3-mysql80
echo ' '

echo "Starting ${version_20_container_name}"
container_id=$(docker run --detach --name="${version_20_container_name}" "${docker_run_args[@]}" vitess/vttestserver:v20.0.3-mysql80)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${version_20_container_name}")
echo "${version_20_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo -e "|-------------------------------------------------------------------------------------------------------------------------|\n\n"

# Version 21.0.0
echo '|------ Version 21.0.0 ---------------------------------------------------------------------------------------------------|'

echo 'Pre-pulling image to keep logs clean'
docker pull vitess/vttestserver:v21.0.0-mysql80
echo ' '

echo "Starting ${version_21_container_name}"
container_id=$(docker run --detach --name="${version_21_container_name}" "${docker_run_args[@]}" vitess/vttestserver:v21.0.0-mysql80)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${version_21_container_name}")
echo "${version_21_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo -e "|-------------------------------------------------------------------------------------------------------------------------|\n\n"

# Latest
echo '|---------- Latest -------------------------------------------------------------------------------------------------------|'

echo 'Pre-pulling image to keep logs clean'
docker pull vitess/vttestserver:mysql80
echo ' '

echo "Starting ${latest_container_name}"
container_id=$(docker run --detach --name="${latest_container_name}" "${docker_run_args[@]}" vitess/vttestserver:mysql80)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${latest_container_name}")
echo "${latest_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo '|-------------------------------------------------------------------------------------------------------------------------|'

Binary Version

vttestserver-latest

Operating System and Environment details

n/a

Log Fragments

|------ Version 20.0.3 ---------------------------------------------------------------------------------------------------|
Pre-pulling image to keep logs clean
v20.0.3-mysql80: Pulling from vitess/vttestserver
6dce3b49cfe6: Pull complete 
c5980ff74b97: Pull complete 
6439b78e07bb: Pull complete 
634d6d66fbba: Pull complete 
09124c96e3c3: Pull complete 
7b6686cf96ac: Pull complete 
decd25bf86f6: Pull complete 
20936e025e2d: Pull complete 
Digest: sha256:683a99d2a3bbe7751acf7ac268055426306f1261c084152eade175319eff730e
Status: Downloaded newer image for vitess/vttestserver:v20.0.3-mysql80
docker.io/vitess/vttestserver:v20.0.3-mysql80
 
Starting vttestserver-20.0.3
Waiting for container to initialize...
vttestserver-20.0.3 IP: 172.17.0.2
Running netcat for MySQL port
Connection to 172.17.0.2 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
Connection to 172.17.0.2 33575 port [tcp/*] succeeded!
|-------------------------------------------------------------------------------------------------------------------------|


|------ Version 21.0.0 ---------------------------------------------------------------------------------------------------|
Pre-pulling image to keep logs clean
v21.0.0-mysql80: Pulling from vitess/vttestserver
6dce3b49cfe6: Already exists 
1ea862120675: Pull complete 
7752a6b7b984: Pull complete 
0d1dcb0a093e: Pull complete 
c41aa64d82d6: Pull complete 
6484229778f0: Pull complete 
c60fbe59811e: Pull complete 
f7690de602ed: Pull complete 
Digest: sha256:ce595dba0a37695a0ead60110d91937c01fa064e04df95be6acf449d9aa01e45
Status: Downloaded newer image for vitess/vttestserver:v21.0.0-mysql80
docker.io/vitess/vttestserver:v21.0.0-mysql80
 
Starting vttestserver-21.0.0
Waiting for container to initialize...
vttestserver-21.0.0 IP: 172.17.0.3
Running netcat for MySQL port
Connection to 172.17.0.3 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
Connection to 172.17.0.3 33575 port [tcp/*] succeeded!
|-------------------------------------------------------------------------------------------------------------------------|


|---------- Latest -------------------------------------------------------------------------------------------------------|
Pre-pulling image to keep logs clean
mysql80: Pulling from vitess/vttestserver
69fb10dc82f9: Pull complete 
9530dbc786c9: Pull complete 
4723cf4cfe1f: Pull complete 
d1c733e95d1a: Pull complete 
e47c9887790a: Pull complete 
6a29be3204ac: Pull complete 
da6352e8c108: Pull complete 
f20a407ca0dd: Pull complete 
Digest: sha256:60dbca30eb144afbe83fe8a97bf290dc966e830f3d109402b4a271c3ab90ae63
Status: Downloaded newer image for vitess/vttestserver:mysql80
docker.io/vitess/vttestserver:mysql80
 
Starting vttestserver-latest
Waiting for container to initialize...
vttestserver-latest IP: 172.17.0.4
Running netcat for MySQL port
Connection to 172.17.0.4 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
nc: connect to 172.17.0.4 port 33575 (tcp) failed: Connection refused
|-------------------------------------------------------------------------------------------------------------------------|

The text was updated successfully, but these errors were encountered:

anirbanmu · 2025-01-01T01:06:19Z

I work with @danjwilks and looked into this issue as well.

Wanted to add some context for how we encountered this issue -- we use vttestserver image to run a service container in Github Actions so that our tests can have Vitess available. The test job itself runs in a container, and my guess is that for container -> container communication in a Github Actions job, the Docker network interface IP is utilized.

This guess is what made us investigate whether we were able to hit the gRPC port using the Docker network interface.

My current suspicion is that this might be something to do with

--grpc_bind_address string                                         Bind address for gRPC calls. If empty, listen on all addresses.

Not specifying this argument might not actually be listening on all addresses even though the documentation for v21.0 says that if given nothing, we listen on all addresses.

To try to prove this I modified the run.sh for vttestserver to allow passing in a value for grpc_bind_address here on my own branch.

I built the container image on my branch using

docker buildx build -f docker/vttestserver/Dockerfile.mysql80 -t vitess/vttestserver:custom-local .

I extended the repro script to also test against my locally built vitess/vttestserver:custom-local image. With my modification to pass --grpc_bind_address 0.0.0.0 when no env var is set for the container, the gRPC port is accessible via netcat on the Docker network interface. I think this confirms that when --grpc_bind_address is not passed to vttestserver it is not actually listening on all addresses.

The modification I made on my branch to run.sh would suffice to unblock our usage I believe but there may be a more proper fix in vttestserver itself? Let me know though, I'm happy to make a PR from my branch in its current state if you'd like to take that modified run.sh change.

I've copy-pasted the extended repro script & the log output below.

Script

#!/usr/bin/env bash

set -eo pipefail

docker_run_args=(
    '-e' 'PORT=33574'
    '-e' 'KEYSPACES=test,unsharded'
    '-e' 'NUM_SHARDS=2,1'
    '-e' 'MYSQL_MAX_CONNECTIONS=70000'
    '-e' 'MYSQL_BIND_HOST=0.0.0.0'
    '--health-cmd="mysqladmin ping -h127.0.0.1 -P33577"'
    '--health-interval=5s'
    '--health-timeout=2s'
    '--health-retries=5'
    '--rm'
)

mysql_port=33577
grpc_port=33575

version_20_container_name=vttestserver-20.0.3
version_21_container_name=vttestserver-21.0.0
latest_container_name=vttestserver-latest
local_container_name=vttestserver-local

trap "docker kill ${version_20_container_name} >/dev/null ; docker kill ${version_21_container_name} >/dev/null ; docker kill ${latest_container_name} >/dev/null ; docker kill ${local_container_name} >/dev/null" EXIT

# Version 20.0.3
echo '|------ Version 20.0.3 ---------------------------------------------------------------------------------------------------|'

echo 'Pre-pulling image to keep logs clean'
docker pull vitess/vttestserver:v20.0.3-mysql80
echo ' '

echo "Starting ${version_20_container_name}"
container_id=$(docker run --detach --name="${version_20_container_name}" "${docker_run_args[@]}" vitess/vttestserver:v20.0.3-mysql80)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${version_20_container_name}")
echo "${version_20_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo -e "|-------------------------------------------------------------------------------------------------------------------------|\n\n"

# Version 21.0.0
echo '|------ Version 21.0.0 ---------------------------------------------------------------------------------------------------|'

echo 'Pre-pulling image to keep logs clean'
docker pull vitess/vttestserver:v21.0.0-mysql80
echo ' '

echo "Starting ${version_21_container_name}"
container_id=$(docker run --detach --name="${version_21_container_name}" "${docker_run_args[@]}" vitess/vttestserver:v21.0.0-mysql80)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${version_21_container_name}")
echo "${version_21_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo -e "|-------------------------------------------------------------------------------------------------------------------------|\n\n"

# Latest
echo '|---------- Latest -------------------------------------------------------------------------------------------------------|'

echo 'Pre-pulling image to keep logs clean'
docker pull vitess/vttestserver:mysql80
echo ' '

echo "Starting ${latest_container_name}"
container_id=$(docker run --detach --name="${latest_container_name}" "${docker_run_args[@]}" vitess/vttestserver:mysql80)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${latest_container_name}")
echo "${latest_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo -e "|-------------------------------------------------------------------------------------------------------------------------|\n\n"

# Local
echo '|---------- Local --------------------------------------------------------------------------------------------------------|'

echo "Starting ${local_container_name}"
container_id=$(docker run --detach --name="${local_container_name}" "${docker_run_args[@]}" vitess/vttestserver:custom-local)
echo 'Waiting for container to initialize...'
sleep 20s

ip_addr=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${local_container_name}")
echo "${local_container_name} IP: ${ip_addr}"

echo "Running netcat for MySQL port"
nc -zv "${ip_addr}" "${mysql_port}" || true
echo "Running netcat for gRPC port"
nc -zv "${ip_addr}" "${grpc_port}" || true

echo '|-------------------------------------------------------------------------------------------------------------------------|'

Logs

|------ Version 20.0.3 ---------------------------------------------------------------------------------------------------|
Pre-pulling image to keep logs clean
v20.0.3-mysql80: Pulling from vitess/vttestserver
Digest: sha256:683a99d2a3bbe7751acf7ac268055426306f1261c084152eade175319eff730e
Status: Image is up to date for vitess/vttestserver:v20.0.3-mysql80
docker.io/vitess/vttestserver:v20.0.3-mysql80

Starting vttestserver-20.0.3
Waiting for container to initialize...
vttestserver-20.0.3 IP: 172.17.0.2
Running netcat for MySQL port
Connection to 172.17.0.2 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
Connection to 172.17.0.2 33575 port [tcp/*] succeeded!
|-------------------------------------------------------------------------------------------------------------------------|


|------ Version 21.0.0 ---------------------------------------------------------------------------------------------------|
Pre-pulling image to keep logs clean
v21.0.0-mysql80: Pulling from vitess/vttestserver
Digest: sha256:ce595dba0a37695a0ead60110d91937c01fa064e04df95be6acf449d9aa01e45
Status: Image is up to date for vitess/vttestserver:v21.0.0-mysql80
docker.io/vitess/vttestserver:v21.0.0-mysql80

Starting vttestserver-21.0.0
Waiting for container to initialize...
vttestserver-21.0.0 IP: 172.17.0.3
Running netcat for MySQL port
Connection to 172.17.0.3 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
Connection to 172.17.0.3 33575 port [tcp/*] succeeded!
|-------------------------------------------------------------------------------------------------------------------------|


|---------- Latest -------------------------------------------------------------------------------------------------------|
Pre-pulling image to keep logs clean
mysql80: Pulling from vitess/vttestserver
Digest: sha256:283ec8180f141f9677171d7c001addb4f157b45dbec36c93f6e546d1a31190a3
Status: Image is up to date for vitess/vttestserver:mysql80
docker.io/vitess/vttestserver:mysql80

Starting vttestserver-latest
Waiting for container to initialize...
vttestserver-latest IP: 172.17.0.4
Running netcat for MySQL port
Connection to 172.17.0.4 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
nc: connect to 172.17.0.4 port 33575 (tcp) failed: Connection refused
|-------------------------------------------------------------------------------------------------------------------------|


|---------- Local --------------------------------------------------------------------------------------------------------|
Starting vttestserver-local
Waiting for container to initialize...
vttestserver-local IP: 172.17.0.5
Running netcat for MySQL port
Connection to 172.17.0.5 33577 port [tcp/*] succeeded!
Running netcat for gRPC port
Connection to 172.17.0.5 33575 port [tcp/*] succeeded!
|-------------------------------------------------------------------------------------------------------------------------|

mattlord · 2025-01-01T20:06:19Z

I'm assuming that this was an unintentional side-effect of #17231

mattlord · 2025-01-01T20:13:45Z

Thank you for letting us know, @danjwilks and @anirbanmu ! Are you interested in opening a PR? I think this is the change we need:

diff --git a/go/vt/vttest/vtprocess.go b/go/vt/vttest/vtprocess.go
index 6371811a60..1719fafd8a 100644
--- a/go/vt/vttest/vtprocess.go
+++ b/go/vt/vttest/vtprocess.go
@@ -205,7 +205,7 @@ func VtcomboProcess(environment Environment, args *Config, mysql MySQLManager) (
 	if args.VtComboBindAddress != "" {
 		vtcomboBindAddress = args.VtComboBindAddress
 	}
-	grpcBindAddress := "127.0.0.1"
+	grpcBindAddress := ""
 	if servenv.GRPCBindAddress() != "" {
 		grpcBindAddress = servenv.GRPCBindAddress()
 	}

Since I believe that is making it so there is ALWAYS a bind address value passed to the process (an empty/zero string value is what led to the previous behavior).

Clearly we have some test coverage gaps as well, so we should try and add more unit tests in this area.

GuptaManan100 · 2025-01-02T05:35:33Z

Matt is correct, the fix should be what he proposed

…If none is passed in, it should pass nothing causing gRPC port to be bound to all interfaces (instead of just 127.0.0.1). Fixes vitessio#17396

anirbanmu · 2025-01-02T16:47:11Z

Thanks all! I verified that the suggested fix works & opened the PR. I haven't added tests as I'm unfamiliar with the test setup in this project. Please feel free to add tests to the PR if you think it necessary; I believe edits from maintainers is enabled on the PR.

danjwilks added Needs Triage This issue needs to be correctly labelled and triaged Type: Bug labels Dec 16, 2024

GuptaManan100 added Component: vttestserver and removed Needs Triage This issue needs to be correctly labelled and triaged labels Dec 19, 2024

mattlord mentioned this issue Jan 1, 2025

Add support for receiving grpc_bind_adress on vttestserver #17231

Merged

5 tasks

anirbanmu linked a pull request Jan 2, 2025 that will close this issue

vttestserver should only pass a BindAddressGprc if one is passed in. … #17457

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug Report: vttestserver-latest Connection refused on grpc port #17396

Bug Report: vttestserver-latest Connection refused on grpc port #17396

danjwilks commented Dec 16, 2024 •

edited

Loading

anirbanmu commented Jan 1, 2025 •

edited

Loading

mattlord commented Jan 1, 2025

mattlord commented Jan 1, 2025

GuptaManan100 commented Jan 2, 2025

anirbanmu commented Jan 2, 2025

Bug Report: vttestserver-latest Connection refused on grpc port #17396

Bug Report: vttestserver-latest Connection refused on grpc port #17396

Comments

danjwilks commented Dec 16, 2024 • edited Loading

Overview of the Issue

Reproduction Steps

Binary Version

Operating System and Environment details

Log Fragments

anirbanmu commented Jan 1, 2025 • edited Loading

Script

Logs

mattlord commented Jan 1, 2025

mattlord commented Jan 1, 2025

GuptaManan100 commented Jan 2, 2025

anirbanmu commented Jan 2, 2025

danjwilks commented Dec 16, 2024 •

edited

Loading

anirbanmu commented Jan 1, 2025 •

edited

Loading