DBAPIError exception wrapped from (pymysql.err.InternalError) (1105, u'vtgate: http://gnocchi-gnocchizone-vtgate:15000/: reserved connections are not supported on old gen gateway')

[Smita8081]

Overview of the Issue

I am working onto use vitess cluster integrated with externaldb, to connect to application. The application's pod that I am trying to connect to vtgate showed up below mentioned error in logs

2020-07-30 10:48:07,113 [1] DEBUG    gnocchi.service: ********************************************************************************
2020-07-30 10:48:09,097 [1] INFO     gnocchi.cli.manage: Upgrading indexer SQLAlchemyIndexer: mysql+pymysql://***:***@100.20.6.122:3306/gnocchi?charset=utf8
2020-07-30 10:48:09,205 [1] ERROR    oslo_db.sqlalchemy.exc_filters: DBAPIError exception wrapped from (pymysql.err.InternalError) (1105, u'vtgate: http://gnocchi-gnocchizone-vtgate-e0fa99c5-8dbbbdb49-qq74f:15000/: reserved connections are not supported on old gen gateway') (Background on this error at: http://sqlalche.me/e/2j85)

Reproduction Steps

Here I am using Gnocchi, which is an open source service, and integrating the mysql used in this service with Vitess cluster deployed using k8 operators.
vttablet in vitess cluster is in turn connected with externaldb (mysql-percona)

Binary version

[dev] root@m1-kms0001:~# vtgate --version
Version: 5245945 (Git branch 'HEAD') built on Mon May 25 19:17:32 UTC 2020 by runner@fv-az98 using go1.13.11 linux/amd64

Operating system and Environment details

OS, Architecture, and any other information you can provide
about the environment.

• Operating system :

[dev] root@m1-kms0001:~# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.7 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.7"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.7 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.7"

• Kernel version :

[dev] root@m1-kms0001:~# uname -sr
Linux 3.10.0-1062.12.1.el7.x86_64

• Architecture :

[dev] root@m1-kms0001:~# uname -m
x86_64

Log Fragments

Logs from vtgate pod are:

I0730 10:42:46.645347       1 legacy_healthcheck.go:493] HealthCheckUpdate(Type Change): gnocchizone-2007104406, tablet: gnocchizone-2007104406 (22.10.133.248), target gnocchi/- (SPARE) => gnocchi/- (MASTER), reparent time: 1596105763
I0730 10:42:46.645481       1 asm_amd64.s:1357] HealthCheckUpdate(Serving State): gnocchizone-2007104406, tablet: gnocchizone-2007104406 (22.10.133.248) serving => true for gnocchi/- (MASTER) reason: healthCheck update
E0730 10:43:35.235955       1 conn.go:1119] Got unhandled packet (default) from client 1 (192.168.1.102:39660), returning error: [4 97 108 101 109 98 105 99 95 118 101 114 115 105 111 110 0]
E0730 10:43:35.236538       1 conn.go:1119] Got unhandled packet (default) from client 1 (192.168.1.102:39660), returning error: [4 97 114 99 104 105 118 101 95 112 111 108 105 99 121 0]

here are the logs from etcd pod
10:02

2020-07-30 10:38:13.583800 N | etcdserver/membership: updated the cluster version from 3.0 to 3.3
2020-07-30 10:38:13.583904 I | etcdserver/api: enabled capabilities for version 3.3
proto: no coders for int
proto: no encoder for ValueSize int [GetProperties]
2020-07-30 10:42:46.639824 W | wal: sync duration of 2.795457359s, expected less than 1s
2020-07-30 10:42:46.640914 W | etcdserver: read-only range request "key:\"health\" " with result "range_response_count:0 size:4" took too long (1.670915063s) to execute
2020-07-30 10:42:47.628702 W | etcdserver: request "header:<ID:16969690523956385938 > txn:<compare:<target:MOD key:\"/vitess/gnocchi/local/gnocchizone/tablets/gnocchizone-2007104406/Tablet\" mod_revision:17 > success:<request_put:<key:\"/vitess/gnocchi/local/gnocchizone/tablets/gnocchizone-2007104406/Tablet\" value_size:113 >> failure:<>>" with result "size:14" took too long (600.703572ms) to execute
2020-07-30 10:42:47.628885 W | etcdserver: read-only range request "key:\"/vitess/gnocchi/local/gnocchizone/tablets/gnocchizone-2007104406/Tablet\" " with result "range_response_count:1 size:203" took too long (267.949778ms) to execute
2020-07-30 10:42:48.212000 W | etcdserver: read-only range request "key:\"/vitess/gnocchi/local/gnocchizone/keyspaces/gnocchi/SrvKeyspace\" " with result "range_response_count:1 size:114" took too long (760.273525ms) to execute
2020-07-30 10:42:48.212173 W | etcdserver: request "header:<ID:11710612059099188586 > txn:<compare:<target:MOD key:\"/vitess/gnocchi/global/keyspaces/gnocchi/shards/-/Shard\" mod_revision:7 > success:<request_put:<key:\"/vitess/gnocchi/global/keyspaces/gnocchi/shards/-/Shard\" value_size:39 >> failure:<>>" with result "size:14" took too long (265.162575ms) to execute
2020-07-30 10:42:48.947367 W | etcdserver: read-only range request "key:\"/vitess/gnocchi/local/gnocchizone/keyspaces/\" range_end:\"/vitess/gnocchi/local/gnocchizone/keyspaces0\" keys_only:true " with result "range_response_count:2 size:166" took too long (108.691866ms) to execute
2020-07-30 10:42:48.947430 W | etcdserver: read-only range request "key:\"/vitess/gnocchi/local/gnocchizone/tablets/gnocchizone-2007104406/Tablet\" " with result "range_response_count:1 size:203" took too long (119.029633ms) to execute
2020-07-30 10:42:48.947482 W | etcdserver: read-only range request "key:\"/vitess/gnocchi/local/gnocchizone/keyspaces/gnocchi/SrvKeyspace\" " with result "range_response_count:1 size:114" took too long (117.403597ms) to execute

[harshit-gangal]

The build you are running is quite old. You will not face the issue in release-7.0 branch or current master.

[Smita8081]

I am already using image docker.io/vitess/lite:v7.0.0 for vtctld, vtgate, vttablet, vtbackup pods running in k8 cluster. Do I need to update vtctlclient locally installed? but application is still connecting to v7 vitess cluster only i guess.

[deepthi]

Check your vtgate params - if you have -gateway_implementation=discoverygateway you will want to remove that.
It seems that SQLAlchemy issues queries that are not supported by the old gateway.
Also, I recommend using vitess/lite:latest temporarily, we are going to make a 7.0.1 release on Aug 25. Once that is out you can use that. It takes up to 24 hours for the docker images to become available.

[deepthi]

7.0.1 release is now available on dockerhub.

[Smita8081]

This issue do no more exist with new version(v7.0.1) of Vitess images. So I am closing it. Thanks for support.