Helm chart update hangs the pod

[boopathykpm]

Overview of the Issue

While updating the helm chart after making some changes to the values.yaml, the pod already created is in status "Terminating" and it continues to be in that state for long hours with no change.

Reproduction Steps

Update some values in the values.yaml and do helm upgrade, the pod is getting updated but hangs at status "Terminating" for long hours,

NAME                          READY   STATUS        RESTARTS   AGE
zone1-coda-test-0-replica-0   6/6     Terminating   0          23h

Because of this, some features are not working as expected.

Binary version

Example:

giaquinti@workspace:~$ vtgate --version
Version: a95cf5d (Git branch 'HEAD') built on Fri May 18 16:54:26 PDT 2018 by giaquinti@workspace using go1.10 linux/amd64

Below is the error from the mysql container inside the pod

I0225 07:21:59.230947       1 mysqld.go:480] Mysqld.Shutdown
I0225 07:21:59.231236       1 mysqld.go:518] No mysqld_shutdown hook, running mysqladmin directly
I0225 07:21:59.231647       1 mysqld.go:584] execCmd: /usr/bin/mysqladmin /usr/bin/mysqladmin [--defaults-extra-file=/tmp/example594926482 --shutdown-timeout=300 --connect-timeout=30 --wait=10 shutdown]
I0225 07:22:01.011124       1 mysqld.go:411] Mysqld.Start(1582615317) exit: <nil>
I0225 07:22:01.238283       1 mysqld.go:598] execCmd: /usr/bin/mysqladmin output:
I0225 07:22:01.238306       1 mysqld.go:559] Mysqld.Shutdown: waiting for socket file (/vtdataroot/tabletdata/mysql.sock) and pid file (/vtdataroot/tabletdata/mysql.pid) to disappear
I0225 07:22:01.239667       1 mysqld.go:349] Mysqld.Start(1582615321): No mysqld_start hook, running mysqld_safe directly
I0225 07:22:01.239707       1 mysqld.go:380] Mysqld.Start(1582615321) &exec.Cmd{Path:"/usr/bin/mysqld_safe", Args:[]string{"/usr/bin/mysqld_safe", "--defaults-file=/vtdataroot/tabletdata/my.cnf", "--basedir=/usr"}, Env:[]string{"LD_LIBRARY_PATH=/usr/lib/mysql"}, Dir:"/usr", Stdin:io.Reader(nil), Stdout:io.Writer(nil), Stderr:io.Writer(nil), ExtraFiles:[]*os.File(nil), SysProcAttr:(*syscall.SysProcAttr)(nil), Process:(*os.Process)(nil), ProcessState:(*os.ProcessState)(nil), ctx:context.Context(nil), lookPathErr:error(nil), finished:false, childFiles:[]*os.File(nil), closeAfterStart:[]io.Closer(nil), closeAfterWait:[]io.Closer(nil), goroutine:[]func() error(nil), errch:(chan error)(nil), waitDone:(chan struct {})(nil)}
I0225 07:22:01.240157       1 mysqld.go:447] Waiting for mysqld socket file (/vtdataroot/tabletdata/mysql.sock) to be ready...
I0225 07:22:01.309388       1 mysqld.go:398] Mysqld.Start(1582615321) stdout: 200225 07:22:01 mysqld_safe Logging to '/vtdataroot/tabletdata/error.log'.
I0225 07:22:01.327824       1 mysqld.go:398] Mysqld.Start(1582615321) stdout: 200225 07:22:01 mysqld_safe Starting mysqld daemon with databases from /vtdataroot/tabletdata/data
W0225 07:29:46.269597       1 server.go:696] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams failed to receive the preface from client: EOF"

[boopathykpm]

While deleting the helm chart, the dependency resources are getting delete, this halts the pod getting terminated and the pod get hung while terminating

nt.coda on 100.64.0.10:53: no such host". Reconnecting...
W0226 13:22:14.259960       1 clientconn.go:1191] grpc: addrConn.createTransport failed to connect to {etcd-global-client.coda:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup etcd-global-client.coda on 100.64.0.10:53: no such host". Reconnecting...
W0226 13:22:21.923088       1 clientconn.go:1191] grpc: addrConn.createTransport failed to connect to {etcd-global-client.coda:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup etcd-global-client.coda on 100.64.0.10:53: no such host". Reconnecting...
W0226 13:23:19.569186       1 clientconn.go:1191] grpc: addrConn.createTransport failed to connect to {etcd-global-client.coda:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup etcd-global-client.coda on 100.64.0.10:53: no such host". Reconnecting...
W0226 13:23:23.175314       1 clientconn.go:1191] grpc: addrConn.createTransport failed to connect to {etcd-global-client.coda:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup etcd-global-client.coda on 100.64.0.10:53: no such host". Reconnecting...
W0226 13:24:53.973454       1 clientconn.go:1191] grpc: addrConn.createTransport failed to connect to {etcd-global-client.coda:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup etcd-global-client.coda on 100.64.0.10:53: no such host". Reconnecting...
W0226 13:25:19.622595       1 clientconn.go:1191] grpc: addrConn.createTransport failed to connect to {etcd-global-client.coda:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup etcd-global-client.coda on 100.64.0.10:53: no such host". Reconnecting...

If there is a graceful termination applied while deleting the helm chart hope this will not happen.

[gedgar]

I'm closing this issue as the Helm charts were deprecated in Vitess 7.0. Thanks!