bug: using the latest image(3.0.0-centos) to upgrade helm release failed: container cannot connect to etcd

[xinxinh2020]

Current Behavior

When i upgraded the image of helm release from 2.15.0-centos(or 2.15.0-alpine) to the latest one, the latest container failed to run due to failing to connect to etcd cluster. The error message in container logs like:

/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
Warning! Request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused, retry time=1
Warning! Request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused, retry time=2
request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused
all etcd nodes are unavailable

I have recorded all the steps to reproduce this bug using asciinema, see: https://asciinema.org/a/4q10QLD1kJw3N5viGAR7JgYd3

Expected Behavior

Upgraded successfully.

Error Logs

/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
Warning! Request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused, retry time=1
Warning! Request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused, retry time=2
request etcd endpoint 'http://127.0.0.1:2379/version' error, connection refused
all etcd nodes are unavailable

Steps to Reproduce

https://asciinema.org/a/4q10QLD1kJw3N5viGAR7JgYd3

Environment

• APISIX version (run apisix version): 3.0.0
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[spacewander]

Could you provide the helm chart commit version which you modify from?

[spacewander]

BTW, we are developing the feature to use 3.0 image (which is not backward compatible) via a separate flag. So soon we don't need to manually modify the helm chart (which is not recommended).

[xinxinh2020]

Could you provide the helm chart commit version which you modify from?

I think it is the latest version in the https://charts.apiseven.com repo:

helm search repo apisix
NAME                            	CHART VERSION	APP VERSION	DESCRIPTION
apisix/apisix                   	0.11.1       	2.15.0     	A Helm chart for Apache APISIX

[xinxinh2020]

BTW, we are developing the feature to use 3.0 image (which is not backward compatible) via a separate flag. So soon we don't need to manually modify the helm chart (which is not recommended).

Is there a latest helm chart for 3.0 image ?
If not now, how can i manually modify the helm chart to upgrade to 3.0 image ?

[xinxinh2020]

BTW, we are developing the feature to use 3.0 image (which is not backward compatible) via a separate flag. So soon we don't need to manually modify the helm chart (which is not recommended).

Is there a latest helm chart for 3.0 image ? If not now, how can i manually modify the helm chart to upgrade to 3.0 image ?

I think i have figured out the problem: the configuration structure in 3.0.0 is changed - The etcd field is moved to deployment: #7860

I will try to manually modify the configmap for the config.yaml later

Hope the latest helm chart to be released soon and If there's anything I can do, I would be pleased to help.

[spacewander]

BTW, we are developing the feature to use 3.0 image (which is not backward compatible) via a separate flag.

We plan to carry out this feature this week.

[tzssangglass]

I will try to manually modify the configmap for the config.yaml later

wait for #390 merged, I'm working on this.