request help: Intermittent failure when starting apisix container on kubernetes.

[rdouman]

Issue description

I have a 10 node kubernetes cluster running APISIX as a gateway, We're using etcd 3.4.14 deployed as a cluster via the bitnami helm chart. We're intermittently receiving the errors below on container startup.

2021/04/14 14:09:03 [error] 42#42: 1 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["/ecom/"],"vars":{},"hosts":["dev.url","develop.url"],"id":"350417588708180914","update_time":1618401644,"status":1,"create_time":1618394758,"name":"ecom_dev_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.development.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","POST","PUT","DELETE"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 42#42: 1 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["/ecom/"],"vars":{},"hosts":["uat.url","stage.url"],"id":"350426062628324274","update_time":1618401371,"status":1,"create_time":1618399809,"name":"ecom_uat_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.uat.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","HEAD","POST","PUT","DELETE","OPTIONS","PATCH"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 41#41: 2 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["/ecom/"],"vars":{},"hosts":["dev.url","develop.url"],"id":"350417588708180914","update_time":1618401644,"status":1,"create_time":1618394758,"name":"ecom_dev_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.development.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","POST","PUT","DELETE"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 41#41: 2 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["/ecom/"],"vars":{},"hosts":["uat.url","stage.url"],"id":"350426062628324274","update_time":1618401371,"status":1,"create_time":1618399809,"name":"ecom_uat_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.uat.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","HEAD","POST","PUT","DELETE","OPTIONS","PATCH"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 45#45: 3 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["/ecom/"],"vars":{},"hosts":["dev.url","develop.url"],"id":"350417588708180914","update_time":1618401644,"status":1,"create_time":1618394758,"name":"ecom_dev_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.development.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","POST","PUT","DELETE"]}, context: init_worker_by_lua*
2021/04/14 14:09:03 [error] 45#45: 3 [lua] config_etcd.lua:230: load_full_data(): failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["/ecom/"],"vars":{},"hosts":["uat.url","stage.url"],"id":"350426062628324274","update_time":1618401371,"status":1,"create_time":1618399809,"name":"ecom_uat_v2","upstream":{"hash_on":"vars","scheme":"http","type":"roundrobin","timeout":{"read":6000,"connect":6000,"send":6000},"pass_host":"pass","nodes":[{"host":"ecom-edg.uat.svc.cluster.local","weight":1,"port":8085,"priority":0}]},"priority":0,"methods":["GET","HEAD","POST","PUT","DELETE","OPTIONS","PATCH"]}, context: init_worker_by_lua*

APISIX appears to be listening on the exposed port, but returns 404 route not found whenever trying to access one of the configured end points on containers with the above error in the logs.

Expected behavior

APIXSIX should start up with no errors in the error logs and successfully route traffic on the exposed routes.

How to Reproduce

1. Deploy APISIX gateway on kubernetes cluster. Scale the deployment and check the logs.

Environment

• apisix version (cmd: apisix version):
  apache/apisix:2.2-alpine

• OS (cmd: uname -a):
  /etc/os-release
  NAME="openSUSE MicroOS"

VERSION="20210408"

ID="opensuse-microos"
ID_LIKE="suse opensuse opensuse-tumbleweed"
VERSION_ID="20210408"
PRETTY_NAME="openSUSE MicroOS"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:microos:20210408"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:MicroOS"
LOGO="distributor-logo"

• OpenResty / Nginx version (cmd: nginx -V or openresty -V):

• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):
  3.4.14

• apisix-dashboard version, if have:
  apache/apisix-dashboard:2.3

[tokers]

What's your apisix version?

[Firstsawyou]

failed to check item data of [/apisix/routes] err:failed to validate the 'vars' expression: rule too short ,val: {"uris":["/ecom/"],"vars":{},

According to the prompt information here, there seems to be a problem with the vars configured in the route. Is the vars expression set in your route "vars":{}? Maybe you should provide detailed configuration information for routing.

[spacewander]

failed to validate the 'vars' expression: rule too short

This message is added in APISIX 2.5, and apisix-dashboard 2.3 is too old to work with APISIX 2.5

[souzens]

failed to validate the 'vars' expression: rule too short

This message is added in APISIX 2.5, and apisix-dashboard 2.3 is too old to work with APISIX 2.5

the same wrong ..

NOW etcd data is created by apisix-dashboard 2.4 . and After upgrading to apisix 2.5, the current problem is that the existing etcd data can no longer be supported, and all of the created routes has been invalid.

according to https://github.com/apache/apisix/blob/master/docs/zh/latest/admin-api.md

vars is still use list type , apisix etcd 2.5 report vars use dict type .

[spacewander]

the current problem is that the existing etcd data can no longer be supported, and all of the created routes has been invalid.

My bad. I underestimate the effect of the additional check.

vars is still use list type , apisix etcd 2.5 report vars use dict type .

No, the vars is still list type. The problem is that we add validation to reject empty table. We also update the dashboard but of course, the existing routes are broken now.

[spacewander]

Just submit a PR to fix it: #4056

[spacewander]

Can be solved by applied #4056 or wait for 2.6 release.

[spacewander]

BTW, none of the apisix-dashboard releases support APISIX 2.5 now. If you need to use the dashboard, please use APISIX 2.4

[rdouman]

Hi all

Thanks you for the quick response. I just want to point out that I'm not using APISIX 2.5. I'm using the docker image for APISIX /apache/apisix:2.2-alpine with the apache/apisix-dashboard:2.3 dashboard.

One thing that I've noticed is that the apache/apisix:2.2-alpine has been updated 10 days ago, so my question is has this image perhaps been erroneous updated ?

I've tracked down the intermittent nature of the issue:
Docker images that are working are apache/apisix:2.2-alpine (digest 186f0052e1191)
Docker images that are not working are apache/apisix:2.2-alpine (digest 22a167dfcf31e)