bug: apisix with kubernetes discovery will fail after token file expires

[jaysonsantos]

Current Behavior

apisix uses a singleton to load the service account file and kubernetes rotates roughly every 90 days and after that time, the discovery will fail to get new pods with Unauthorized returned from kubernetes' API leading to stale pods in memory and nginx making calls to pods that do not exist anymore (in case deployments were rolled out)

Expected Behavior

apisix should re-read the token file every X days

Error Logs

apisix-57c57fd48b-hqzq9 apisix 2024/11/22 13:34:23 [error] 57#57: *509002587 [lua] informer_factory.lua:295: list failed, kind: Endpoints, reason: Unauthorized, message : {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
apisix-57c57fd48b-xgcjv apisix 2024/11/22 13:34:24 [error] 57#57: *508864131 [lua] informer_factory.lua:295: list failed, kind: Endpoints, reason: Unauthorized, message : {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
apisix-57c57fd48b-zgp7b apisix 2024/11/22 13:34:25 [error] 56#56: *508946548 [lua] informer_factory.lua:295: list failed, kind: Endpoints, reason: Unauthorized, message : {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

Steps to Reproduce

If there is a way to rotate the service account, this would reproduce it but i am not sure it is possible

Environment

• APISIX version (run apisix version):
  /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua version
  3.5.0
• Operating system (run uname -a):
  Linux apisix-7bd7684cdf-2k524 5.10.220-209.869.amzn2.x86_64 change: added doc of how to load plugin. #1 SMP Wed Jul 17 15:10:20 UTC 2024 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V):

nginx version: openresty/1.21.4.2
built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
built with OpenSSL 1.1.1s  1 Nov 2022
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.2.0 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PA
NIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.2 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2
 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.25 --add-module=../ngx_lua_upstream-0.07 --add-modu
le=../headers-more-nginx-module-0.34 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.13 --with-ld-opt='-Wl,-rpa
th,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr
/local/openresty/openssl111/lib' --add-module=/tmp/tmp.0EeoYgSz2t/openresty-1.21.4.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.0EeoYgSz2t/openresty-1.21.4.2/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.0EeoYgSz2t/openresty-1.21.4.2/../apisix-nginx-modu
le-1.14.0 --add-module=/tmp/tmp.0EeoYgSz2t/openresty-1.21.4.2/../apisix-nginx-module-1.14.0/src/stream --add-module=/tmp/tmp.0EeoYgSz2t/openresty-1.21.4.2/../apisix-nginx-module-1.14.0/src/meta --add-module=/tmp/tmp.0EeoYgSz2t/openresty-1.21.4.2/../wasm-nginx-mod
ule-0.6.5 --add-module=/tmp/tmp.0EeoYgSz2t/openresty-1.21.4.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.0EeoYgSz2t/openresty-1.21.4.2/../grpc-client-nginx-module-v0.4.3 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-st
ream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_
link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module