bug: APISIX executes reload, but the old process does not exit for a long time

[hansedong]

Current Behavior

As APISIX executes reload, but the old process does not exit for a long time.

root      885278       1  0 Jul05 ?        00:00:00 nginx: master process /usr/bin/openresty -p /usr/local/apisix -c /usr/local/apisix/conf/nginx.conf
nobody    885280  885278  0 Jul05 ?        00:09:03 nginx: worker process is shutting down
nobody    885281  885278  0 Jul05 ?        00:08:39 nginx: worker process is shutting down
nobody    885282  885278  0 Jul05 ?        00:08:24 nginx: worker process is shutting down
nobody   1288295  885278  0 18:09 ?        00:00:03 nginx: worker process is shutting down
nobody   1288517  885278  5 18:10 ?        00:03:10 nginx: worker process
nobody   1288518  885278  4 18:10 ?        00:02:16 nginx: worker process
nobody   1288519  885278  4 18:10 ?        00:02:15 nginx: worker process
nobody   1288520  885278  4 18:10 ?        00:02:12 nginx: worker process
nobody   1288521  885278  0 18:10 ?        00:00:01 nginx: cache manager process
root     1288522  885278  3 18:10 ?        00:01:41 nginx: privileged agent process
root     1296427 1280594  0 19:04 pts/0    00:00:00 grep --color=auto nginx

From above, it can be seen that the creation time of the new process is 18:10, but it is already 19:05 now

[root@knode10-72-73-177 logs]# date
Fri Jul  7 19:05:05 CST 2023

The value of the configuration item "worker_shutdown_timeout" in the APISIX configuration file is set to 240 seconds.

nginx_config:                     # config for render the template to generate nginx.conf
  #user: root                     # specifies the execution user of the worker process.
                                  # the "user" directive makes sense only if the master process runs with super-user privileges.
                                  # if you're not root user,the default is current user.
  error_log: /DATA1/apisix/apisix/logs/error.log
  error_log_level:  warn          # warn,error
  worker_processes: 4             # if you want use multiple cores in container, you can inject the number of cpu as environment variable "APISIX_WORKER_PROCESSES"
  enable_cpu_affinity: false      # disable CPU affinity by default, if APISIX is deployed on a physical machine, it can be enabled and work well.
  worker_rlimit_nofile: 6553500     # the number of files a worker process can open, should be larger than worker_connections
  worker_shutdown_timeout: 240s   # timeout for a graceful shutdown of worker processes

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

systemctl reload apisix

Environment

• APISIX version (run apisix version): 3.4.0
• Operating system (run uname -a): Linux knode10-72-73-177 5.15.29-200.el7.x86_64 #1 SMP Thu Mar 31 14:09:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V):

nginx version: openresty/1.21.4.1
built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
built with OpenSSL 1.1.1s  1 Nov 2022
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.1.8 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.21 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.11 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.YzVafXtnkf/openresty-1.21.4.1/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.YzVafXtnkf/openresty-1.21.4.1/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.YzVafXtnkf/openresty-1.21.4.1/../apisix-nginx-module-1.12.0 --add-module=/tmp/tmp.YzVafXtnkf/openresty-1.21.4.1/../apisix-nginx-module-1.12.0/src/stream --add-module=/tmp/tmp.YzVafXtnkf/openresty-1.21.4.1/../apisix-nginx-module-1.12.0/src/meta --add-module=/tmp/tmp.YzVafXtnkf/openresty-1.21.4.1/../wasm-nginx-module-0.6.4 --add-module=/tmp/tmp.YzVafXtnkf/openresty-1.21.4.1/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.YzVafXtnkf/openresty-1.21.4.1/../grpc-client-nginx-module-v0.4.2 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module

• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.9
• APISIX Dashboard version, if relevant: 3.0.1
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

/usr/local/bin/luarocks 3.8.0
LuaRocks main command-line interface

[jiangfucheng]

I reproduced this bug. It seems cause by that the worker process not close connection with etcd in woker exit phase. Let me investigate how to fix it, it seems a bit complicated to me.