-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: keep healthcheck target state when upstream changes #10312
fix: keep healthcheck target state when upstream changes #10312
Conversation
conf/config-default.yaml
Outdated
@@ -145,7 +145,7 @@ nginx_config: # Config for render the template to generate n | |||
# effective if the master process runs with super-user privileges. | |||
error_log: logs/error.log # Location of the error log. | |||
error_log_level: warn # Logging level: info, debug, notice, warn, error, crit, alert, or emerg. | |||
worker_processes: auto # Automatically determine the optimal number of worker processes based | |||
worker_processes: 4 # Automatically determine the optimal number of worker processes based |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do you change this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
Do I need to add/remove nodes from upstream frequently? |
Yeah, but keep setting the same upstream is ok |
@@ -114,7 +114,7 @@ function _M.fire_all_clean_handlers(item) | |||
clean_handler.f(item) | |||
end | |||
|
|||
item.clean_handlers = nil | |||
item.clean_handlers = {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if the clean_handler
is put into the parents (routers), it will cause error like this:
2023/10/10 20:53:16 [error] 1057074#1057074: *2020 lua entry thread aborted: runtime error: /home/liuwei/api7/apisix/apisix/core/config_util.lua:61: attempt to index field 'clean_handlers' (a nil value)
stack traceback:
coroutine 0:
/home/liuwei/api7/apisix/apisix/core/config_util.lua: in function 'add_clean_handler'
/home/liuwei/api7/apisix/apisix/upstream.lua:152: in function 'fetch_healthchecker'
/home/liuwei/api7/apisix/apisix/upstream.lua:342: in function 'set_upstream'
/home/liuwei/api7/apisix/apisix/init.lua:549: in function 'handle_upstream'
/home/liuwei/api7/apisix/apisix/init.lua:730: in function 'http_access_phase'
@monkeyDluffy6017 I get this error when I run the wrk command that you shared: |
@shreemaan-abhishek You need to use wrk2 or remove the |
I got the following error, when I ran the commands you mentioned:
I hope this is the same expected error. |
@shreemaan-abhishek I don't know how you get this error, it has nothing to do with my changes. |
Signed-off-by: Ashish Tiwari <ashishjaitiwari15112000@gmail.com>
* upstream/master: (83 commits) fix: make install failed on mac (apache#10403) feat(zipkin): add variable (apache#10361) test(clickhouse-logger): to show that different endpoints will be chosen randomly (apache#8777) chore(deps): bump actions/setup-node from 3.8.1 to 4.0.0 (apache#10381) ci: fix the grpc test error (apache#10388) ci: trigger ci when doc-lint.yml changes (apache#10382) docs: fix usage of incorrect default admin api port (apache#10391) feat: Add authorization params to openid-connect plugin (apache#10058) feat: integrate authz-keycloak with secrets resource (apache#10353) fix(traffic-split): post_arg match fails because content-type contains charset (apache#10372) fix(consul): worker will not exit while reload or quit (apache#10342) chore: update rules for unresponded issues (apache#10354) docs: Update APISIX usecases in README (apache#10358) test: use http2 to test limit-req plugin (apache#10334) test: use http2 to test limit-conn plugin (apache#10332) chore: remove stream_proxy.only in config-default.yaml (apache#10337) docs: update underscore to hyphen in HTTP headers in `response-rewrite` plugin (apache#10347) fix: typos in comments (apache#10330) feat: support config stream_route upstream in service (apache#10298) fix: keep healthcheck target state when upstream changes (apache#10312) ...
Description
When we modify the upstream (adding or deleting nodes), APISIX first deletes the health check object and then recreates the health check object. The process of deletion and recreation may affect other parallel requests searching for healthy nodes, leading to the following error:
So we will replace
checker:clear()
with functionchecker:delayed_clear()
. This function marks all targets to be removed, but do not actually remove them. If before the delay parameter any of them is re-added, it is unmarked for removal.This function makes it possible to keep target state during config changes, where the targets might be removed and then re-added.
It's very hard to add test cases. I tried but failed.
Follow these steps to reproduce:
Fixes # (issue)
Checklist