Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add new lua-resty-events as events implementation #10550

Merged
merged 46 commits into from
Dec 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
7aea90c
feat: move to apisix.events
bzp2010 Dec 4, 2023
692feb8
test env
bzp2010 Dec 4, 2023
a1d9c28
update ngx template
bzp2010 Dec 4, 2023
11d35ed
fix
bzp2010 Dec 4, 2023
bbb2fc1
fix ci env
bzp2010 Dec 5, 2023
df17f01
mock event list implementation
bzp2010 Dec 5, 2023
ecee33b
test
bzp2010 Dec 5, 2023
dcdb282
test patch
bzp2010 Dec 6, 2023
3ca211f
fix call
bzp2010 Dec 6, 2023
0ee5494
revert global rule
bzp2010 Dec 6, 2023
06701ea
revert fault injection
bzp2010 Dec 6, 2023
827ff8c
fix rh based test
bzp2010 Dec 6, 2023
14c0117
clean
bzp2010 Dec 6, 2023
10999fa
try fix multiple worker health check
bzp2010 Dec 6, 2023
76dc27b
fix mismatched debug log
bzp2010 Dec 6, 2023
1f68994
try force resty events
bzp2010 Dec 6, 2023
2cc1fbc
remove no error log on healthcheck discovery
bzp2010 Dec 6, 2023
ee50b1a
fix healthcheck case
bzp2010 Dec 6, 2023
bd8af2f
fix lint
bzp2010 Dec 6, 2023
7cf568b
fix cli test
bzp2010 Dec 6, 2023
b7549bc
fix lint
bzp2010 Dec 6, 2023
9e2e58f
fix lint
bzp2010 Dec 6, 2023
6ed090a
fix lint
bzp2010 Dec 6, 2023
1f3e632
fix lint
bzp2010 Dec 6, 2023
db2da3a
fix lint
bzp2010 Dec 6, 2023
af3b366
ensure consul finish health check
bzp2010 Dec 6, 2023
6ffc4ab
test multiple events module in ubuntu
bzp2010 Dec 6, 2023
8a93fed
fix
bzp2010 Dec 6, 2023
4c9b9b9
test only lua-resty-worker-events in centos and redhat
bzp2010 Dec 6, 2023
7645c38
clean
bzp2010 Dec 6, 2023
4e9725f
fix healthcheck passive
bzp2010 Dec 7, 2023
a4141d8
fix lint
bzp2010 Dec 7, 2023
4a1424b
recover original passive health check
bzp2010 Dec 7, 2023
0e97a95
add new passive healthcheck case for lua-resty-events
bzp2010 Dec 7, 2023
b215d35
fix typo
bzp2010 Dec 7, 2023
ec6ffc7
increase wait time
bzp2010 Dec 7, 2023
87ede50
rollback apisix runtime build script
bzp2010 Dec 8, 2023
b772ff3
move to apisix-build-tools master
bzp2010 Dec 8, 2023
8955d39
test new lib on centos and ubi
bzp2010 Dec 8, 2023
fa97305
clean
bzp2010 Dec 8, 2023
ac50b44
clean
bzp2010 Dec 8, 2023
a3cdf11
test clean
bzp2010 Dec 8, 2023
7119cb1
add default events module for test
bzp2010 Dec 8, 2023
2c20df4
test old lib in legacy os
bzp2010 Dec 11, 2023
8b4282e
fix healthcheck
bzp2010 Dec 12, 2023
b45bb9b
fix healthcheck
bzp2010 Dec 12, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ jobs:
- ubuntu-20.04
os_name:
- linux_openresty
events_module:
- lua-resty-worker-events
- lua-resty-events
test_dir:
- t/plugin/[a-k]*
- t/plugin/[l-z]*
Expand Down Expand Up @@ -169,6 +172,7 @@ jobs:
- name: Linux Script
env:
TEST_FILE_SUB_DIR: ${{ matrix.test_dir }}
TEST_EVENTS_MODULE: ${{ matrix.events_module }}
run: sudo -E ./ci/${{ matrix.os_name }}_runner.sh script

- if: ${{ steps.cache-images.outputs.cache-hit != 'true' }}
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/centos7-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ jobs:
strategy:
fail-fast: false
matrix:
events_module:
- lua-resty-worker-events
- lua-resty-events
test_dir:
- t/plugin/[a-k]*
- t/plugin/[l-z]*
Expand Down Expand Up @@ -111,8 +114,9 @@ jobs:
- name: Run centos7 docker and mapping apisix into container
env:
TEST_FILE_SUB_DIR: ${{ matrix.test_dir }}
TEST_EVENTS_MODULE: ${{ matrix.events_module }}
run: |
docker run -itd -v ${{ github.workspace }}:/apisix --env TEST_FILE_SUB_DIR="$TEST_FILE_SUB_DIR" --name centos7Instance --net="host" --dns 8.8.8.8 --dns-search apache.org docker.io/centos:7 /bin/bash
docker run -itd -v ${{ github.workspace }}:/apisix --env TEST_FILE_SUB_DIR="$TEST_FILE_SUB_DIR" --env TEST_EVENTS_MODULE="$TEST_EVENTS_MODULE" --name centos7Instance --net="host" --dns 8.8.8.8 --dns-search apache.org docker.io/centos:7 /bin/bash
# docker exec centos7Instance bash -c "cp -r /tmp/apisix ./"

- name: Cache images
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/redhat-ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ jobs:
strategy:
fail-fast: false
matrix:
events_module:
- lua-resty-worker-events
- lua-resty-events
test_dir:
- t/plugin/[a-k]*
- t/plugin/[l-z]*
Expand Down Expand Up @@ -107,8 +110,9 @@ jobs:
- name: Run redhat docker and mapping apisix into container
env:
TEST_FILE_SUB_DIR: ${{ matrix.test_dir }}
TEST_EVENTS_MODULE: ${{ matrix.events_module }}
run: |
docker run -itd -v ${{ github.workspace }}:/apisix --env TEST_FILE_SUB_DIR="$TEST_FILE_SUB_DIR" --name ubiInstance --net="host" --dns 8.8.8.8 --dns-search apache.org registry.access.redhat.com/ubi8/ubi:8.6 /bin/bash
docker run -itd -v ${{ github.workspace }}:/apisix --env TEST_FILE_SUB_DIR="$TEST_FILE_SUB_DIR" --env TEST_EVENTS_MODULE="$TEST_EVENTS_MODULE" --name ubiInstance --net="host" --dns 8.8.8.8 --dns-search apache.org registry.access.redhat.com/ubi8/ubi:8.6 /bin/bash

- name: Cache images
id: cache-images
Expand Down
24 changes: 24 additions & 0 deletions apisix/cli/ngx_tpl.lua
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,17 @@ stream {
apisix.stream_init_worker()
}

{% if (events.module or "") == "lua-resty-events" then %}
# the server block for lua-resty-events
server {
listen unix:{*apisix_lua_home*}/logs/stream_worker_events.sock;
access_log off;
content_by_lua_block {
require("resty.events.compat").run()
}
}
{% end %}

server {
{% for _, item in ipairs(stream_proxy.tcp or {}) do %}
listen {*item.addr*} {% if item.tls then %} ssl {% end %} {% if enable_reuseport then %} reuseport {% end %} {% if proxy_protocol and proxy_protocol.enable_tcp_pp then %} proxy_protocol {% end %};
Expand Down Expand Up @@ -483,6 +494,19 @@ http {
apisix.http_exit_worker()
}

{% if (events.module or "") == "lua-resty-events" then %}
# the server block for lua-resty-events
server {
listen unix:{*apisix_lua_home*}/logs/worker_events.sock;
access_log off;
location / {
content_by_lua_block {
require("resty.events.compat").run()
}
}
}
{% end %}

{% if enable_control then %}
server {
listen {* control_server_addr *};
Expand Down
93 changes: 81 additions & 12 deletions apisix/events.lua
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,27 @@
-- limitations under the License.
--

local require = require
local error = error
local ngx = ngx

local _M = {}
local require = require
local error = error
local assert = assert
local tostring = tostring
local pairs = pairs
local setmetatable = setmetatable
local ngx = ngx
local core = require("apisix.core")

local _M = {
events_module = nil,
}

_M.EVENTS_MODULE_LUA_RESTY_WORKER_EVENTS = 'lua-resty-worker-events'
_M.EVENTS_MODULE_LUA_RESTY_EVENTS = 'lua-resty-events'


-- use lua-resty-worker-events
local function init_worker_events()
local function init_resty_worker_events()
_M.events_module = _M.EVENTS_MODULE_LUA_RESTY_WORKER_EVENTS

local we = require("resty.worker.events")
local shm = ngx.config.subsystem == "http" and "worker-events" or "worker-events-stream"
local ok, err = we.configure({shm = shm, interval = 0.1})
Expand All @@ -38,17 +47,53 @@ local function init_worker_events()
end


-- use lua-resty-events
local function init_resty_events()
_M.events_module = _M.EVENTS_MODULE_LUA_RESTY_EVENTS

local listening = "unix:" .. ngx.config.prefix() .. "logs/"
if ngx.config.subsystem == "http" then
listening = listening .. "worker_events.sock"
else
listening = listening .. "stream_worker_events.sock"
end
core.log.info("subsystem: " .. ngx.config.subsystem .. " listening sock: " .. listening)

local opts = {
unique_timeout = 5, -- life time of unique event data in lrucache
broker_id = 0, -- broker server runs in nginx worker #0
listening = listening, -- unix socket for broker listening
}

local we = require("resty.events.compat")
assert(we.configure(opts))
assert(we.configured())

return we
end


function _M.init_worker()
if _M.inited then
-- Prevent duplicate initializations in the same worker to
-- avoid potentially unanticipated behavior
-- prevent duplicate initializations in the same worker to
-- avoid potentially unexpected behavior
return
end

_M.inited = true

-- use lua-resty-worker-events default now
_M.worker_events = init_worker_events()
local conf = core.config.local_conf()
local module_name = core.table.try_read_attr(conf, "apisix", "events", "module")
or _M.EVENTS_MODULE_LUA_RESTY_WORKER_EVENTS

if module_name == _M.EVENTS_MODULE_LUA_RESTY_EVENTS then
-- use lua-resty-events as an event module via the apisix.events.module
-- key in the configuration file
_M.worker_events = init_resty_events()
else
-- use lua-resty-worker-events default now
_M.worker_events = init_resty_worker_events()
end
end


Expand All @@ -57,8 +102,23 @@ function _M.register(self, ...)
end


function _M.event_list(self, ...)
return self.worker_events.event_list(...)
function _M.event_list(self, source, ...)
-- a patch for the lua-resty-events to support event_list
-- this snippet is copied from the lua-resty-worker-events lib
if self.events_module == _M.EVENTS_MODULE_LUA_RESTY_EVENTS then
local events = { _source = source }
for _, event in pairs({...}) do
events[event] = event
end
return setmetatable(events, {
__index = function(_, key)
error("event '"..tostring(key).."' is an unknown event", 2)
end
})
end

-- the lua-resty-worker-events has a built-in event_list implementation
return self.worker_events.event_list(source, ...)
end


Expand All @@ -67,4 +127,13 @@ function _M.post(self, ...)
end


function _M.get_healthcheck_events_modele(self)
if self.events_module == _M.EVENTS_MODULE_LUA_RESTY_EVENTS then
return "resty.events"
else
return "resty.worker.events"
end
end


return _M
9 changes: 9 additions & 0 deletions apisix/upstream.lua
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ local core = require("apisix.core")
local discovery = require("apisix.discovery.init").discovery
local upstream_util = require("apisix.utils.upstream")
local apisix_ssl = require("apisix.ssl")
local events = require("apisix.events")
local error = error
local tostring = tostring
local ipairs = ipairs
Expand Down Expand Up @@ -110,10 +111,18 @@ local function create_checker(upstream)
end
upstream.is_creating_checker = true

core.log.debug("events module used by the healthcheck: ", events.events_module,
", module name: ",events:get_healthcheck_events_modele())

local checker, err = healthcheck.new({
name = get_healthchecker_name(healthcheck_parent),
shm_name = "upstream-healthcheck",
checks = upstream.checks,
-- the events.init_worker will be executed in the init_worker phase,
-- events.healthcheck_events_module is set
-- while the healthcheck object is executed in the http access phase,
-- so it can be used here
events_module = events:get_healthcheck_events_modele(),
})

if not checker then
Expand Down
13 changes: 10 additions & 3 deletions ci/centos7-ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,19 @@ install_dependencies() {

# install openresty to make apisix's rpm test work
yum install -y yum-utils && yum-config-manager --add-repo https://openresty.org/package/centos/openresty.repo
wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime-debug-centos7.sh"
wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime.sh"

# TODO: disabled temporarily, waiting for APISIX 3.8 to be released to synchronize the apisix-runtime version
#wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime-debug-centos7.sh"
#wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime.sh"
wget "https://raw.githubusercontent.com/api7/apisix-build-tools/master/build-apisix-runtime-debug-centos7.sh"
wget "https://raw.githubusercontent.com/api7/apisix-build-tools/master/build-apisix-runtime.sh"
chmod +x build-apisix-runtime-debug-centos7.sh
chmod +x build-apisix-runtime.sh
./build-apisix-runtime-debug-centos7.sh

# patch lua-resty-events
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why update this ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is impossible to execute a reload without a transient interruption because the worker that was handling the unix socket connection is being restarted, and the other workers will not be able to connect to the broker over the unix socket again until the new worker that replaces it is up and running, so the logs will be some error level logs about connection closed, which will interrupt our tests, so we'll have to change the code here to warn level.

There is no need to apply this patch in production as it has no impact other than causing some error logs, when the worker is available, the worker and broker will reconnect, and event push will be available again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok.

sed -i 's/log(ERR, "event worker failed: ", perr)/log(ngx.WARN, "event worker failed: ", perr)/' /usr/local/openresty/lualib/resty/events/worker.lua

# install luarocks
./utils/linux-install-luarocks.sh

Expand Down Expand Up @@ -95,7 +102,7 @@ run_case() {
make init
set_coredns
# run test cases
FLUSH_ETCD=1 prove --timer -Itest-nginx/lib -I./ -r ${TEST_FILE_SUB_DIR} | tee /tmp/test.result
FLUSH_ETCD=1 TEST_EVENTS_MODULE=$TEST_EVENTS_MODULE prove --timer -Itest-nginx/lib -I./ -r ${TEST_FILE_SUB_DIR} | tee /tmp/test.result
rerun_flaky_tests /tmp/test.result
}

Expand Down
7 changes: 6 additions & 1 deletion ci/linux-install-openresty.sh
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,11 @@ if [ "$OPENRESTY_VERSION" == "source" ]; then
fi
fi

wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime.sh"
# TODO: disabled temporarily, waiting for APISIX 3.8 to be released to synchronize the apisix-runtime version
#wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime.sh"
wget "https://raw.githubusercontent.com/api7/apisix-build-tools/master/build-apisix-runtime.sh"
chmod +x build-apisix-runtime.sh
./build-apisix-runtime.sh latest

# patch lua-resty-events
sudo sed -i 's/log(ERR, "event worker failed: ", perr)/log(ngx.WARN, "event worker failed: ", perr)/' /usr/local/openresty/lualib/resty/events/worker.lua
2 changes: 1 addition & 1 deletion ci/linux_openresty_common_runner.sh
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ script() {
start_grpc_server_example

# APISIX_ENABLE_LUACOV=1 PERL5LIB=.:$PERL5LIB prove -Itest-nginx/lib -r t
FLUSH_ETCD=1 prove --timer -Itest-nginx/lib -I./ -r $TEST_FILE_SUB_DIR | tee /tmp/test.result
FLUSH_ETCD=1 TEST_EVENTS_MODULE=$TEST_EVENTS_MODULE prove --timer -Itest-nginx/lib -I./ -r $TEST_FILE_SUB_DIR | tee /tmp/test.result
rerun_flaky_tests /tmp/test.result
}

Expand Down
12 changes: 9 additions & 3 deletions ci/redhat-ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,18 @@ install_dependencies() {
yum install -y openresty-openssl111 openresty-openssl111-devel pcre pcre pcre-devel xz
yum -y install https://repos.apiseven.com/packages/centos/apache-apisix-repo-1.0-1.noarch.rpm

wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime-debug-centos7.sh"
wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime.sh"
# TODO: disabled temporarily, waiting for APISIX 3.8 to be released to synchronize the apisix-runtime version
#wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime-debug-centos7.sh"
#wget "https://raw.githubusercontent.com/api7/apisix-build-tools/apisix-runtime/${APISIX_RUNTIME}/build-apisix-runtime.sh"
wget "https://raw.githubusercontent.com/api7/apisix-build-tools/master/build-apisix-runtime-debug-centos7.sh"
wget "https://raw.githubusercontent.com/api7/apisix-build-tools/master/build-apisix-runtime.sh"
chmod +x build-apisix-runtime.sh
chmod +x build-apisix-runtime-debug-centos7.sh
./build-apisix-runtime-debug-centos7.sh

# patch lua-resty-events
sed -i 's/log(ERR, "event worker failed: ", perr)/log(ngx.WARN, "event worker failed: ", perr)/' /usr/local/openresty/lualib/resty/events/worker.lua

# install luarocks
./utils/linux-install-luarocks.sh

Expand Down Expand Up @@ -94,7 +100,7 @@ run_case() {
make init
set_coredns
# run test cases
FLUSH_ETCD=1 prove --timer -Itest-nginx/lib -I./ -r ${TEST_FILE_SUB_DIR} | tee /tmp/test.result
FLUSH_ETCD=1 TEST_EVENTS_MODULE=$TEST_EVENTS_MODULE prove --timer -Itest-nginx/lib -I./ -r ${TEST_FILE_SUB_DIR} | tee /tmp/test.result
rerun_flaky_tests /tmp/test.result
}

Expand Down
4 changes: 4 additions & 0 deletions conf/config-default.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,10 @@ apisix:
# with the new key. Removing the old keys directly can render the data
# unrecoverable.

events: # Event distribution module configuration
module: lua-resty-events # Sets the name of the events module used.
# Supported module: lua-resty-worker-events and lua-resty-events

nginx_config: # Config for render the template to generate nginx.conf
# user: root # Set the execution user of the worker process. This is only
# effective if the master process runs with super-user privileges.
Expand Down
22 changes: 22 additions & 0 deletions t/APISIX.pm
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,12 @@ if ($custom_dns_server) {
}


my $events_module = $ENV{TEST_EVENTS_MODULE} // "lua-resty-events";
my $default_yaml_config = read_file("conf/config-default.yaml");
# enable example-plugin as some tests require it
$default_yaml_config =~ s/#- example-plugin/- example-plugin/;
$default_yaml_config =~ s/enable_export_server: true/enable_export_server: false/;
$default_yaml_config =~ s/module: lua-resty-events/module: $events_module/;

my $user_yaml_config = read_file("conf/config.yaml");
my $ssl_crt = read_file("t/certs/apisix.crt");
Expand Down Expand Up @@ -437,6 +439,14 @@ _EOC_

$extra_stream_config

server {
listen unix:$apisix_home/t/servroot/logs/stream_worker_events.sock;
access_log off;
content_by_lua_block {
require("resty.events.compat").run()
}
}

# fake server, only for test
server {
listen 1995;
Expand Down Expand Up @@ -687,6 +697,18 @@ _EOC_
}
}

_EOC_

$http_config .= <<_EOC_;
server {
listen unix:$apisix_home/t/servroot/logs/worker_events.sock;
access_log off;
location / {
content_by_lua_block {
require("resty.events.compat").run()
}
}
}
_EOC_

$block->set_value("http_config", $http_config);
Expand Down
Loading
Loading