feat: add pubsub framework #7028

bzp2010 · 2022-05-11T10:01:53Z

Description

Support for publish-subscribe scenarios implemented in the form of websocket + protobuf, this PR contains its basic framework implementation.

Split from #6995

Checklist

I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

bzp2010 · 2022-05-11T12:18:14Z

Update

The pubsub module core and documentation was split to the current PR, and the review comments in #6995 have been modified.

bzp2010 · 2022-05-11T18:04:49Z

Update

The current pubsub module switches to use the pubsub module-level independent pb_state database, just like the pb_state cached in lrucache in grpc-transcode, which will switch to the cached pb_state before each decoding command.

But what I'm not sure is if in the extreme case grpc-transcode and pubsub switch pb_state concurrently at the same time will cause confusion. 🤔

spacewander · 2022-05-12T02:15:29Z

Update

The current pubsub module switches to use the pubsub module-level independent pb_state database, just like the pb_state cached in lrucache in grpc-transcode, which will switch to the cached pb_state before each decoding command.

But what I'm not sure is if in the extreme case grpc-transcode and pubsub switch pb_state concurrently at the same time will cause confusion. 🤔

There won't be concurrent issue as pubsub only modify the global pb_state when the APISIX starts.

apisix/core/pubsub.lua

docs/en/latest/pubsub.md

tzssangglass · 2022-05-12T03:05:15Z

docs/en/latest/pubsub.md

+
+## How to support other messaging systems
+
+An extensible pubsub module is implemented in Apache APISIX, which is responsible for starting the WebSocket server, coding and decoding communication protocols, handling client commands, and through which new messaging system support can be simply added.


Suggested change

An extensible pubsub module is implemented in Apache APISIX, which is responsible for starting the WebSocket server, coding and decoding communication protocols, handling client commands, and through which new messaging system support can be simply added.

Apache APISIX implemented an extensible pubsub module responsible for starting the WebSocket server, coding and decoding communication protocols, handling client commands, and adding support for the new messaging system.

docs/en/latest/pubsub.md

docs/zh/latest/pubsub.md

t/pubsub/pubsub.t

spacewander · 2022-05-12T09:16:59Z

.github/workflows/build.yml

@@ -31,7 +31,7 @@ jobs:
          - linux_openresty_1_17
        test_dir:
          - t/plugin
-          - t/admin t/cli t/config-center-yaml t/control t/core t/debug t/discovery t/error_page t/misc
+          - t/admin t/cli t/config-center-yaml t/control t/core t/debug t/discovery t/error_page t/misc t/pubsub
          - t/node t/router t/script t/stream-node t/utils t/wasm t/xds-library t/xrpc


plugin < pubsub < node, so we should put it here

fixed #7043 , According to #6995 (comment), I have alphabetically aligned it between node and router

spacewander · 2022-05-12T09:19:32Z

apisix/core/pubsub.lua

+-- no error exists.
+--
+-- @function core.pubsub.on
+-- @tparam string command to add callback


The syntax is @tparam type name desc, see

apisix/apisix/core/json.lua

Line 110 in 6ddca10

-- @tparam string|table data The data to be encoded.

continue adjust #7043

spacewander · 2022-05-12T09:20:15Z

apisix/core/pubsub.lua

+        -- handle client close connection
+        if raw_type == "close" then
+            ws:send_close()
+            return


Better to use break in the while loop and handle the common logic in the same place

This is not an error, there is no return value (in fact the return value of wait has also been removed, which does not cause confusion in the location of the error handling code), and if the client initiates a close connection, the server will also close the connection and exit the loop directly, without logging and subsequent processing. The other cases have been logged and changed to post-processing.

We can use break here and check fatal_err? It is smelly to use both break and return in the loop,
This will cause error-prone control flow,

spacewander · 2022-05-12T09:21:59Z

apisix/core/pubsub.lua

+        if err then
+            -- terminate the event loop when a fatal error occurs
+            if ws.fatal then
+                ws:send_close()


Should we always close connection outside the loop?

spacewander · 2022-05-12T09:25:18Z

apisix/core/pubsub.lua

+
+                local resp, err = handler(value)
+                if not resp then
+                    send_error_resp(ws, sequence, err)


Would be better to handle err in the various send_xxx operation

fixed #7043

spacewander · 2022-05-12T09:27:38Z

t/pubsub/pubsub.t

+
+            for _, data in ipairs(data) do
+                local code, body = t(data.url, ngx.HTTP_PUT, data.data)
+                ngx.say(code..body)


Better not directly print the 201 status code, because when we rerun the test, code 200 will return instead.

tzssangglass · 2022-05-12T11:44:59Z

Other is good for me, CI is broken.

membphis · 2022-05-12T12:15:32Z

apisix/core/pubsub.lua

+
+
+local function send_error_resp(ws, sequence, err_msg)
+    ws:send_binary(pb.encode("PubSubResp", {


we need to capture the return value, all right?

pb.encode and ws:send_binary, they may fail.

fixed, according to the practice in grpc-transcode, pb.encode does error handling via pcall, and logs are printed for errors in both cases

apisix/core/pubsub.lua

bzp2010 added 7 commits May 11, 2022 17:31

feat: add pubsub proto define

27a0f2f

feat: add pubsub core module

6ab23d6

docs: add tparam comment

c728ca2

chore: pubsub in-situ error handling

d8e95f0

fix: review

b623413

chore: install pubsub proto

06d417e

docs: add

285dcc2

bzp2010 added enhancement New feature or request doc Documentation things labels May 11, 2022

bzp2010 requested review from spacewander, membphis, shuaijinchao and tzssangglass May 11, 2022 10:01

bzp2010 self-assigned this May 11, 2022

bzp2010 added 3 commits May 11, 2022 19:34

test: add pubsub cases

2881b97

fix: lint

bedb6f7

test: add pubsub

ee1c552

bzp2010 added 3 commits May 12, 2022 01:48

feat: ensure use instance level unique pb_state

ddcbe5f

feat: ensure use module level pb_state

c921ff1

fix: pubsub test lib pb_state cache

3015c34

tzssangglass reviewed May 12, 2022

View reviewed changes

bzp2010 added 2 commits May 12, 2022 11:26

fix: review

3d734f2

fix: lint

36c9039

bzp2010 requested a review from tzssangglass May 12, 2022 03:32

fix: zh doc

d5ee31d

membphis reviewed May 12, 2022

View reviewed changes

docs/zh/latest/pubsub.md Outdated Show resolved Hide resolved

docs/zh/latest/pubsub.md Outdated Show resolved Hide resolved

t/pubsub/pubsub.t Outdated Show resolved Hide resolved

docs: remove chinese docs

04c1be0

bzp2010 added 5 commits May 12, 2022 14:45

test: add text ws send helper

bb09cc7

test: add more cases

2fe4f76

fix: typo

1f329ba

docs: remove supported kafka

80dfd9c

test: add undecodable case

bd0dc52

bzp2010 requested a review from membphis May 12, 2022 07:08

spacewander reviewed May 12, 2022

View reviewed changes

bzp2010 added 2 commits May 12, 2022 18:28

chore: delete kafka of pubsub proto

1091934

docs: delete kafka example

abe114d

bzp2010 added 4 commits May 12, 2022 20:01

feat: add ping/pong and empty command

132a664

test: change to ping and empty test cases

c9a026b

fix: code..body resp

0d86e27

chore: adjust control flow to avoid errors

2986572

membphis reviewed May 12, 2022

View reviewed changes

fix: ldoc comment

096773d

membphis reviewed May 12, 2022

View reviewed changes

apisix/core/pubsub.lua Show resolved Hide resolved

membphis reviewed May 12, 2022

View reviewed changes

apisix/core/pubsub.lua Show resolved Hide resolved

bzp2010 added 3 commits May 12, 2022 20:37

fix: lint

1b914f9

fix: pb.encode and ws.send errors not handled

874905f

fix: typo error

646b3b8

tzssangglass approved these changes May 12, 2022

View reviewed changes

bzp2010 requested review from membphis and spacewander May 12, 2022 14:03

membphis approved these changes May 12, 2022

View reviewed changes

spacewander approved these changes May 12, 2022

View reviewed changes

spacewander merged commit 4690feb into apache:master May 12, 2022

bzp2010 mentioned this pull request May 13, 2022

feat: support kafka consumer for pubsub scenario #6995

Closed

5 tasks

Liu-Junlin pushed a commit to Liu-Junlin/apisix that referenced this pull request May 20, 2022

feat: add pubsub framework (apache#7028)

93184fa

spacewander pushed a commit that referenced this pull request Jun 30, 2022

feat: add pubsub framework (#7028)

4c2dcbb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add pubsub framework #7028

feat: add pubsub framework #7028

bzp2010 commented May 11, 2022 •

edited

Loading

bzp2010 commented May 11, 2022

bzp2010 commented May 11, 2022

spacewander commented May 12, 2022

Update

tzssangglass May 12, 2022

bzp2010 May 12, 2022

spacewander May 12, 2022

bzp2010 May 13, 2022

spacewander May 12, 2022

bzp2010 May 12, 2022

bzp2010 May 13, 2022

spacewander May 12, 2022 •

edited

Loading

bzp2010 May 12, 2022

spacewander May 12, 2022

bzp2010 May 12, 2022

spacewander May 12, 2022 •

edited

Loading

bzp2010 May 12, 2022

spacewander May 12, 2022

bzp2010 May 13, 2022

spacewander May 12, 2022

bzp2010 May 12, 2022

tzssangglass commented May 12, 2022

membphis May 12, 2022

bzp2010 May 12, 2022


		## How to support other messaging systems

		An extensible pubsub module is implemented in Apache APISIX, which is responsible for starting the WebSocket server, coding and decoding communication protocols, handling client commands, and through which new messaging system support can be simply added.

	An extensible pubsub module is implemented in Apache APISIX, which is responsible for starting the WebSocket server, coding and decoding communication protocols, handling client commands, and through which new messaging system support can be simply added.
	Apache APISIX implemented an extensible pubsub module responsible for starting the WebSocket server, coding and decoding communication protocols, handling client commands, and adding support for the new messaging system.



		local function send_error_resp(ws, sequence, err_msg)
		ws:send_binary(pb.encode("PubSubResp", {

feat: add pubsub framework #7028

feat: add pubsub framework #7028

Conversation

bzp2010 commented May 11, 2022 • edited Loading

Description

Checklist

bzp2010 commented May 11, 2022

Update

bzp2010 commented May 11, 2022

Update

spacewander commented May 12, 2022

Update

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spacewander May 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spacewander May 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tzssangglass commented May 12, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bzp2010 commented May 11, 2022 •

edited

Loading

spacewander May 12, 2022 •

edited

Loading

spacewander May 12, 2022 •

edited

Loading