request help: kafka logger supports logging request body

[spacewander]

Issue description

Add a request_body switch to the schema, and each body can be used by expr to decide whether to log or not. Without this switch, the body is not logged.

"kafka-logger": {
   "broker_list":{
       "127.0.0.1":9092
    },
   "kafka_topic" : "test2",
   "request_body": {
       "expr": [
          ["request_length", "<", "1024"],
       ]
   },
   "key" : "key1",
   "batch_max_size": 1,
   "name": "kafka logger"
}

expr can be evaluated by lua-resty-expr. request body can be fetched by core.request.get_body.

Environment

• apisix version (cmd: apisix version): 2.10.1
• OS (cmd: uname -a):
• OpenResty / Nginx version (cmd: nginx -V or openresty -V):
• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):
• apisix-dashboard version, if have:
• the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
• luarocks version, if the issue is about installation (cmd: luarocks --version):

[windyrjc]

core.request.get_body and core.response.hold_body_chunk can't �be used in 'log_by_lua' phase,it seems that we can logging log it in other phase or set request_body and response_body in ctx. Does it have a better way?

[tokers]

core.request.get_body and core.response.hold_body_chunk can't �be used in 'log_by_lua' phase,it seems that we can logging log it in other phase or set request_body and response_body in ctx. Does it have a better way?

Do this in an early phase, say, access phase.

[windyrjc]

Other phase like filter_by_lua may rewrite request and response by other plugin. If we get request_body and response_body in filter_by_lua and set it in ctx,it may cause performance issue

[tzssangglass]

get request body in access_by_lua phase, and get response body in log_by_lua phase?

[spacewander]

it may cause performance issue

Yes. The log of body is expensive. So we should only log it if need.

Other phase like filter_by_lua may rewrite request and response by other plugin.

In practice, it is not a problem. The gzip filter runs after lua filter, so there is no way to log the gzipped response body in the Lua. However, very few people want to log the gzipped body instead of the original one. If people need a 100% real body sent to the client, they should do it in another proxy or a traffic capture sidecar.

[windyrjc]

In practice, it is not a problem. The gzip filter runs after lua filter

Other plugin like response_rewrite plugin will also change the response.So do you means I should change all the kafka-logger code to the filter_by_lua phase and set the kafka-logger priority after all the other plugin except gzip ?

[spacewander]

Since the kafka-logger's priority is low enough, we can just keep it untouched.
We are engineers, not mathematicians. We need to know when to trade off.

BTW, the gzip plugin doesn't do the gzip directly, it just controls the behavior of gzip.

[spacewander]

BTW, we are not expected to discuss the logging response body in an issue about the request body. It's off-topic. 😄

[windyrjc]

OK,thanks bro~

[SomalianIvan]

@spacewander can I give this a try?

[windyrjc]

@spacewander can I give this a try?

I'm already submit a PR😁 #5501

[membphis]

@spacewander can I give this a try?

you can can try to fix other issues: https://github.com/apache/apisix/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22