bug: no connection to MQTT broker possible by using mqtt_proxy

[danielkocot]

Issue description

By using the mqtt_proxy plugin it isn't possible to make up a connection to the mqtt broker, in this case mosquitto. All done with docker compose.

Environment

• apisix version (cmd: apisix version): 2.11.0
• OS (cmd: uname -a): Darwin Kernel Version 20.6.0

Steps to reproduce

1. create docker environment based on APISIX-docker with additional mosquitto service
2. configure stream_proxy in config.yaml
3. create stream_routes according to the example provided in the documentation

Actual result

mqtt pub -t test -m "Hello" -h 127.0.0.1 -p 1883 creates a topic test with message Hello.
mqtt pub -t test -m "Hello" -h 127.0.0.1 -p 9100 does nothing the mqtt client is running into a timeout

Error log

2021/12/14 09:16:03 [warn] 46#46: 4 [lua] plugin.lua:172: load(): new plugins: {"zipkin":true,"openid-connect":true,"key-auth":true,"ip-restriction":true,"uri-blocker":true,"batch-requests":true,"fault-injection":true,"basic-auth":true,"request-validation":true,"grpc-transcode":true,"example-plugin":true,"kafka-logger":true,"ldap-auth":true,"wolf-rbac":true,"limit-req":true,"consumer-restriction":true,"limit-count":true,"tcp-logger":true,"serverless-post-function":true,"serverless-pre-function":true,"echo":true,"client-control":true,"ext-plugin-post-req":true,"sls-logger":true,"azure-functions":true,"udp-logger":true,"gzip":true,"syslog":true,"http-logger":true,"proxy-cache":true,"skywalking-logger":true,"api-breaker":true,"authz-casbin":true,"jwt-auth":true,"server-info":true,"traffic-split":true,"real-ip":true,"proxy-mirror":true,"response-rewrite":true,"limit-conn":true,"redirect":true,"cors":true,"ua-restriction":true,"datadog":true,"prometheus":true,"ext-plugin-pre-req":true,"authz-keycloak":true,"proxy-rewrite":true,"hmac-auth":true,"referer-restriction":true,"request-id":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 48#48: 3 [lua] plugin.lua:172: load(): new plugins: {"zipkin":true,"openid-connect":true,"key-auth":true,"ip-restriction":true,"uri-blocker":true,"batch-requests":true,"fault-injection":true,"basic-auth":true,"request-validation":true,"grpc-transcode":true,"example-plugin":true,"kafka-logger":true,"ldap-auth":true,"wolf-rbac":true,"limit-req":true,"consumer-restriction":true,"limit-count":true,"tcp-logger":true,"serverless-post-function":true,"serverless-pre-function":true,"echo":true,"client-control":true,"ext-plugin-post-req":true,"sls-logger":true,"azure-functions":true,"udp-logger":true,"gzip":true,"syslog":true,"http-logger":true,"proxy-cache":true,"skywalking-logger":true,"api-breaker":true,"authz-casbin":true,"jwt-auth":true,"server-info":true,"traffic-split":true,"real-ip":true,"proxy-mirror":true,"response-rewrite":true,"limit-conn":true,"redirect":true,"cors":true,"ua-restriction":true,"datadog":true,"prometheus":true,"ext-plugin-pre-req":true,"authz-keycloak":true,"proxy-rewrite":true,"hmac-auth":true,"referer-restriction":true,"request-id":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 49#49: 1 [lua] plugin.lua:172: load(): new plugins: {"zipkin":true,"openid-connect":true,"key-auth":true,"ip-restriction":true,"uri-blocker":true,"batch-requests":true,"fault-injection":true,"basic-auth":true,"request-validation":true,"grpc-transcode":true,"example-plugin":true,"kafka-logger":true,"ldap-auth":true,"wolf-rbac":true,"limit-req":true,"consumer-restriction":true,"limit-count":true,"tcp-logger":true,"serverless-post-function":true,"serverless-pre-function":true,"echo":true,"client-control":true,"ext-plugin-post-req":true,"sls-logger":true,"azure-functions":true,"udp-logger":true,"gzip":true,"syslog":true,"http-logger":true,"proxy-cache":true,"skywalking-logger":true,"api-breaker":true,"authz-casbin":true,"jwt-auth":true,"server-info":true,"traffic-split":true,"real-ip":true,"proxy-mirror":true,"response-rewrite":true,"limit-conn":true,"redirect":true,"cors":true,"ua-restriction":true,"datadog":true,"prometheus":true,"ext-plugin-pre-req":true,"authz-keycloak":true,"proxy-rewrite":true,"hmac-auth":true,"referer-restriction":true,"request-id":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 47#47: 2 [lua] plugin.lua:172: load(): new plugins: {"zipkin":true,"openid-connect":true,"key-auth":true,"ip-restriction":true,"uri-blocker":true,"batch-requests":true,"fault-injection":true,"basic-auth":true,"request-validation":true,"grpc-transcode":true,"example-plugin":true,"kafka-logger":true,"ldap-auth":true,"wolf-rbac":true,"limit-req":true,"consumer-restriction":true,"limit-count":true,"tcp-logger":true,"serverless-post-function":true,"serverless-pre-function":true,"echo":true,"client-control":true,"ext-plugin-post-req":true,"sls-logger":true,"azure-functions":true,"udp-logger":true,"gzip":true,"syslog":true,"http-logger":true,"proxy-cache":true,"skywalking-logger":true,"api-breaker":true,"authz-casbin":true,"jwt-auth":true,"server-info":true,"traffic-split":true,"real-ip":true,"proxy-mirror":true,"response-rewrite":true,"limit-conn":true,"redirect":true,"cors":true,"ua-restriction":true,"datadog":true,"prometheus":true,"ext-plugin-pre-req":true,"authz-keycloak":true,"proxy-rewrite":true,"hmac-auth":true,"referer-restriction":true,"request-id":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 52#52: 5 [lua] plugin.lua:172: load(): new plugins: {"zipkin":true,"openid-connect":true,"key-auth":true,"ip-restriction":true,"uri-blocker":true,"batch-requests":true,"fault-injection":true,"basic-auth":true,"request-validation":true,"grpc-transcode":true,"example-plugin":true,"kafka-logger":true,"ldap-auth":true,"wolf-rbac":true,"limit-req":true,"consumer-restriction":true,"limit-count":true,"tcp-logger":true,"serverless-post-function":true,"serverless-pre-function":true,"echo":true,"client-control":true,"ext-plugin-post-req":true,"sls-logger":true,"azure-functions":true,"udp-logger":true,"gzip":true,"syslog":true,"http-logger":true,"proxy-cache":true,"skywalking-logger":true,"api-breaker":true,"authz-casbin":true,"jwt-auth":true,"server-info":true,"traffic-split":true,"real-ip":true,"proxy-mirror":true,"response-rewrite":true,"limit-conn":true,"redirect":true,"cors":true,"ua-restriction":true,"datadog":true,"prometheus":true,"ext-plugin-pre-req":true,"authz-keycloak":true,"proxy-rewrite":true,"hmac-auth":true,"referer-restriction":true,"request-id":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 49#49: 1 [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 47#47: 2 [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 48#48: 3 [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 46#46: 4 [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 52#52: 5 [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 47#47: 6 stream [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 48#48: 7 stream [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 49#49: 8 stream [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 46#46: 9 stream [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua
2021/12/14 09:16:03 [warn] 52#52: 10 stream [lua] plugin.lua:222: load_stream(): new plugins: {"ip-restriction":true,"mqtt-proxy":true,"limit-conn":true}, context: init_worker_by_lua

Expected result

The creation of the topic test with the message Hello

[moonming]

@danielkocot please try #5816; thanks for your report 👍

[danielkocot]

Hi @moonming, for me that still isn't working. I don't see any traffic in the access.log.

[moonming]

@spacewander please take a look, thanks

[spacewander]

The access.log is only for HTTP requests. Can mosquitto_sub -t test -v subscribe the proxy publish info?

[danielkocot]

I'm using the mqtt client but subscription doesn't work either.

[juzhiyuan]

doesn't work either.

May I know if there have more error information that could help to debug?

[spacewander]

I tried mosquitto_pub -t test -m "Hello" -h 127.0.0.1 -p 9100 & mosquitto_sub -t test -v, and it worked well with me.

Maybe you can provide the detailed commands & configuration so we can reproduce it?

[danielkocot]

I used the mqtt client from HiveMQ. Now I used mosquito_pub with mosquitto_pub -t test -m "Hello" -h 127.0.0.1 -p 9100 and the result is Error: The connection is lost.

[spacewander]

Interesting. Here is the result running on my machine:

 incubator-apisix (master) 0
 ￥ mosquitto_sub -t test -v &
[1] 23990
 incubator-apisix (master) 0
 ￥ mosquitto_pub -t test -m "Hello" -p 9100
test Hello

My stream route conf is:

{"plugins":{"mqtt-proxy":{"protocol_level":4,"protocol_name":"MQTT","upstream":{"host":"127.0.0.1","port":1883}}}}

My config.yaml is:

apisix:
  #dns_resolver:                  # If not set, read from `/etc/resolv.conf`
    #- 127.0.0.1:1053
  enable_admin: true
  stream_proxy:                 # TCP/UDP proxy
    only: false
    tcp:                        # TCP proxy port list
      - addr: 9100

My mosquitto_pub version is:

mosquitto_pub --version
mosquitto_pub version 2.0.14 running on libmosquitto 2.0.14.

[danielkocot]

I use the stream route configuration from the documentation
{ "remote_addr": "127.0.0.1", "plugins": { "mqtt-proxy": { "protocol_name": "MQTT", "protocol_level": 5, "upstream": { "host": "127.0.0.1", "port": 1883 } } } }
I will test it with your configuration :). Maybe remote_addris the problem ;).

[juzhiyuan]

Maybe remote_addris the problem ;).

Hi Daniel, if this is the root cause, kindly let us know 😄

[danielkocot]

Hi, sorry for the delay. It works now. But using docker on MacOS the upstream host needs the value host.docker.internal.

[juzhiyuan]

Glad to know your issue is resolved, if docs need to be updated, kindly submit a PR 😄