Broker Disconnect After CONNECT Request

[pascaldekloe]

Environment:

Docker image emqx/emqx:4.2.5.

What happened and what you expected to happen:

The broker closes the connection after receiving a CONNECT request during startup shortly.
The broker must respond to incoming CONNECT requests when listening on a TCP port.

How to reproduce it (as minimally and precisely as possible):

(1) Download client from https://gobinaries.com/pascaldekloe/mqtt/cmd/mqttc.
(2) Start broker with something like docker run -d -p 1883:1883 emqx/emqx:4.2.5.
(3) Run mqttc localhost quickly after (2).

The error printed is mqtt: broker closed the connection (EOF); CONNECT not confirmed. Wireshark confirms connection loss after request submission.

[tigercl]

@pascaldekloe Can you provide the emqx log, it is usually .../emqx/log/emqx.log.*. Or you can try another client, such as mqttx and emqtt.

[pascaldekloe]

The issue is not about the client. I have given you a full analysis + steps to reproduce.

[qzhuyan]

@pascaldekloe could you provide source code of mqttc?

[zmstone]

Hi.
@pascaldekloe
First of all thank you for the detailed steps.

We are not blaming the client, this is such a basic scenario, sometimes it's a lot easier to understand what went wrong if you happen to have tried other clients to compare the behaviour on both client/server side.

And I find your comment a bit harsh, not to mention it's a random bash script you are asking us to download and run.

[zmstone]

@qzhuyan it's a shebang bin/sh which downloads some other binary and run if I understand correct from my quick glance

[zmstone]

this? https://github.com/pascaldekloe/mqtt
I'd be happy to try it out.

[zmstone]

I failed. It does not seem to be allowed to create an issue in your repo: https://github.com/pascaldekloe/mqtt
@pascaldekloe

default branch go run crashes
go version 1.14

runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed

runtime stack:
runtime.throw(0xa3b43e, 0xc)
        /usr/local/go/src/runtime/panic.go:1112 +0x72
runtime.mlockGsignal(0xc000680480)
        /usr/local/go/src/runtime/os_linux_x86.go:72 +0x107
runtime.mpreinit(0xc000023500)
        /usr/local/go/src/runtime/os_linux.go:341 +0x78
runtime.mcommoninit(0xc000023500)
        /usr/local/go/src/runtime/proc.go:630 +0x108
runtime.allocm(0xc000043000, 0xa82448, 0x2e62b595)
        /usr/local/go/src/runtime/proc.go:1390 +0x14e
runtime.newm(0xa82448, 0xc000043000)
        /usr/local/go/src/runtime/proc.go:1704 +0x39
runtime.startm(0x0, 0xc000582301)
        /usr/local/go/src/runtime/proc.go:1869 +0x12a
runtime.wakep(...)
        /usr/local/go/src/runtime/proc.go:1953
runtime.resetspinning()
        /usr/local/go/src/runtime/proc.go:2415 +0x93
runtime.schedule()
        /usr/local/go/src/runtime/proc.go:2527 +0x2de
runtime.goschedImpl(0xc0001ecd80)
        /usr/local/go/src/runtime/proc.go:2705 +0xd6
runtime.gopreempt_m(0xc0001ecd80)
        /usr/local/go/src/runtime/proc.go:2733 +0x34
runtime.newstack()
        /usr/local/go/src/runtime/stack.go:1025 +0x1ae
runtime.morestack()
        /usr/local/go/src/runtime/asm_amd64.s:449 +0x8f

[pascaldekloe]

Seems like your Go runtime doesn't like the OS it's on. 😬 You need Go version 1.16 for this project though.

What I'm trying to say is that the client of choice doesn't matter. It's just a plain connection close after the intial CONNECT packet. You can even use netcat to reproduce.

% hexdump CONNECT.bin 
0000000 10 0a 00 04 4d 51 54 54 04 00 00               
000000b
% docker run -d -p 1883:1883 emqx/emqx:4.2.5
% cat CONNECT.bin - | nc localhost 1883 | hexdump

[pascaldekloe]

To be clear: this issue exists only for a second or two during startup. Once the daemon is fully started, things are fine. Also, EMQ X is not the only one with this boot problem.

[zmstone]

Maybe allow the service to start successfully ?

i.e. Wait for EMQ X Broker 4.2.5 is started successfully! log ?

[pascaldekloe]

How would parsing text for daemon ready-ness work? Do you suggest a script with a regular expression, which exposes the TCP port on a match? Better hope that log line does not change in future versions then. 😄

[zmstone]

if you are suggesting that there should be a readiness probe for the service running in the container.
sure that's something we can work on.

[pascaldekloe]

I'm suggesting the daemon should be ready when it listens to a socket. Fixing this issue prevents a whole lot of complexity on container level.

[zmstone]

docker exposes the port immediately to your host.

[zmstone]

if you are writing automated tests, I guess you'll need some sort of readiness probe?
querying docker container health status is something can be generic for all containers.
--- yes, this is something we can and should provide.

[zmstone]

tested it with:
path/to/emqx/bin/emqx start && go run cmd/mqttc/main.go localhost

there is no error.