Can't reconfigure mqtts listener #349
Comments
|
When connecting with hivemq mqtt-cli I get this log |
|
I checked that the certificates are correct, and the 8883 port is correctly opened, so I guess the error is from the configuration? Here is the config: |
|
@maelp there's currently a minor issue coming from configuring default listeners in the release package. Will be fixed in next release. if the reconfigure error bothers you, try setting an MQTT default listener too To arrive at those, you have to translate to the ENV variable format here. I'll look at that client side error later, maybe related to allowed protocol version? 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
(the error seems to be that the server closes the socket when trying to do a SSL connection) |
|
I'm using a client which handles MQTT 3,4,5 I think (hive mqtt-cli), I'm not sure what version the VerneMQ server is handling, is that configurable? |
|
I have seen this error @ioolkos , might be fixed on my currently open PR in VerneMQ but we have to double check |
|
@ioolkos I tried adding the variables you sent above, now I have those logs |
|
Here's what I added |
|
But for now it seems I can't connect to VerneMQ with SSL, but perhaps I can do it using a Nginx in front, but this defeats the purpose of using VerneMQ TLS setting which can be convenient (to auto-set the user with the CommonName) |
|
I'll do some testing on the branch I've opened with these settings. What does |
|
@maelp MQTT protocol versions can be allowed per listener.
Terminating TLS in HAProxy or similar component is often done, yes. Of course, Verne can also terminate TLS without problems; so we'll find out what's happing here. 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
@codeadict this is the result |
|
(weird that there's no mqtts, is that because of the error above?) |
|
also not sure why there is mqttws although I haven't enabled it? |
|
If this helps, when I set the log level to debug, I have a lot of these: |
|
@ioolkos @codeadict when I disable "mqtt" and only enable "mqtts" in the config, I still have the "can't reconfigure mqtts" error but then each time I try to connect with SSL I now have this (before I didn't even have a log) and the connection still closes: |
|
@maelp the AE exchange logs above are unrelated and harmless (coming from the sync protocol). There are other users having troubles with MQTTS listeners in the Docker image, apparently: https://erlangforums.com/t/latest-vernemq-not-allowing-tls-traffic-with-self-signed-ca-certificate/2394 Have you enabled |
|
@ioolkos I added this, but same error |
|
Btw, the WS listener comes from the fact that we seem to start one via start script, in case the DOCKER_VERNEMQ variable is not set. Line 214 in 148c27a |
|
Can you check on the Pod, what the exact config injected into etc/vernemq/vernemq.conf is and post here? 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
@ioolkos this is what I get |
|
Can you also add: You could also attach to VerneMQ in the Pod by using
Detach from Verne with I'm currently unclear whether you manage to setup an SSL listener or not. (check with 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
Adding the default depth of 10 did not change anything for the command: if I include the final if I don't include it, I get an empty response |
|
If I do it with "1883" instead of "8883" I get this so I guess for some reason the mqtts handler is not launched, although I think I configured everything? is that related to the error above where it complains that it cannot reconfigure the listener (not sure why it tries to reconfigure the mqtts listener on 1883 instead of 8883?) |
|
It seems though that the 8883 port is correctly opened (and correctly proxied-through by the ingress-nginx TCP proxy in front), because when I try to make a MQTTS request on 8883 I get this log to appear on the service logs |
|
and if I try to send a non-SSL MQTT message on port 8883 (to try), I don't see a log on the server, and the socket is closed immediately |
|
Yeah, it's weird. Can you add the TLS listener not with 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
I'm not sure how to do this, since I guess with Kubernetes each time I deploy I might be on a different node IP no? |
|
for the "1883 reconfigure", the thing which surprises me is that it says both "mqtts" and "1883". Shouldn't it be "8883" that it tries to reconfigure for mqtts? |
|
The bug with the default listener should be fixed in my branch vernemq/vernemq#2078, if you can build Docker pointing to that it would be great. Although I believe there is still some bug that might have been caused by our latest ranch upgrade, I'm still researching the changes in ranch. But this is what I see: ^C%
codeadict@andromeda vernemq % _build/default/rel/vernemq/bin/vmq-admin listener stop address="127.0.0.1" port=8883
Done
codeadict@andromeda vernemq % _build/default/rel/vernemq/bin/vmq-admin listener show
+-------+---------+-----------+-------+------------+-----------+
| type | status | address | port | mountpoint | max_conns |
+-------+---------+-----------+-------+------------+-----------+
| http | running | 127.0.0.1 | 8888 | | 10000 |
+-------+---------+-----------+-------+------------+-----------+
| vmq | running | 0.0.0.0 | 44053 | | 10000 |
+-------+---------+-----------+-------+------------+-----------+
| mqtts | stopped | 127.0.0.1 | 8883 | | 10000 |
+-------+---------+-----------+-------+------------+-----------+
| mqtt | running | 127.0.0.1 | 1883 | | 10000 |
+-------+---------+-----------+-------+------------+-----------+
codeadict@andromeda vernemq % _build/default/rel/vernemq/bin/vmq-admin listener start address="127.0.0.1" port=8883
can't start listener due to '{already_started,<0.20463.0>}'
codeadict@andromeda vernemq % _build/default/rel/vernemq/bin/vmq-admin listener show
+-------+---------+-----------+-------+------------+-----------+
| type | status | address | port | mountpoint | max_conns |
+-------+---------+-----------+-------+------------+-----------+
| http | running | 127.0.0.1 | 8888 | | 10000 |
+-------+---------+-----------+-------+------------+-----------+
| vmq | running | 0.0.0.0 | 44053 | | 10000 |
+-------+---------+-----------+-------+------------+-----------+
| mqtts | stopped | 127.0.0.1 | 8883 | | 10000 |
+-------+---------+-----------+-------+------------+-----------+
| mqtt | running | 127.0.0.1 | 1883 | | 10000 |
+-------+---------+-----------+-------+------------+-----------+Notice when I try to start the listener again, it believes it's already started and remains stopped. |
|
I don't have a build script for the repo now, do you think you will publish an update soon of the package if this fixes the bug? |
|
@codeadict start and restart in What I do not understand in the open issue, is why we do not see a MQTTS listener running at all on 8883. (using Maybe you can configure a different port for your MQTTS listener and targetport to decouple things from that 1883/8883 question. 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
So you mean I should configure VerneMQ to listen to eg 8884? And have my ingress-nginx do the 8883 -> 8884? |
|
@ioolkos could the issue come from the Helm template? |
|
Yeah, that's what I meant. 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
🤣 yes kubernetes is sometimes a bit complicated I'll try changing the port. The |
|
I doubt it's due to Pulumi, the only thing Pulumi does is translating some DSL to actual |
|
Although this file seems to be handling the |
|
changing the port to 8884 doesn't change the issue |
|
For the "certificates" variables issue, could it be because in the certicicates-secret template there is a "$root" which is not in the statefulset file? certificate-secret: statefulset: |
|
There seems to be an issue in the Docker images for The issue does not seem to be in the Linux packages. In the This will get Nothing logged in the Verne log, this is a bummer. Investigating. This seems not related to configuration issues. (maybe OpenSSL version?) 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
I think I found the issue. I traced the SSL statemachine and when a connection comes in it gets an enoent error for the Now, I can manually do a But exec into the pod and do a 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
I get this indeed: |
|
👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
I found an easy fix for this. Use the full linked path for the CAFILE in your 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
It might also be interesting to add info / error logs in the pod if the certificates couldn't be loaded correctly, because for now there was no such logs |
|
Setting the ENV variables explicitly does not seem to solve the issue on my Helm deployment |
|
|
it seems to be due to the "cant_parse_connect_fixed_header", is there a way to debug this? Perhaps check whether the bytes do indeed correspond to some valid message? |
|
Please add Also, note that the The CA issue is unfortunately not captured by the Verne boot validators. We have 3 simple validators on those cert files: 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
I guess the reason is that for a self-signed situation the client also needs your specific CAfile with the cert chain. 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
Okay, got it working, I'll document the minimal changes that are required |
|
Seems to work with those options for some reason, it doesn't work if I don't set explictly |
|
Ugh, seems like an Heisenbug, I tried deleting the VerneMQ service and recreating it with the same options, and suddenly it no longer works |
|
Verne only picks up the settings injected into Be aware that the additional method to configure certs that you see in the main branch was only added 3 weeks ago: 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
|
@maelp We are facing the same issue. Could you let me know if you managed to fix it? |
|
@ManuelMueller1st the underlying cause in the packages was fixed but it seems with Helm it can still be an issue. Please determine what listeners you want to have running. Then we can determine which listeners to start manually (by adding to ENV), and which listeners are started implicitly by the service definitions. Possibly we can also just solve it by the naming of the listeners. 👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq |
Hi, I'm trying to set up a mqtts listener, I've added the mqtts.enabled=true in the values, as well as the secretMount with the certificates (it seemed that it didn't work by adding directly the "certificates.{certs,ca,key}" as documented in the README), and when I look at logs I see this:
and trying to send data on MQTT on port 8883 fails with an error (although it works fine for 1883)
is there something I should configure?
The text was updated successfully, but these errors were encountered: