Replies: 2 comments 7 replies
-
|
The key info is in Logs from node 1 above. Initially I also thought #14401 would address it. However, I see it was backported to 4.1.4, but we still encounter this issue after upgrading to 4.1.6. |
Beta Was this translation helpful? Give feedback.
-
|
Also please explain exactly what "Once it happened, the only solution to recover is to clean out the volumes to start from empty." means? Deleting data on RabbitMQ volumes without first removing the node from the cluster (forget_node) is unsupported and very likely to result in hard to fix issues. Also are these queues short lived? i.e. are the deleted and re-created frequently? |
Beta Was this translation helpful? Give feedback.
-
|
@kjnilsson Unfortunately, our SRE team has rebuilt the rabbitmq cluster. We will try that command next time we reproduce the problem. For the second question: I mean rebuilding the rabbitmq cluster, essentially. Shutdown all old pods, and create new ones with new volumes. Just the IPs are still the same as before. The queues are long-lived. There are queue deletion in the log because the queues have TTL. When the problem happens, apps fails at basic-consume step (app gets a timeout exception while rabbitmq server logged |
Beta Was this translation helpful? Give feedback.
-
|
re-creating the cluster should not be necessary, you can use the delete|add member functions in the |
Beta Was this translation helpful? Give feedback.
-
|
@kjnilsson Good to know that. But I can understand why our SRE team simply recreate the cluster. As we have ~800 queues, and there are some of them in a bad state. You'd need to find those bad queues by reading logs, and find out which nodes they are on currently. And most probably they scattered around all the nodes in the end. It is much easier to recreate the cluster as we are running it in kubernetes. |
Beta Was this translation helpful? Give feedback.
-
|
@qrli if you are willing to do that, you might as well export the definitions (of one virtual host), delete the virtual host and re-import the definitions using That would also preserve the logs which we will inevitably ask for. You can use |
Beta Was this translation helpful? Give feedback.
-
|
@qrli you were recommended to upgrade to |
Beta Was this translation helpful? Give feedback.
-
|
we will try👍 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Community Support Policy
RabbitMQ version used
4.1.2
Erlang version used
27.3.x
Operating system (distribution) used
ubuntu
How is RabbitMQ deployed?
Community Docker image
rabbitmq-diagnostics status output
See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics
Logs from node 1 (with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
Logs from node 2 (if applicable, with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
Logs from node 3 (if applicable, with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
rabbitmq.conf
See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location
PASTE advanced.config HERE, BETWEEN BACKTICKS
Kubernetes deployment file
What problem are you trying to solve?
Would be better to know how to avoid the unconsumable queues from happen. It is not happening every time. A rough estimation is 20% chance. Once it happened, the only solution to recover is to clean out the volumes to start from empty.
Beta Was this translation helpful? Give feedback.
All reactions