Definition load fails for dynamic shovels when management.load_definitions is used #2428
-
|
Hi, According to Changelog, #2384 should have fixed issues with loading definitions at startup.
Unfortunately, it seems that the import of definitions is incomplete: the dynamic shovel is not loaded. I run a version 3.8.6 via the official Docker image with definitions from rabbitmq/rabbitmq-shovel/issues/13 (tested with plugins enabled offline and "not offline") Logs : At I didn't found recent conversations in the Google groups about this issue. Thanks, Regards. |
Beta Was this translation helpful? Give feedback.
Answered by
michaelklishin
Aug 12, 2020
Replies: 10 comments 4 replies
-
|
According to the log, definition import has been performed twice: once during plugin startup, another one after all plugins were enabled. The image still relies on |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
michaelklishin
-
|
I updated |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
I thank it was a bug, sorry about that. |
Beta Was this translation helpful? Give feedback.
-
|
We realize that the difference is not obvious. I have removed Also, here's what should change in the image: docker-library/rabbitmq#429. I'll look into a PR in a bit. You have two options at the moment:
|
Beta Was this translation helpful? Give feedback.
-
Yup, the second import was done manually via the web interface.
To avoid this, instead of using the standard
If I'm right (not sure) RUN is not possible after CMD/ENTRYPOINT. So I'm using this HEALTHCHECK trick (docker/compose/issues/1809#issuecomment-657815188) to do the #!/usr/bin/env bash
set -e
FIRST_READY_STATUS_FLAG='/tmp/.FIRST_READY_STATUS_FLAG'
if [ ! -f "${FIRST_READY_STATUS_FLAG}" ]; then
touch "${FIRST_READY_STATUS_FLAG}"
rabbitmqctl await_startup
rabbitmqctl import_definitions /etc/rabbitmq/config.json # reload definitions
# also tried with this
# curl -X POST -H "Accept:application/json" -H "Content-Type: application/json" -u guest:guest -d @/etc/rabbitmq/config.json "http://localhost:15672/api/definitions"
fi
I've declared a config ( It seems that shovel import works only via the web interface. Am I doing something wrong? |
Beta Was this translation helpful? Give feedback.
-
|
@Gounlaf would you mind posting your definitions file so that we can try it? We don't know much about your environment and would strongly prefer to not guess. If |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
As mentioned above, I've done all tests with definitions from rabbitmq/rabbitmq-shovel/issues/13 so that it would be simple to reproduce :) {
"users": [
{
"name": "guest",
"password_hash": "YhXgK79L2qYsuPCUEmBRY3np5B3TsfM9u6u+wzDvA0iPMeGh",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": "administrator"
}
],
"vhosts": [
{
"name": "/"
}
],
"permissions": [
{
"user": "guest",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
}
],
"parameters": [
{
"value": {
"src-uri": "amqp://localhost",
"src-queue": "source",
"dest-uri": "amqp://localhost",
"dest-queue": "destination",
"add-forward-headers": false,
"ack-mode": "on-confirm",
"delete-after": "never"
},
"vhost": "/",
"component": "shovel",
"name": "test"
}
],
"policies": [],
"queues": [
{
"name": "source",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "destination",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
}
],
"exchanges": [],
"bindings": []
} |
Beta Was this translation helpful? Give feedback.
-
|
I have tried definition import on boot using {
"rabbit_version": "3.8.6",
"rabbitmq_version": "3.8.6",
"product_name": "RabbitMQ",
"product_version": "3.8.6",
"users": [
{
"name": "langohr",
"password_hash": "tDEJUA3QNWHQ6w9fBy0Q8FsZE+6rBEM98VDSmAozhq8eXSKv",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": ""
},
{
"name": "bunny_reader",
"password_hash": "BY8m0H2PaE/uuqfQYryOKmXDGkBM96Lm0aptpiWwtYDvJVyG",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": ""
},
{
"name": "bunny_gem",
"password_hash": "lDFQcG2+kjFpZV2yhZJBBvMihq7Yyz5HxVokan8KfNnbXJT8",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": ""
},
{
"name": "alt3",
"password_hash": "nh6g0pStjX7UOI5BnP6af2Ir+6dEFXAsWXkhyFYSAk7HKkcg",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": ""
},
{
"name": "guest",
"password_hash": "/cLEk21Y6zj4pmzq6qxud76eZB0UWGN5zYVTGe5C3sttFWXb",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": "administrator"
},
{
"name": "temp-user",
"password_hash": "5MrZF0vctgvPFTPyQXHO2vJivwg8UWnWsgNSNFe7YtNoxkWG",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": "management"
},
{
"name": "alt-user",
"password_hash": "tt2zwcZmuwm9H3i7h2/3AD/hW5VWuPh6JlzuKuOqcAbehyGN",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": ""
}
],
"vhosts": [
{
"name": "langohr_testbed"
},
{
"name": "vh1"
},
{
"name": "bunny_testbed"
},
{
"name": "vh2"
},
{
"name": "rabbit/hole"
},
{
"name": "/"
}
],
"permissions": [
{
"user": "guest",
"vhost": "langohr_testbed",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "langohr",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "bunny_reader",
"vhost": "bunny_testbed",
"configure": "^---$",
"write": "^---$",
"read": ".*"
},
{
"user": "guest",
"vhost": "vh2",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "guest",
"vhost": "vh1",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "guest",
"vhost": "rabbit/hole",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "temp-user",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "guest",
"vhost": "bunny_testbed",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "langohr",
"vhost": "langohr_testbed",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "bunny_gem",
"vhost": "bunny_testbed",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "guest",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
}
],
"topic_permissions": [
{
"user": "guest",
"vhost": "/",
"exchange": "amq.topic",
"write": ".*",
"read": ".*"
}
],
"parameters": [
{
"value": [],
"vhost": "/",
"component": "vhost-limits",
"name": "limits"
},
{
"value": {
"ack-mode": "on-confirm",
"dest-add-forward-headers": false,
"dest-protocol": "amqp091",
"dest-queue": "dest-q",
"dest-uri": "amqp://",
"src-delete-after": "never",
"src-protocol": "amqp091",
"src-queue": "src-q",
"src-uri": "amqp://"
},
"vhost": "/",
"component": "shovel",
"name": "sh-1"
}
],
"global_parameters": [
{
"name": "cluster_name",
"value": "rabbitmq@localhost"
},
{
"name": "internal_cluster_id",
"value": "rabbitmq-cluster-id-3_EHwwVS-FL1YjYws__6EA"
}
],
"policies": [
{
"vhost": "/",
"name": "limited",
"pattern": "^limited\\.",
"apply-to": "queues",
"definition": {
"max-length": 11
},
"priority": 0
}
],
"queues": [
{
"name": "queue1",
"vhost": "vh1",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "qq.1",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-max-length": 992,
"x-queue-type": "quorum"
}
},
{
"name": "dest-q",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "fq.cq1",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "classic"
}
},
{
"name": "limited.cq.1",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "classic"
}
},
{
"name": "federation: federated.fanout1 -> hare@localhost",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-internal-purpose": "federation"
}
},
{
"name": "limited.qq.1",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-queue-type": "quorum"
}
},
{
"name": "src-q",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "langohr.tests2.queues.client-named.durable.non-exclusive.non-auto-deleted",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "langohr.test.recovery.q1",
"vhost": "/",
"durable": true,
"auto_delete": true,
"arguments": {}
}
],
"exchanges": [
{
"name": "bunny.stress.concurrent.consumers.topic",
"vhost": "bunny_testbed",
"type": "topic",
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
},
{
"name": "federation: federated.fanout1 -> hare@localhost B",
"vhost": "/",
"type": "x-federation-upstream",
"durable": true,
"auto_delete": true,
"internal": true,
"arguments": {
"x-downstream-name": "hare@localhost",
"x-internal-purpose": "federation",
"x-max-hops": 1
}
},
{
"name": "langohr.tests.exchanges.topic2",
"vhost": "/",
"type": "topic",
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
},
{
"name": "langohr.test.recovery.fanout2",
"vhost": "/",
"type": "fanout",
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
},
{
"name": "federated.fanout1",
"vhost": "/",
"type": "fanout",
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
},
{
"name": "exchangeA",
"vhost": "/",
"type": "topic",
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
},
{
"name": "langohr.tests.exchanges.fanout2",
"vhost": "/",
"type": "fanout",
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
},
{
"name": "langohr.tests.exchanges.direct2",
"vhost": "/",
"type": "direct",
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
}
],
"bindings": [
{
"source": "federation: federated.fanout1 -> hare@localhost B",
"vhost": "/",
"destination": "federation: federated.fanout1 -> hare@localhost",
"destination_type": "queue",
"routing_key": "",
"arguments": {}
}
]
}and it works as expected. A shovel connection is running and functional. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
@Gounlaf your file is imported successfully via HTTP API, using |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
What I see after the most recent attempt with
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I have the following patch for the image's entry point: ---
docker-entrypoint.sh | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh
index 1b16604..a06e709 100755
--- a/docker-entrypoint.sh
+++ b/docker-entrypoint.sh
@@ -386,8 +386,10 @@ if [ "$1" = 'rabbitmq-server' ] && [ "$shouldWriteConfig" ]; then
# https://www.rabbitmq.com/management.html#load-definitions
managementDefinitionsFile='/etc/rabbitmq/definitions.json'
if [ -f "$managementDefinitionsFile" ]; then
- # see also https://github.com/docker-library/rabbitmq/pull/112#issuecomment-271485550
- rabbit_set_config 'management.load_definitions' "$managementDefinitionsFile"
+ # We use `load_definitions` (the built-in setting as of 3.8.2+) instead
+ # of `management.load_definitions`.
+ # See https://github.com/docker-library/rabbitmq/issues/429 for details.
+ rabbit_set_config 'load_definitions' "$managementDefinitionsFile"
fi
fi
fiand produced a "base" image like so from the community image repository: docker build -t rabbitmq:issue-429 .then, using the folloding FROM rabbitmq:issue-429
RUN rabbitmq-plugins enable --offline rabbitmq_shovel rabbitmq_shovel_management
COPY dynamic_shovels.json /etc/rabbitmq/definitions.json
COPY rabbitmq.conf /etc/rabbitmq/rabbitmq.conf
ENV LOG debug
ENV RABBITMQ_CONFIG_FILE /etc/rabbitmq/rabbitmq.conf
ENV RABBITMQ_ALLOW_INPUT 1
EXPOSE 5672 15672
CMD ["rabbitmq-server"]{
"users":[
{
"name":"visitor",
"password_hash":"PU4rzAKrEpp+r47JRHptn88ifHwbS1IntWoAdD19qlJyuJG3",
"hashing_algorithm":"rabbit_password_hashing_sha256",
"tags":"administrator"
}
],
"vhosts": [
{
"name": "/"
}
],
"permissions": [
{
"user": "visitor",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
}
],
"parameters": [
{
"value": {
"src-uri": "amqp://visitor:visitor@localhost",
"src-queue": "source",
"dest-uri": "amqp://visitor:visitor@localhost",
"dest-queue": "destination",
"add-forward-headers": false,
"ack-mode": "on-confirm",
"delete-after": "never"
},
"vhost": "/",
"component": "shovel",
"name": "test"
}
],
"policies": [],
"queues": [
{
"name": "source",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "destination",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {}
}
],
"exchanges": [],
"bindings": []
}log.file.level = debug
log.console.level = debug
load_definitions = /etc/rabbitmq/definitions.json
# Default data seeding will be skipped if load_definitions is used.
# default_user = visitor
# default_pass = visitorI build and run the image like this: docker build -t rabbitmq:issue-429-final .
docker run -p 15672:15672 -p 5672:5672 -it rabbitmq:issue-429-finalI observe the following when I sign in:
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
To conclude: I could not reproduce the issue with docker-library/rabbitmq#430, which uses |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
@michaelklishin thanks for your investigation. I've just retried from scratch at home and, as you said, everything is working. Until your patch for the Docker image is accepted, I still can make it work if I provide a config and I put the definitions somewhere other than the one tested in the entry-point. Thanks again. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Great, thanks for the confirmation :) The Docker image maintainers are quite responsive, it should not take long. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
According to the log, definition import has been performed twice: once during plugin startup, another one after all plugins were enabled.
The image still relies on
management.load_definitionsinstead of the coreload_definitions. It therefore will kick off definition import when the management plugin is enabled.