Shovel creation crashes with certain dest-queue-args

[maciejharczuk]

Describe the bug

I have identified two cases where shovel creation request to management api fail because dest-queue-args are not handled correctly, resulting in HTTP 500 and a stack trace being logged.

Reproduction steps

Case 1:
Creating a shovel for a quorum queue, trying to specify DLX and DLX routing key for the destination queue that we want the shovel to create
curl -XPUT http://localhost:15672/api/parameters/shovel/vhost1/stream-test -u guest:guest -d "@./shovel_payload.json" -v
with the following payload

{
  "value": {
    "src-protocol": "amqp091",
    "src-uri": "amqp:///vhost1",
    "src-queue": "quorum1",
    "src-consumer-args": {},
    "dest-protocol": "amqp091",
    "dest-uri": "amqp:///vhost1",
    "dest-queue": "quorum2",
    "dest-queue-args": {
      "x-queue-type": "quorum",
      "x-dead-letter-exchange": "dlx1",
      "x-dead-letter-routing-key": "rk1"
    }
  }
}

Results in a following stack trace:

2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     initial call: cowboy_stream_h:request_process/3
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     pid: <0.2003.0>
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     registered_name: []
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     exception error: bad argument
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>       in function  lists:keysearch/3
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>          called as lists:keysearch(<<"x-dead-letter-exchange">>,1,<<"rk1">>)
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>          *** argument 3: not a list
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>       in call from rabbit_misc:table_lookup/2 (rabbit_misc.erl, line 383)
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>       in call from rabbit_amqqueue:check_dlxrk_arg/2 (rabbit_amqqueue.erl, line 1034)
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>       in call from rabbit_parameter_validation:'-proplist/3-fun-0-'/3 (rabbit_parameter_validation.erl, line 69)
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>       in call from lists:foldl_1/3 (lists.erl, line 1355)
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>       in call from rabbit_parameter_validation:proplist/3 (rabbit_parameter_validation.erl, line 63)
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>       in call from rabbit_parameter_validation:'-proplist/3-fun-0-'/3 (rabbit_parameter_validation.erl, line 80)
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>       in call from lists:foldl_1/3 (lists.erl, line 1355)
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     ancestors: [<0.2002.0>,<0.818.0>,<0.812.0>,<0.811.0>,<0.809.0>,
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>                   rabbit_web_dispatch_sup,<0.798.0>]
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     message_queue_len: 0
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     messages: []
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     links: [<0.2002.0>]
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     dictionary: []
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     trap_exit: false
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     status: running
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     heap_size: 6772
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     stack_size: 28
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>     reductions: 3851
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>   neighbours:
2023-05-24 12:00:37.328179+00:00 [error] <0.2003.0>
2023-05-24 12:00:37.328926+00:00 [error] <0.2002.0> Ranch listener {acceptor,{0,0,0,0,0,0,0,0},15672}, connection process <0.2002.0>, stream 1 had its request process <0.2003.0> exit with reason badarg and stacktrace [{lists,keysearch,[<<"x-dead-letter-exchange">>,1,<<"rk1">>],[{error_info,#{module => erl_stdlib_errors}}]},{rabbit_misc,table_lookup,2,[{file,"rabbit_misc.erl"},{line,383}]},{rabbit_amqqueue,check_dlxrk_arg,2,[{file,"rabbit_amqqueue.erl"},{line,1034}]},{rabbit_parameter_validation,'-proplist/3-fun-0-',3,[{file,"rabbit_parameter_validation.erl"},{line,69}]},{lists,foldl_1,3,[{file,"lists.erl"},{line,1355}]},{rabbit_parameter_validation,proplist,3,[{file,"rabbit_parameter_validation.erl"},{line,63}]},{rabbit_parameter_validation,'-proplist/3-fun-0-',3,[{file,"rabbit_parameter_validation.erl"},{line,80}]},{lists,foldl_1,3,[{file,"lists.erl"},{line,1355}]}]

Case 2:
Creating a shovel for a stream, trying to specify x-max-age for the destination stream.
curl -XPUT http://localhost:15672/api/parameters/shovel/vhost1/stream-test -u guest:guest -d "@./shovel_payload.json" -v
JSON payload:

{
  "value": {
    "src-protocol": "amqp091",
    "src-uri": "amqp:///vhost1",
    "src-queue": "stream1",
    "src-consumer-args": {},
    "dest-protocol": "amqp091",
    "dest-uri": "amqp:///vhost1",
    "dest-queue": "quorum2",
    "dest-queue-args": {
      "x-queue-type": "stream",
      "x-max-age": "12h"
    }
  }
}

2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>   crasher:
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>     initial call: cowboy_stream_h:request_process/3
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>     pid: <0.2123.0>
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>     registered_name: []
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>     exception error: no function clause matching
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>                      rabbit_amqqueue:check_max_age_arg(<<"x-max-age">>,
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>                                                        <<"12h">>) (rabbit_amqqueue.erl, line 977)
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>       in function  rabbit_parameter_validation:'-proplist/3-fun-0-'/3 (rabbit_parameter_validation.erl, line 69)
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>       in call from lists:foldl_1/3 (lists.erl, line 1355)
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>       in call from rabbit_parameter_validation:proplist/3 (rabbit_parameter_validation.erl, line 63)
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>       in call from rabbit_parameter_validation:'-proplist/3-fun-0-'/3 (rabbit_parameter_validation.erl, line 80)
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>       in call from lists:foldl_1/3 (lists.erl, line 1355)
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>       in call from rabbit_parameter_validation:proplist/3 (rabbit_parameter_validation.erl, line 63)
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>       in call from rabbit_shovel_parameters:validate/5 (rabbit_shovel_parameters.erl, line 41)
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>     ancestors: [<0.2122.0>,<0.820.0>,<0.812.0>,<0.811.0>,<0.809.0>,
2023-05-24 12:07:25.533806+00:00 [error] <0.2123.0>                   rabbit_web_dispatch_sup,<0.798.0>]

Expected behavior

I would expect the dest-queue-args to be used when the shovel declares a destination queue. Alternatively, if those particular args can not be supported for whatever reasons, it should be caught during parameter validation and HTTP 400 with meaningful error message should be returned instead of a 500 with no useful info.

Additional context

No response

[lukebakken]

Hello, what version of RabbitMQ and Erlang are you using?

Streams are definitely not supported at this time, FYI.

(@michaelklishin please don't convert this to a discussion. There are legitimate things to work on for the second item, and I'm investigating the first).

[maciejharczuk]

I was able to reproduce it with rabbitmq:3.11.16-management-alpine docker image, which means it's RabbitMQ 3.11.16 & Erlang 25.3.2.

Interestingly, a shovel gets created and declares a stream if I remove the 'x-max-age' but leave "x-queue-type": "stream", so it kind of works, although more by accident than by design if I understood the code correctly.

[lukebakken]

It looks like in the first case that the json isn't getting correctly converted into an AMQP table. That feature was introduced in this PR.

Thanks again @G-r-F, we'll look into this as time allows.

[johanrhodin]

Streams are definitely not supported at this time, FYI.

Why would that be? I don't see it documented as not being supported. There is functionality especially dedicated to this: de8dd5f

[michaelklishin]

@johanrhodin because we haven't done any meaningful amount of work to support streams. Shovels don't have to use AMQP 0-9-1 for streams, for example.