DeadLetter messages related to metrics processing

[GrimMinion]

Hi,

Firstly thank you for this resource, the ability to export channel related metrics is appreciated, implementation was straight forward.
Running: v5.2.3
MQ: 9.2.0
Environment: EKS

We are now running this monitoring approach in Production, the monitor program runs as a service.

DEFINE SERVICE(MQPROMETHEUS) +
CONTROL(QMGR) +
SERVTYPE(SERVER) +
STARTCMD('/etc/mqm/mq_prometheus.sh') +
STARTARG(+QMNAME+) +
STOPCMD('/usr/bin/kill ' ) +
STOPARG(+MQ_SERVER_PID+) +
STDOUT('/var/mqm/errors/mq_prometheus.out') +
STDERR('/var/mqm/errors/mq_prometheus.out') +
DESCR('MQ exporter for Prometheus') +
REPLACE

We have been receiving DeadLetter messages on the Queue Manager deadletter queue.
We've previously configured the SYSTEM.ADMIN.TOPIC to not use a DLQ due to issues with queudepth etc, but that's a different story.

The messages on the queue have an application ID of: amqpcsea
This appears to be similar to the forum topic below.
http://www.mqseries.net/phpBB2/viewtopic.php?p=431613&sid=778397c983305eaa070d00ba364ab268

Ultimately it appears the command server response is trying to post to a queue that no longer exists as the monitoring process has been shutdown and the temporary dynamic queue no longer exists. Is there a way of avoiding this situation, by making sure that the process has shutdown cleanly? Has anyone else experienced this problem?
I'm reluctant to implement a deadletter handler to drop these non-persistent messages if possible.
An option would be to make use of a named channel, therefore after restart it will again be available for the in-flight responses, is this possible?
As far as I can tell the shutdown and restart is graceful, there is no forced termination.

Any help appreciated.

[ibmmqmet]

The programs assume use of a model queue for replies to make it easy to guarantee that noone else will be trying to share the queues that are created, and that they start out empty - there are two different reply queues for different uses, but only the single model needs to be named in the configuration. While it would be technically possible to change the code to support local queue definitions instead of the model, it's not in today's code nor in ideas for the future.

The Reason field in the deadletter headers would confirm that the reason the messages are ending up on the DLQ is that the replyQ is no longer available. I can't see any option for the command server that would bypass putting the message on a DLQ when there is any failure during its replies.

Perhaps more important would be the question on why a production system is having its monitor shutdown or why it might be dying. And I'd hope you would already be running some dead-letter handling process to report on or deal with problems that might be happening in other applications.

[GrimMinion]

Thanks for the response, I thought as much, in regards to the local queues.
We run MQ as a container on a Kubernetes platform, during routine maintenance a descheduler process re-distributes the workloads to ensure an even spread across the nodes available, during this activity MQ will be shutdown after receiving a SIGTERM message and restart on a different host. This happens gracefully the queue is quiesced etc, has not been a problem for the application queues.
Agreed on the deadletter handler, I'll put that forward as a potential improvement.