After uninstalling Nomad file locks on the "alloc secrets" directory persists

[idrennanvmware]

Nomad version

Nomad v0.12.0 (8f7fbc8)

Operating system and Environment details

PhotonOS3

Issue

After draining a node of allocations, and stopping the Nomad agent entirely - file locks persist on the secrets folders for allocs.

We verified this by logging in to the vm and we are unable to remove the directories from even the command line (see output)

rm: cannot remove '/opt/nomad/alloc/41cf69f3-6a66-9b68-c100-b207e61d9bcf/telegraf-statsd-logging-task/secrets': Device or resource busy
rm: cannot remove '/opt/nomad/alloc/41cf69f3-6a66-9b68-c100-b207e61d9bcf/run-telegraf-with-statsd/secrets': Device or resource busy
rm: cannot remove '/opt/nomad/alloc/45375c6f-ee38-6ef0-73a1-f1dae791046a/fabio/secrets': Device or resource busy
rm: cannot remove '/opt/nomad/alloc/946bc1fa-d8ea-6c05-ed37-b258a4d64b26/journalbeat-task/secrets': Device or resource busy
rm: cannot remove '/opt/nomad/alloc/946bc1fa-d8ea-6c05-ed37-b258a4d64b26/filebeat-task/secrets': Device or resource busy

[tgross]

Hi @idrennanvmware! This looks to be another case of #7848. If the client node doesn't get a chance to GC before it's shut down, it's leaving these tmpfs mounts for secrets behind.

[idrennanvmware]

Gotcha! Thanks @tgross