The problem of non-well-behaved containers (eg Mosquitto)

[Paraphraser]

The problem of non-well-behaved containers (eg Mosquitto)

This is a long post and I apologise for that in advance. The problem it discusses is the underlying cause of many Discord questions. The proposed solution will, I hope, become a model for all non-well-behaved containers and serve IOTstack users well into the future. I'd appreciate feedback.

The post:

• Explains the concept of a non-well-behaved container
• Uses Mosquitto as a case study, exploring the kinds of things that can (and do) go wrong
• Describes a strategy for dealing with those problems by building a local image with a Dockerfile
• Demonstrates the self-repair functions in action.

What's a non-well-behaved container?

This is best understood by considering well-behaved containers like:

• Node-RED
• InfluxDB
• Grafana
• Portainer-CE
• PiHole
• ...

You can reset any of those containers to "factory fresh" by taking them down, erasing their persistent storage area, and bringing them up again. For example, suppose you want to reset Node-RED:

$ cd ~/IOTstack
$ docker-compose down
$ sudo rm -rf ./volumes/nodered
$ docker-compose up -d

During the "up", docker-compose re-creates ./volumes/nodered/data, and then the container automatically re-creates all the necessary default structures. This is the end result:

/home/pi/IOTstack/volumes/nodered
└── data
    ├── lib
    │   └── flows
    ├── node_modules
    ├── package.json
    └── settings.js

Most importantly, Node-RED is ready for business. It displays its web GUI and lets you start creating flows.

The other containers in the well-behaved list have similar properties. Thus, a well-behaved container is one which:

1. Makes few assumptions about any pre-existing file system environment;
2. Takes responsibility for auto-repairing anything that is essential to its function; and
3. Avoids, as much as is possible, the "crash, burn and dump it in the user's lap" approach to dealing with problems.

A non-well-behaved container, therefore, is one which:

1. Expects the necessary directory and file structures to be in place before the container comes up;
2. Takes no responsibility for auto-repairing anything; and
3. Is just as likely to go into a restart loop when its environment is not exactly according to its needs.

The directoryfix.sh band-aid

IOTstack tries to deal with non-well-behaved containers through the directoryfix.sh mechanism (sometimes renamed build.sh in new menu). In fact, the presence of a directoryfix.sh in a container's template can be a strong indication that a non-well-behaved container lurks within.

The basic idea is that the menu system runs directoryfix.sh at build time. Note the emphasis.

The problem, then, is what happens if a non-well-behaved container's environment changes after directoryfix.sh has done its work? The answer is, "you get a mess".

Messes lead to unhappy users, issues and Discord questions. IOTstackers deserve better and the purpose of this issue is to explore how to turn non-well-behaved containers into well-behaved containers.

Users can run a container's directoryfix.sh manually but they have to:

• be aware of the script's existence;
• know when it is appropriate to run it; and
• run it properly.

Running directoryfix.sh properly means:

$ cd ~/IOTstack
$ ./.templates/«container»/directoryfix.sh

The approach many first-time users seem to take is:

$ cd ~/IOTstack/.templates/«container»
$ ./directoryfix.sh

That results in a cure-worse-than-the-disease kind of mess because directoryfix.sh scripts rely on relative paths and take their actions using sudo.

Any directoryfix.sh can always be improved but it would still suffer from the basic problem that it is only ever run by the menu or when the user "just knows" to run it by hand.

A case study: Mosquitto

Mosquitto is quite a good example of a non-well-behaved container.

I'm not trying to give the impression that there's anything wrong with either Mosquitto or the way the Eclipse-Mosquitto container is currently built. There isn't. I just don't think it goes quite far enough for our (IOTstack) needs.

Defining problems is the first step in solving problems so here is a list of specific problems, as I see them:

Multiple identity disorder

IOTstack's service.yml for Mosquitto defines:

user: "1883"

This causes docker-compose to spin up the container as userID 1883. Within the container, userID 1883 is the user "mosquitto".

The docker-entrypoint.sh script is how the container starts running:

#!/bin/ash
set -e

# Set permissions
user="$(id -u)"
if [ "$user" = '0' ]; then
	[ -d "/mosquitto" ] && chown -R mosquitto:mosquitto /mosquitto || true
fi

exec "$@"

It is not until after the exec statement that Mosquitto (the process) unconditionally downgrades its own privileges to userID 1883.

I think that the design intention of the docker-entrypoint.sh script is clear. The script expects the container to be launched as root so that it can ensure that the persistent storage directory and everything below it is owned by mosquitto:mosquitto (ie 1883:1883 outside the container) before the mosquitto process is launched and downgrades its privileges.

The presence of the user: "1883" directive in IOTstack's service.yml template prevents the self-repair code from ever being executed.

That's a problem, albeit one that is fairly easy to fix.

Too many cooks spoil the persistent-storage area

Pull Requests 274 and 275, changed Mosquitto's volumes definitions to:

- ./volumes/mosquitto/data:/mosquitto/data
- ./volumes/mosquitto/log:/mosquitto/log
- ./volumes/mosquitto/pwfile:/mosquitto/pwfile
- ./services/mosquitto:/mosquitto/config:ro

Previously, the 4^th line was split into two file paths, which caused problems of its own because missing files would be recreated as directories with root ownership.

The definitions can be split into two groups:

1. the volumes paths (the three subdirectories of ./volumes/mosquitto):

   If directoryfix.sh is run before the container is brought up then the volumes paths will be owned by userID 1883 (correct). In other words, directoryfix.sh is simply doing what docker-entrypoint.sh would do, were it not for the presence of the user: "1883" directive in docker-compose.yml.

   If directoryfix.sh is not run before the container is brought up and any of the volumes paths is not present, then the missing directories will be auto-created by docker-compose and will owned by root (incorrect). docker-entrypoint.sh would be able to fix this, were it not for the presence of the user: "1883" directive. The result is Mosquitto running inside the container having insufficient privileges and that usually leads to a restart loop.

2. the services path ./services/mosquitto:

   directoryfix.sh doesn't touch this so that script is not relevant to this part of the discussion.

   If the services path is present when the container is brought up then it will be owned by userID 1000 ("pi:pi") with mode 644, and will therefore be read-only to Mosquitto running inside the container. This is not a problem, per-se, because Mosquitto does not try to write to the services path or its contents.

   If the services path is absent when the container is brought up, docker-compose automatically creates the directory at the missing path, with root ownership. Again, this is not a problem, per-se, because Mosquitto does not try to write to the services path or its contents. It can, however, cause user confusion because of the subsequent need to use sudo when working with that directory.

   Irrespective of whether the services path is owned by "pi" or "root", Mosquitto expects it to contain:

   • mosquitto.conf (required)
   • filter.acl (required if filtering enabled)

   Mosquitto will go into a restart loop if:

   • mosquitto.conf is missing; or
   • mosquitto.conf is present and enables filtering but filter.acl is missing.

The pwfile chicken-and-egg

The default mosquitto.conf contains these two lines:

#password_file /mosquitto/pwfile/pwfile
allow_anonymous true

Of the four possible combinations of those two lines, the above is the only combination that actually works with an out-of-the-box installation by the IOTstack menu. If the user:

• enables the password_file line but doesn't provide a pwfile (even just a touch pwfile) then Mosquitto goes into a restart loop, This occurs irrespective of whether allow_anonymous is true or false.
• disables allow_anonymous (so it defaults to false) with password_file disabled, then Mosquitto will start but it won't accept any connections.

The user just has to "know" to create a password file before changing mosquitto.conf. This is somewhat counterintuitive and, judging by issues and Discord questions, is a frequent source of frustration and confusion.

The obvious solution is for "something" to ensure that there is at least an empty pwfile at the path:

~/IOTstack/volumes/mosquitto/pwfile/pwfile

At the moment, that "something" is limited to the menu or directoryfix.sh. But there is a better way. Keep reading.

Other directoryfix.sh quirks

At the moment, Mosquitto's directoryfix.sh:

1. Checks for the presence of:

   ./volumes/mosquitto
   

   If it is not present, it is created with root ownership.

2. The next step depends on the presence and contents of:

   ~/IOTstack/services/mosquitto/service.yml
   

   If that file:

   • is present and contains user: "1883" (even if the directive is commented-out) then directoryfix.sh ensures that the following directories exist:

      ./volumes/mosquitto/data/
      ./volumes/mosquitto/log/
      ./volumes/mosquitto/pwfile/
     

     and finishes off by unconditionally changing the ownership of the ./volumes/mosquitto hierarchy to 1883:1883 (correct).

   • is present and contains user: "0" (even if the directive is commented-out) then directoryfix.sh will unconditionally change the ownership of the ./volumes/mosquitto hierarchy to be root:root (incorrect).

     Note:

     • In this situation, directoryfix.sh does not ensure that data, log and pwfile exist.

   • otherwise, directoryfix.sh performs no additional actions.

What happens after directoryfix.sh finishes depends on what is in docker-compose.yml. Either:

• Mosquitto will launch as root and docker-entrypoint.sh will fix the ownership to 1883:1883; or
• the presence of a user: directive nominating a non-zero userID will cause the container to launch as that user, in which case docker-entrypoint.sh will do nothing and Mosquitto is then at risk of going into a restart loop because of permission conflicts.

On a standard IOTstack install, docker-compose.yml contains user: 1883 so the second option is the more likely.

Also, while it will generally be the case on a fresh install that the Mosquitto definition in service.yml will be the same as the definition in docker-compose.yml, there are no guarantees that it will stay that way. Nothing keeps the two files in sync with each other.

Summary

Drawing these threads together:

• The multiple identity disorder problem means:

  • the Mosquitto container only executes its self-repair code if it is launched as root.
  • IOTstack currently defaults to launching the container as userID 1883 so this code never executes.
  • Even if the user: "1883" was removed, the self-repair startup code limits itself to fixing ownership. It takes no responsibility for ensuring the presence of sensible defaults for mosquitto.conf, filter.acl or pwfile.

• The too many cooks problem means there is a wide variety of potential outcomes, many of which can lead to a restart loop, all depending on what existed before the container started, who owned what, when directoryfix.sh was last run, and whether Mosquitto's service.yml matches what is in docker-compose.yml.

• The pwfile chicken-and-egg can easily lead to either a restart loop or Mosquitto rejecting valid traffic.

• directoryfix.sh:

  • tries to do what the container's startup code would do if the container was launched as root but only if it has reason to believe that the container will be (future tense) started as userID 1883. This assumption generally holds but there are no guarantees.
  • doesn't actually do a thorough job of making sure that every piece of scaffolding Mosquitto needs in order to start properly is always present. The script could be improved – but it still …
  • … suffers from the limitation that there is no Docker mechanism to ensure that it is always run before Mosquitto is brought up.

Basically, there are a wide variety of situations where the Mosquitto container will go into a restart loop if the ground is not prepared properly before it is brought up. The whole thing is reminiscent of one of those semi-humorous flow charts that always wind up a few consonants shy of "bucked up!"

A Dockerfile solution

Turning non-well-behaved containers into well-behaved containers can be done with Dockerfiles.

The files described in the rest of this section are all located in:

~/IOTstack/.templates/mosquitto

The rationale for placing everything in .templates rather than services is that the former is the factory for all IOTstack implementations, while the latter is a customisation point for each user's implementation. This is a "factory" solution.

service.yml

  mosquitto:
    container_name: mosquitto
    build: ./.templates/mosquitto/.
    restart: unless-stopped
    ports:
      - "1883:1883"
    volumes:
      - ./volumes/mosquitto/config:/mosquitto/config
      - ./volumes/mosquitto/data:/mosquitto/data
      - ./volumes/mosquitto/log:/mosquitto/log
      - ./volumes/mosquitto/pwfile:/mosquitto/pwfile

Changes:

1. Rather than downloading the eclipse-mosquitto image from DockerHub, the service expects to find a Dockerfile located in .templates.
2. The user: directive is removed. This implies that docker-compose will launch the container as root, meaning that docker-entrypoint.sh will have the privileges it needs to perform auto-repair functions.

Dockerfile

# Download base image
FROM eclipse-mosquitto:latest

# Add support tools
RUN apk add --no-cache rsync tzdata

# where IOTstack template files are stored
ENV IOTSTACK_DEFAULTS_DIR="iotstack_defaults"

# copy template files to image
COPY --chown=mosquitto:mosquitto ${IOTSTACK_DEFAULTS_DIR} /${IOTSTACK_DEFAULTS_DIR}

# replace the docker entry-point script
ENV IOTSTACK_ENTRY_POINT="docker-entrypoint.sh"
COPY ${IOTSTACK_ENTRY_POINT} /${IOTSTACK_ENTRY_POINT}
RUN chmod 755 /${IOTSTACK_ENTRY_POINT}
ENV IOTSTACK_ENTRY_POINT=

# IOTstack also declares these paths
VOLUME ["/mosquitto/config", "/mosquitto/pwfile"]

# EOF

Actions:

1. Starts with the "eclipse-mosquitto:latest" image on Dockerhub.

2. Adds two packages to the container:

   • rsync is needed because the cp command provided with Alpine Linux does not implement the -n aka "no clobber" option which can be used to replace missing files while not overwriting existing files.
   • tzdata causes the container to respect the "TZ" environment variable and display log timestamps in local time.

3. Copies the contents of an iotstack_defaults directory structure into the image. The assumed structure is documented below. The file permissions are set in the template and persist into the image.

4. Replaces the existing docker-entrypoint.sh with a revised version.

5. The Dockerfile for the Mosquitto base image declares /mosquitto/data and /mosquitto/log. This declares the paths added for IOTstack and causes docker-compose to treat those paths identically to those declared in the Mosquitto base image.

iotstack_defaults

iotstack_defaults is a directory with the structure shown below. It is copied "as is" into the image by the Dockerfile.

iotstack_defaults (755)
├── config (755)
│   ├── filter.acl (644)
│   └── mosquitto.conf (644)
└── pwfile (755)
    └── pwfile (600)

This structure is inherently extensible. Any directory and/or file combination needed to establish sensible working defaults for Mosquitto can be added to the IOTstack repository and it will "just work".

iotstack_defaults/config/filter.acl

No change in content from the current template.

user admin
topic read #
topic write #

pattern read #
pattern write #

iotstack_defaults/config/mosquitto.conf

Content as per PRs 274 and 275:

# required by https://mosquitto.org/documentation/migrating-to-2-0/
#
listener 1883

# persistence enabled for remembering retain flag across restarts
#
persistence true
persistence_location /mosquitto/data

# logging options:
#   enable one of the following (stdout = less wear on SD cards but
#   logs do not persist across restarts)
#log_dest file /mosquitto/log/mosquitto.log
log_dest stdout
log_timestamp_format %Y-%m-%dT%H:%M:%S

# password handling:
#   password_file commented-out allow_anonymous true =
#     open access
#   password_file commented-out allow_anonymous false =
#     no access
#   password_file activated     allow_anonymous true =
#     passwords omitted is permitted but
#     passwords provided must match pwfile
#   password_file activated     allow_anonymous false =
#     no access without passwords
#     passwords provided must match pwfile
#
#password_file /mosquitto/pwfile/pwfile
allow_anonymous true

# Uncomment to enable filters
#acl_file /mosquitto/config/filter.acl

iotstack_defaults/pwfile/pwfile

An empty file. Its presence (even if empty) means that the user can utilise all four combinations of password_file and allow_anonymous, before defining any passwords, without any risk of creating a restart loop.

docker-entrypoint.sh

The original was documented above. This is my proposed version:

#!/bin/ash
set -e

# Set permissions
user="$(id -u)"
if [ "$user" = '0' -a -d "/mosquitto" ]; then

   rsync -ar --ignore-existing /${IOTSTACK_DEFAULTS_DIR}/ "/mosquitto"

   chown -R mosquitto:mosquitto /mosquitto
   
fi

exec "$@"

The changes are:

1. Group the "are we running as root?" and "does /mosquitto exist?" tests into a single statement. In practice, docker-compose guarantees the presence of /mosquitto so whether the container was launched as root is the only material consideration. The omission of the user: "1883" directive from service.yml (and, by inference, docker-compose.yml) means that the "if" test will always succeed in the IOTstack environment.
2. The addition of the rsync command. This performs all auto-repair each time the container is launched. Any directory or file that has gone missing will be recreated automatically, with the correct permissions. The --ignore-existing guarantees that nothing will be overwritten.

Correct 1883:1883 ownership is enforced throughout the /mosquitto hierarchy by the chown statement.

As mentioned before, the exec launches Mosquitto (the process) over the top of docker-entrypoint.sh and one of its first actions is to change its privileges to userID 1883. So, while the container starts as root, it runs as userID 1883, exactly as before.

directoryfix.sh

Changed to be a single-use migration tool. If the user performs a git pull and then runs the menu, the menu will run directoryfix.sh. At that point, the expected situation is:

./volumes/mosquitto/config

will not exist, but these files will exist:

./services/mosquitto/mosquitto.conf
./services/mosquitto/filter.acl

Given that situation, the correct migration behaviour is to create the config directory and move the existing mosquitto.conf and filter.acl files into that directory, thus preserving any existing configuration.

#!/bin/bash

# should not run as root
[ "$EUID" -eq 0 ] && echo "This script should NOT be run using sudo" && exit -1

# expects to run from IOTstack
[ $(basename "$PWD") = "IOTstack" ] || echo -e \
"Warning: This script expects to be run from ~/IOTstack.\n" \
"        The script will continue but may produce unexpected results."

# define paths
CONFIG_OLD=./services/mosquitto
CONFIG_NEW=./volumes/mosquitto/config
MOSQUITTO_CONFIG=mosquitto.conf
MOSQUITTO_FILTER=filter.acl

# this may be running before Mosquitto has come up for the first time so
# make sure the directory exists
[ -d $CONFIG_NEW ] || sudo mkdir -p $CONFIG_NEW

# migrate config file if possible
if [ -e $CONFIG_OLD/$MOSQUITTO_CONFIG -a ! -e $CONFIG_NEW/$MOSQUITTO_CONFIG ] ; then
   echo "Note: migrating $CONFIG_OLD/$MOSQUITTO_CONFIG to $CONFIG_NEW/$MOSQUITTO_CONFIG"
   sudo mv $CONFIG_OLD/$MOSQUITTO_CONFIG $CONFIG_NEW/$MOSQUITTO_CONFIG
fi

# migrate filter.acl file if possible
if [ -e $CONFIG_OLD/$MOSQUITTO_FILTER -a ! -e $CONFIG_NEW/$MOSQUITTO_FILTER ] ; then
   echo "Note: migrating $CONFIG_OLD/$MOSQUITTO_FILTER to $CONFIG_NEW/$MOSQUITTO_FILTER"
   sudo mv $CONFIG_OLD/$MOSQUITTO_FILTER $CONFIG_NEW/$MOSQUITTO_FILTER
fi

# note: no "chown" performed - that is left to Mosquitto when it launches

directoryfix.sh will not overwrite mosquitto.conf and/or filter.acl if those already exist in config. In that situation, both source and target files are preserved to make it easy for the user to compare and cherry-pick.

Nothing happens if the user does a git pull but then does not do anything to cause the new service definition to be put into production. The old service definition (with all its problems) and the directories and files to which it refers, will continue to work.

Tests

Simulate first run of the new service definition and Dockerfile structure

The starting position is:

1. Mosquitto not running:

   $ docker ps --format "table {{.Names}}\t{{.RunningFor}}\t{{.Status}}" --filter "name=mosquitto"
   NAMES     CREATED   STATUS
   

2. Persistent storage area erased:

   $ sudo rm -rf ./volumes/mosquitto
   

3. Add markers added to mosquitto.conf and filter.acl in ./services/mosquitto:

   $ sed -i '1i# WAS IN SERVICES' ./services/mosquitto/mosquitto.conf ./services/mosquitto/filter.acl
   
   $ head -1 ./services/mosquitto/mosquitto.conf ./services/mosquitto/filter.acl
   ==> ./services/mosquitto/mosquitto.conf <==
   # WAS IN SERVICES
   
   ==> ./services/mosquitto/filter.acl <==
   # WAS IN SERVICES
   

4. Simulate running of directoryfix.sh by the menu:

   $ cd ~/IOTstack
   $ ./.templates/mosquitto/directoryfix.sh 
   Note: migrating ./services/mosquitto/mosquitto.conf to ./volumes/mosquitto/config/mosquitto.conf
   Note: migrating ./services/mosquitto/filter.acl to ./volumes/mosquitto/config/filter.acl
   

5. Show effect of running directoryfix.sh:

   $ [ -d ./volumes/mosquitto ] && tree -pu ./volumes/mosquitto
   ./volumes/mosquitto
   └── [drwxr-xr-x root    ]  config
       ├── [-rw-r--r-- pi      ]  filter.acl
       └── [-rw-r--r-- pi      ]  mosquitto.conf
   
   1 directory, 2 files
   

Mosquitto is now in "post-menu" state. Note the mixture of "pi" and "root" ownerships.

Show that the container starts correctly - post migration

Bring up Mosquitto:

$ docker-compose up -d mosquitto
Creating mosquitto ... done

Show Mosquitto is running:

$ docker ps --format "table {{.Names}}\t{{.RunningFor}}\t{{.Status}}" --filter "name=mosquitto"
NAMES       CREATED         STATUS
mosquitto   35 seconds ago   Up 34 seconds

Show the combined effect of the earlier directoryfix.sh and launching the container:

$ [ -d ./volumes/mosquitto ] && tree -pu ./volumes/mosquitto
./volumes/mosquitto
├── [drwxr-xr-x 1883    ]  config
│   ├── [-rw-r--r-- 1883    ]  filter.acl
│   └── [-rw-r--r-- 1883    ]  mosquitto.conf
├── [drwxr-xr-x 1883    ]  data
├── [drwxr-xr-x 1883    ]  log
└── [drwxr-xr-x 1883    ]  pwfile
    └── [-rw------- 1883    ]  pwfile

4 directories, 3 files

Note:

• Directory structure is owned by 1883:1883. This is enforced, automatically, by docker-entrypoint.sh each time Mosquitto launches or restarts. There is no need for directoryfix.sh to make this happen.

Confirm that the migrated mosquitto.conf and filter.acl were not overwritten by the self-repair process when the container launched:

$ head -1 ./volumes/mosquitto/config/mosquitto.conf ./volumes/mosquitto/config/filter.acl
==> ./volumes/mosquitto/config/mosquitto.conf <==
# WAS IN SERVICES

==> ./volumes/mosquitto/config/filter.acl <==
# WAS IN SERVICES

The markers are still in place so those are the files that came from ~IOTstack/services. Take the container down:

$ docker-compose stop mosquitto; docker-compose rm -f mosquitto
Stopping mosquitto ... done
Going to remove mosquitto
Removing mosquitto ... done

Create a "clean slate"

Erase the persistent storage area:

$ sudo rm -rf ./volumes/mosquitto

Mosquitto is now in "never run" state.

Bring the container up:

$ docker-compose up -d mosquitto
Creating mosquitto ... done

Show Mosquitto is running:

$ docker ps --format "table {{.Names}}\t{{.RunningFor}}\t{{.Status}}" --filter "name=mosquitto"
NAMES       CREATED         STATUS
mosquitto   25 seconds ago   Up 23 seconds

Show that the persistent storage area has been re-initialised correctly:

$ [ -d ./volumes/mosquitto ] && tree -pu ./volumes/mosquitto
./volumes/mosquitto
├── [drwxr-xr-x 1883    ]  config
│   ├── [-rw-r--r-- 1883    ]  filter.acl
│   └── [-rw-r--r-- 1883    ]  mosquitto.conf
├── [drwxr-xr-x 1883    ]  data
├── [drwxr-xr-x 1883    ]  log
└── [drwxr-xr-x 1883    ]  pwfile
    └── [-rw------- 1883    ]  pwfile

4 directories, 3 files

Confirm that mosquitto.conf and filter.acl are no longer the migrated versions:

$ head -1 ./volumes/mosquitto/config/mosquitto.conf ./volumes/mosquitto/config/filter.acl
==> ./volumes/mosquitto/config/mosquitto.conf <==
listener 1883

==> ./volumes/mosquitto/config/filter.acl <==
user admin

Restart Mosquitto (which causes Mosquitto to flush its internal database to disk):

$ docker-compose restart mosquitto
Restarting mosquitto ... done

Confirm that the database has been saved:

$ [ -d ./volumes/mosquitto ] && tree -pu ./volumes/mosquitto
./volumes/mosquitto
├── [drwxr-xr-x 1883    ]  config
│   ├── [-rw-r--r-- 1883    ]  filter.acl
│   └── [-rw-r--r-- 1883    ]  mosquitto.conf
├── [drwxr-xr-x 1883    ]  data
│   └── [-rw------- 1883    ]  mosquitto.db
├── [drwxr-xr-x 1883    ]  log
└── [drwxr-xr-x 1883    ]  pwfile
    └── [-rw------- 1883    ]  pwfile

4 directories, 4 files

Turn on password handling

Enable the pwfile in the config:

$ sudo sed -i \
  's/^#password_file/password_file/' \
  ./volumes/mosquitto/config/mosquitto.conf
 
$ grep "^password_file" ./volumes/mosquitto/config/mosquitto.conf
password_file /mosquitto/pwfile/pwfile

Restart the container, wait 10 seconds, show container is running:

$ docker-compose restart mosquitto; sleep 10; docker ps --format "table {{.Names}}\t{{.RunningFor}}\t{{.Status}}" --filter "name=mosquitto"
Restarting mosquitto ... done

NAMES       CREATED          STATUS
mosquitto   16 minutes ago   Up 10 seconds

Note:

• Previously, this would have resulted in Mosquitto going into a restart loop. The Dockerfile process has ensured that an empty password file is present. This keeps Mosquitto happy.

Show that a password can be created:

$ docker exec mosquitto mosquitto_passwd -b /mosquitto/pwfile/pwfile mossieUser mossiePassword

$ sudo cat ./volumes/mosquitto/pwfile/pwfile 
mossieUser:$7$101$v5MZODc42FvFusUc$9Lbqu2USfxKKQLuidREJwTCXCDyPw0P3KzzFBsChSykYXQIHQj9DSgSaSmiZAjtNT0xGe83cA1JjJmCnlc355w==

Notes:

• Previously, the user either had to create a null password file before running this command, or pass the -c option to tell Mosquitto to create the file. Now, there is no difference between how the first password is created, and how second and subsequent passwords are created.

• Guaranteeing the presence of a pwfile, even if it is an empty file, means Mosquitto will always start properly, irrespective of whether allow_anonymous is true or false. If allow_anonymous is:

  • true and credentials are:

    • provided on an MQTT request, the credentials will be verified
    • not provided, the anonymous MQTT request will be permitted

  • false and credentials are:

    • provided on an MQTT request, the credentials will be verified
    • not provided, the anonymous MQTT request will be rejected

Turn on logging to disk

Enable logging to disk in the config:

$ sudo sed -i \
  -e 's/^#log_dest file/log_dest file/' \
  -e 's/^log_dest stdout/#log_dest stdout/' \
  ./volumes/mosquitto/config/mosquitto.conf

$ grep "^log_dest" ./volumes/mosquitto/config/mosquitto.conf
log_dest file /mosquitto/log/mosquitto.log

$ docker-compose restart mosquitto; sleep 10; docker ps --format "table {{.Names}}\t{{.RunningFor}}\t{{.Status}}" --filter "name=mosquitto"
Restarting mosquitto ... done

NAMES       CREATED          STATUS
mosquitto   20 minutes ago   Up 10 seconds

Show that the log is now being written:

$ [ -d ./volumes/mosquitto ] && tree -pu ./volumes/mosquitto
./volumes/mosquitto
├── [drwxr-xr-x 1883    ]  config
│   ├── [-rw-r--r-- 1883    ]  filter.acl
│   └── [-rw-r--r-- 1883    ]  mosquitto.conf
├── [drwxr-xr-x 1883    ]  data
│   └── [-rw------- 1883    ]  mosquitto.db
├── [drwxr-xr-x 1883    ]  log
│   └── [-rw------- 1883    ]  mosquitto.log
└── [drwxr-xr-x 1883    ]  pwfile
    └── [-rw------- 1883    ]  pwfile

4 directories, 5 files

End of tests.

Wrapping it up

The "multiple identity disorder" goes away if the container is allowed to start as root. That gives it sufficient privileges for the self-repair code to run before the Mosquitto process downgrades its privileges.

The "too many cooks spoil the persistent-storage area" goes away if all the paths are under ~/IOTstack/volumes/mosquitto and all are declared, properly, during the Dockerfile run, so that Docker handles them consistently.

The "pwfile chicken-and-egg" problem goes away if the self-repair code is permitted to guarantee that the pwfile is always present.

If there is no need for a directoryfix.sh performing repair functions (as distinct from a one-time migration function), there can be no "quirks" and no need for users to remember when or how to run it. Even if directoryfix.sh doesn't run when it should in its role as a migration aid, the container will still start and won't go into a restart loop.

The next step

I've been running Mosquitto like this for the last few months, on three RPi4s and one 3B+. It just works. In terms of getting updates, it's more like Node-RED and requires a conscious decision:

$ cd ~/IOTstack
$ docker-compose build --no-cache --pull mosquitto
$ docker-compose up -d mosquitto
$ docker system prune
$ docker system prune

Given the chaos that was caused when Mosquitto was updated to require the listener and allow_anonymous directives, I'd say a bit more friction in the update process was no bad thing.

The next step is a Pull Request to implement these changes for Mosquitto.

In the meantime, I'd appreciate feedback on this proposal.