Mergerfs service can't start

[roubachof]

Since 2 days mergerfs won't start.
When I run rclean I got:

Job for mergerfs.service failed because the control process exited with error code.
See "systemctl status mergerfs.service" and "journalctl -xeu mergerfs.service" for details.
Waiting on mounts to be created...
Waiting on mounts to be created...
Waiting on mounts to be created...

The status command give me:

× mergerfs.service - MergerFS overlay writable storage and local mount
     Loaded: loaded (/etc/systemd/system/mergerfs.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-04-21 12:57:00 UTC; 53s ago
    Process: 2731 ExecStartPre=/bin/mkdir -p ${UPLOADS} ${MEDIA} ${LOCALFILES} (code=exited, status=0/S>
    Process: 2732 ExecStart=/usr/bin/mergerfs -o use_ino,hard_remove,auto_cache,sync_read,allow_other,c>
        CPU: 18ms

The journal gives me:

Apr 21 13:19:51 vps-f078333b mergerfs[25270]: fusermount: mount failed: Operation not permitted
Apr 21 13:19:51 vps-f078333b mergerfs[25276]: fusermount: mount failed: Operation not permitted
Apr 21 13:19:51 vps-f078333b mergerfs[25299]: fusermount: mount failed: Operation not permitted
Apr 21 13:19:52 vps-f078333b mergerfs[25310]: fusermount: mount failed: Operation not permitted
Apr 21 13:19:52 vps-f078333b mergerfs[25340]: fusermount: mount failed: Operation not permitted
Apr 21 13:23:49 vps-f078333b mergerfs[28716]: fusermount: mount failed: Operation not permitted
Apr 21 13:23:49 vps-f078333b mergerfs[28728]: fusermount: mount failed: Operation not permitted
Apr 21 13:23:50 vps-f078333b mergerfs[28733]: fusermount: mount failed: Operation not permitted
Apr 21 13:23:50 vps-f078333b mergerfs[28737]: fusermount: mount failed: Operation not permitted
Apr 21 13:23:50 vps-f078333b mergerfs[28742]: fusermount: mount failed: Operation not permitted

###########################
#  Environment variables  #
#     for Gooby v2.3.0    #
###########################

# Config variables
CONFIGS=/var/local/Gooby
CONFIGVARS=/var/local/Gooby/Docker/.config

# Server settings
MYDOMAIN=xxxx
MYEMAIL=xxxx
IP=xxx
SERVER=xxx
TIMEZONE=Etc/UTC

# User settings
GROUPID=1001
GROUPNAME=roubachof
USERID=1001
USERNAME=roubachof

# App settings
CF_EMAIL=
CF_KEY=
GPUGIDS=44
PLEXCLAIM=claim-xxxx

# Gooby settings
APILOG=/home/roubachof/logs/api.log
CURDIR=/var/local/Gooby/Docker
GOOBYBRANCH=v2
LOGS=/home/roubachof/logs
PROXYVERSION=nginx
VERSION=2.3.0

# Rclone settings
RCLONEHOME=/home/roubachof/.config/rclone
RCLONESERVICE=gdrive
RCLONEFOLDER=
RCLONEUSERNAME=gooby
RCLONEPASSWORD=Go0by

# Mounts and locations
HOMEDIR=/home/roubachof
LOCALFILES=/mnt/local
MEDIA=/mnt/google
RCLONEMOUNT=/mnt/rclone
ROOTMOUNT=/mnt
UPLOADS=/mnt/uploads
DOWNLOADS=/home/roubachof/Downloads

# Legacy variables
GOOGLE=/mnt/google

[TechPerplexed]

What happens if you try

sudo systemctl daemon-reload
sudo systemctl stop mergerfs
sudo systemctl stop rclonefs
/bin/fusermount -uz /mnt/rclone
/bin/fusermount -uz /mnt/google

Then run rclean. This will ensure at least there aren't any mounts hanging up the system :)

Edit: before running rclean, make sure to check if your mount points are all gone in /mnt as well.

[roubachof]

> ls /mnt
local  uploads
> rclean
...
Cleaning mount leftovers

mountpoint: /mnt/rclone: No such file or directory
mountpoint: /mnt/google: No such file or directory

Calling on Upgrade Service

Your system is on v2.3.0 (final version)

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    14  100    14    0     0    114      0 --:--:-- --:--:-- --:--:--   115
done
Job for mergerfs.service failed because the control process exited with error code.
See "systemctl status mergerfs.service" and "journalctl -xeu mergerfs.service" for details.
Waiting on mounts to be created...
Waiting on mounts to be created...

Still the same error...

Apr 21 17:24:18 vps-f078333b mergerfs[5541]: fusermount: mount failed: Operation not permitted
Apr 21 17:24:18 vps-f078333b mergerfs[5548]: fusermount: mount failed: Operation not permitted
Apr 21 17:24:19 vps-f078333b mergerfs[5552]: fusermount: mount failed: Operation not permitted
Apr 21 17:24:19 vps-f078333b mergerfs[5556]: fusermount: mount failed: Operation not permitted
Apr 21 17:24:19 vps-f078333b mergerfs[5560]: fusermount: mount failed: Operation not permitted

[TechPerplexed]

It almost sounds like a permission error. In your mergerfs.service file, could you comment out the lines
User=youruser and Group=youruser? If you need more pointers on how to, let me know. I'm assuming you are familiar with it :)

[roubachof]

well, everything is working now...
Should I let my user commented or should we dig some more to understand what went wrong ?

[roubachof]

maybe I should add the user to certain group, like admin or sudo ?
Anyway, thank you for your fantastic work, I am myself a open source maintainer, and your issue management is flawless :)

[TechPerplexed]

Sorry for the very late reply here 😄 Since it's working well now, I suggest we just chalk it up to "fluke" 🤣 I'll close this now, thanks for your kind words!