File and folder permissions are screwed when executed via Jobber

[ckotte]

If I run my backup scripts via Jobber, the permissions of created files and folders is not correct. It looks like only rw for the owner is set - also for directories.

I can still schedule my jobs via root, but they fail if I execute the job with a normal user account because of a permission issue related to the missing x for the directory.

The jobs create a database dump to a directory with pg_dump and zip it via gpg-zip:

Script executed without Jobber
directory: drwx------ 2 root root 4096 Apr 24 19:06 name-db
file: -rw-r--r-- 1 root root 85497987 Apr 24 19:06 name-20180424-190638.tar.gz.gpg

Script executed via Jobber
directory: drw------- 2 root root 4096 Apr 24 19:04 name-db
file: -rw------- 1 root root 85498005 Apr 24 19:04 name-20180424-190424.tar.gz.gpg

I use Jobber 1.3.2 in a custom Docker container with Alpine Linux 3.7. This doesn't happen with the same Dockerfile and Jobber 1.2 with exec jobberd instead of exec /usr/local/libexec/jobberrunner ${configfile}.

[dshearer]

That's interesting... It looks like there's a difference in the umask.

I'll look into this. In the meantime, you should be able to get around this by explicitly settings the permissions in your script.

[ckotte]

Thanks for having a look into this.

The command that fails is:
pg_dump -U username -h postgres --port=5432 -j 5 -Fd database --no-synchronized-snapshots -f /directory

-Fd dumps the database into a directory that's specified via -f and that directory will be created by pg_dump and needs to be deleted before every execution.

The only workaround is to execute the jobs via root since I can't change the permissions while pg_dump creates the directory and fails right away with a permission denied error message..

[zelenin]

jobber/jobberrunner/main.go

Line 170 in d7276b2

syscall.Umask(0177)

's/0177/0077/'

[ckotte]

thx. this is exactly the same issue: Where to set default file permissions (umask)?(post no. 9). Looks like 077 would do the trick or what is 's/0177/0077/'?

[dshearer]

Thanks, @zelenin. I'm not sure what I was thinking when I set that umask...

[ckotte]

Could you please implement this little fix? I would like to run my jobs via a normal user and not root. Thanks in advance.

[dshearer]

Sure.

[zelenin]

maybe new release?

[dshearer]

Done. Release has been made. v1.3.3.

[ckotte]

Could you update the documentation or share your Dockerfile. Looks like you've introduced new parameter since 1.3.2. I now get the following error if I use exec /usr/local/libexec/jobberrunner ${configfile} in my entrypoint script:

Must specify exactly one of -u and -p

Usage: /usr/local/libexec/jobberrunner [flags] JOBFILE_PATH

Flags:

  -h	help
  -p uint
    	path TCP socket on which to receive commands
  -q string
    	quit socket path
  -u string
    	path to Unix socket on which to receive commands
  -v	version

If I use exec /usr/local/libexec/jobberrunner -u /tmp/jobber_socket ${configfile}, I get this error:

bash-4.4$ jobber list
Jobber doesn't seem to be running for user jobberuser.
(No socket at /var/jobber/1000/cmd.sock.): stat /var/jobber/1000/cmd.sock: no such file or directory

or

bash-4.4$ jobber list
Jobber doesn't seem to be running for user root.
(No socket at /var/jobber/0/cmd.sock.): stat /var/jobber/0/cmd.sock: no such file or directory

It seems to work if I use exec /usr/local/libexec/jobberrunner -u /var/jobber/${EUID}/cmd.sock ${configfile}

[dshearer]

Ah crap.

…

On Aug 28, 2018, at 9:16 AM, Christian Kotte ***@***.***> wrote: Could you update the documentation or share your Dockerfile. Looks like you've introduced new parameter since 1.3.2. I now get the following error if I use exec /usr/local/libexec/jobberrunner ${configfile} in my entrypoint script: Must specify exactly one of -u and -p Usage: /usr/local/libexec/jobberrunner [flags] JOBFILE_PATH Flags: -h help -p uint path TCP socket on which to receive commands -q string quit socket path -u string path to Unix socket on which to receive commands -v version If I use exec /usr/local/libexec/jobberrunner -u /tmp/jobber_socket ${configfile}, I get this error: bash-4.4$ jobber list Jobber doesn't seem to be running for user jobberuser. (No socket at /var/jobber/1000/cmd.sock.): stat /var/jobber/1000/cmd.sock: no such file or directory or bash-4.4$ jobber list Jobber doesn't seem to be running for user root. (No socket at /var/jobber/0/cmd.sock.): stat /var/jobber/0/cmd.sock: no such file or directory It seems to work if I use exec /usr/local/libexec/jobberrunner -u /var/jobber/${EUID}/cmd.sock ${configfile} — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[ckotte]

btw. I get

bash-4.4$ jobber -v
1.3.2

when compiling 1.3.3.

This is the output during the build:

package github.com/dshearer/jobber: no Go files in /tmp/jobber/lib/src/github.com/dshearer/jobber
Note: checking out 'tags/v1.3.3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 543d598 circleci
make: Entering directory '/tmp/jobber/lib/src/github.com/dshearer/jobber'
MAKE GOYACC
# github.com/dshearer/jobber/buildtools/gen/gowkspc/src/golang.org/x/tools/cmd/goyacc
loadinternal: cannot find runtime/cgo
GEN SRC
1.10.1 is greater than or equal to 1.8
BUILD jobbermaster
INSTALL /usr/local/libexec/jobbermaster
1.10.1 is greater than or equal to 1.8
BUILD jobberrunner
INSTALL /usr/local/libexec/jobberrunner
1.10.1 is greater than or equal to 1.8
BUILD jobber
INSTALL /usr/local/bin/jobber
INSTALL /etc/jobber.conf
make: Leaving directory '/tmp/jobber/lib/src/github.com/dshearer/jobber'

This is the Dockerfile:

FROM blacklabelops/alpine:3.8
MAINTAINER Steffen Bleul <sbl@blacklabelops.com>

# build parameters
ARG JOBBER_VERSION=1.3.3
# Image Build Date By Buildsystem
ARG BUILD_DATE=undefined

RUN export JOBBER_HOME=/tmp/jobber && \
    export JOBBER_LIB=$JOBBER_HOME/lib && \
    export GOPATH=$JOBBER_LIB && \
    export CONTAINER_UID=1000 && \
    export CONTAINER_GID=1000 && \
    export CONTAINER_USER=jobberuser && \
    export CONTAINER_GROUP=jobberuser && \
    # Install tools
    apk add --update --no-cache --virtual .build-deps \
      go \
      git \
      curl \
      wget \
      tzdata \
      make \
      musl-dev \
      rsync \
      grep && \
    mkdir -p $JOBBER_HOME && \
    mkdir -p $JOBBER_LIB && \
    # Install Jobber
    addgroup -g $CONTAINER_GID $CONTAINER_USER && \
    adduser -u $CONTAINER_UID -G $CONTAINER_GROUP -s /bin/bash -S $CONTAINER_USER && \
    mkdir -p "/var/jobber/${CONTAINER_UID}" && chown -R $CONTAINER_UID:$CONTAINER_GID "/var/jobber/${CONTAINER_UID}" && \
    mkdir -p "/var/jobber/0" && \
    cd $JOBBER_LIB && \
    go get github.com/dshearer/jobber;true && \
    # can't build container with jobber Alpine Linux package because it depends on openrc
    # wget --directory-prefix=/tmp https://github.com/dshearer/jobber/releases/download/v${JOBBER_VERSION}/jobber-${JOBBER_VERSION}-r0_alpine3.6and3.7.apk && \
    # apk add --allow-untrusted /tmp/jobber-${JOBBER_VERSION}-r0_alpine3.6and3.7.apk ; \
    cd src/github.com/dshearer/jobber && \
    git checkout tags/v${JOBBER_VERSION} && \
    cd $JOBBER_LIB ; \
    make -C src/github.com/dshearer/jobber install && \
    # Cleanup
    apk del .build-deps && \
    rm -rf /var/cache/apk/* && rm -rf /tmp/* && rm -rf /var/log/*

# Image Metadata
LABEL com.blacklabelops.application.jobber.version=$JOBBER_VERSION \
      com.blacklabelops.image.builddate.jobber=${BUILD_DATE}

COPY docker-entrypoint.sh /opt/jobber/docker-entrypoint.sh
ENTRYPOINT ["/sbin/tini","--","/opt/jobber/docker-entrypoint.sh"]
CMD ["jobberd"]

The issue is also not fixed with this version. Do I use 1.3.2 or 1.3.3 with wrong version output? Could you share your Dockerfile please.

[dshearer]

It looks like your Dockerfile is checking out the wrong branch... although I don't immediately see the problem in the Dockerfile.

In any case, please use dshearer/jobber:1-alpine3.7 from now on. In this image, replace /home/jobberuser/.jobber with your own jobfile.

[ckotte]

Please compile the sources. jobber -v shows 1.3.2 if I compile it with master branch or if I check out tag v1.3.3.
I also used the commands you provided in the documentation with the same result:

cd /path/to/your/workspace
mkdir -p src/github.com/dshearer
cd src/github.com/dshearer
git clone https://github.com/dshearer/jobber.git
cd jobber
git checkout v1.3.3
make check
sudo make install DESTDIR=/

[ckotte]

I also cannot use your Docker image. If it shows 1.3.3 for you, can you show how you install Jobber in Docker, please?

[dshearer]

Oh I see what happened. When I made the v1.3.3 tag, I based it on master rather than the maint-1.3 branch. Sorry about that. It is fixed now.

[ckotte]

Ok. I can now checkout tag v1.3.3 and jobber shows 1.3.3. However, the issue isn't fixed. It's only fixed if I remove the umask entirely.

I didn't test my early recommendation of umask 077 though, but I think it's better to let the users decide which umask they want to use.

The default umask on e.g. Alpine is 022 configured in /etc/profile. It could be changed to 077 to reflect the same behavior. If I just add umask 077 to my Docker entrypoint script, then the file permissions are set to 600 instead of 644. I bet it's the same if you replace the umask command in /etc/profile.

It's also not documented that the umask is set via Jobber and it's misleading if you compare the output of a script that was triggered by jobber with the output of a script that was run via bash.

[dshearer]

Sounds like I should just stop setting the umask.

[dshearer]

It looks like cron does not set the umask... I'm not sure where it comes from in cron. For now, I will set the umask to 0002.

The user can control the umask by setting it in the script that jobber executes. E.g.:

umask 0002 && pg_dump -U username -h postgres --port=5432 -j 5 -Fd database --no-synchronized-snapshots -f /directory