Files are made group readable on the data volume

[daniel-muhra-cw]

Describe the issue

Problem running 'git remote update gerrit'
Fetching gerrit
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0660 for '/app/data/ssh/id_ed25519' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/app/data/ssh/id_ed25519": bad permissions
weblate@gerrit.caperwhite.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error: could not fetch gerrit
Errors running git rebase --abort
fatal: No rebase in progress?
(1)

I already tried

• I've read and searched the documentation.
• I've searched for similar filed issues in this repository.

Steps to reproduce the behavior

• After creating an SSH key, the first time I tried to access my private git repo (hosted using gerrit) and was confronted with this issue
• Also I recently updated my Weblate instance (5.5.5.1) and was confronted with this error again when trying to push a commit in Weblate.
• I'm running my Weblate instance in Kubernetes using the latest helm chart (0.5.9)
• Changing the access rights in the pod directly fixed the issue (again)

Expected behavior

• This issue should not occur
• The private key file should always have the correct file permissions set.

Screenshots

Exception traceback

No response

How do you run Weblate?

Docker container

Weblate versions

• Weblate: 5.5.5
• Django: 5.0.6
• siphashc: 2.4.1
• translate-toolkit: 3.13.0
• lxml: 5.2.2
• pillow: 10.3.0
• nh3: 0.2.17
• python-dateutil: 2.9.0.post0
• social-auth-core: 4.5.4
• social-auth-app-django: 5.4.1
• django-crispy-forms: 2.1
• oauthlib: 3.2.2
• django-compressor: 4.4
• djangorestframework: 3.15.1
• django-filter: 24.2
• django-appconf: 1.0.6
• user-agents: 2.2.0
• filelock: 3.14.0
• rapidfuzz: 3.9.0
• openpyxl: 3.1.2
• celery: 5.4.0
• django-celery-beat: 2.6.0
• kombu: 5.3.7
• translation-finder: 2.16
• weblate-language-data: 2024.5
• html2text: 2024.2.26
• pycairo: 1.26.0
• PyGObject: 3.48.2
• diff-match-patch: 20230430
• requests: 2.31.0
• django-redis: 5.4.0
• hiredis: 2.3.2
• sentry-sdk: 1.45.0
• Cython: 3.0.10
• misaka: 2.1.1
• GitPython: 3.1.43
• borgbackup: 1.2.8
• pyparsing: 3.1.2
• ahocorasick_rs: 0.22.0
• python-redis-lock: 4.0.0
• charset-normalizer: 3.3.2
• Python: 3.12.3
• Git: 2.39.2
• psycopg: 3.1.19
• psycopg-binary: 3.1.19
• phply: 1.2.6
• ruamel.yaml: 0.18.6
• tesserocr: 2.7.0
• boto3: 1.34.104
• zeep: 4.2.1
• aeidon: 1.14.1
• iniparse: 0.5
• mysqlclient: 2.2.4
• Mercurial: 6.7.3
• git-svn: 2.39.2
• git-review: 2.4.0
• PostgreSQL server: 15.4
• Database backends: django.db.backends.postgresql
• PostgreSQL implementation: psycopg3 (binary)
• Cache backends: default:RedisCache, avatar:FileBasedCache
• Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.sendgrid.net
• OS encoding: filesystem=utf-8, default=utf-8
• Celery: redis://:weblate@weblate-redis-master:6379/1, redis://:weblate@weblate-redis-master:6379/1, regular
• Platform: Linux 5.13.0-52-generic (x86_64)

Weblate deploy checks

System check identified some issues:

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-5.5.5/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-5.5.5/admin/backup.html

System check identified 5 issues (1 silenced).

Additional context

No response

[nijel]

I've just looked at a few containers and the private key as owner only access in all of them. Didn't you copy the keys from another system, not preserving the permissions? Or maybe the file system you use doesn't preserve permissions?

Weblate internally uses ssh-keygen which creates keys with secure permissions.

[github-actions]

This issue has been marked as a question by a Weblate team member. Why? Because it belongs more to the professional Weblate Care or community Discussions than here. We strive to answer these reasonably fast here, too, but purchasing the support subscription is more responsible and faster for your business. And it makes Weblate stronger as well. Thanks!

In case your question is already answered, making a donation is the right way to say thank you!

[daniel-muhra-cw]

I don't think it's the underlying filesystem. The persistent volume is backed by a local zfs volume and other files seem to retain their permissions.

Also we had to restart the pod (i.e. actually the whole node) and after the restart the file permissions for that file were messed up again.

Also some time ago we replaced the ssh key with the a new one (different cipher). The old key remains on the volume and was not affected by the restart.

Aside from ssh, which processes interact with the key file?

[nijel]

There should be nothing else interacting with the file.

[daniel-muhra-cw]

Ok, I just checked again:

• Restarting the pod resets the file permissions for that key file always to 660
• Other files don't seem to be affected

[nijel]

#307 / #293 / 01f5fcb added fix up for id_rsa, but it really doesn't explain why it is needed.

Can you try creating another file in the volume with 0600 permission to see if other files are affected as well?

[github-actions]

This issue has been automatically marked as stale because there wasn’t any recent activity.

It will be closed soon if no further action occurs.

Thank you for your contributions!

[daniel-muhra-cw]

I created a file id_xyz and change its permissions to 600. After restarting the pod, it looks like this:

-rw-rw-r-- 1 weblate weblate 58 Mar 2 00:56 config
-rw-rw---- 1 weblate weblate 399 Mar 2 00:46 id_ed25519
-rw-rw-r-- 1 weblate weblate 89 Mar 2 00:46 id_ed25519.pub
-rw------- 1 weblate weblate 3369 Mar 2 00:36 id_rsa
-rw-rw-r-- 1 weblate weblate 733 Mar 2 00:36 id_rsa.pub
-rw-rw---- 1 weblate weblate 0 Jun 27 08:38 id_xyz
-rw-rw-r-- 1 weblate weblate 713 Mar 2 01:26 known_hosts

Seems to me like something adds rw rights on group level if none are present. However files like id_ed25519.pub keep their read permissions for 'other'

[nijel]

Does it happen anywhere in the volume, or just in the SSH keys?

[daniel-muhra-cw]

Just created a test file in /app/data and it is affected as well: The restart changed the permissions from 600 to 660

[nijel]

I've adjusted the issue title to refect this, but I have no clue where this could be coming from. As there is no such code in Weblate container it makes me think it has to be something in your Docker environment causing this.

[github-actions]

This issue has been automatically marked as stale because there wasn’t any recent activity.

It will be closed soon if no further action occurs.

Thank you for your contributions!

[daniel-muhra-cw]

Ok, so far I tracked it down to be probably related to the volume provisioner in K8s because we migrated the Weblate instance to another cluster with a different volume provisioner. However other pods which also use the same volume provisioner don't seem to show the same behaviour. Therefore I think it might be a combination of the PV and the base image that weblate is using...

[nijel]

Have you tried creating files with 0600 permissions in the other volumes? There are not really many situations when this is needed, and lack of such files might be the reason you don't see issues in other containers.

Weblate is based on python image, which at last is based on debian image. I guess somebody would notice if these images would have some such issues.

What volume provisioner was used on the problematic cluster?

[daniel-muhra-cw]

That's actually what I did: I created a new file on such a PV, changed permissions to 0600 and deleted the pod (so the underlying ReplicationSet would create a new one). Afterwards the permissions were still the same.
Doing so with the weblate pod changed the permission to 0660.
I even changed the "command" of the weblate container to run "sleep", just to ensure it is not related to the process running, but there it shows the same behaviour.

[nijel]

Maybe it's related to the Weblate container not running as root?