file permission errors in the certificates init container

[blancharda]

Environment

Device and OS: RHEL VMs on Nutanix provisioned infra
App/package versions: v17.0.1
Kubernetes distro being used: RKE2

Steps to reproduce

1. deploy gitlab
2. observe logs in the certificates init container on the webservice pod

Expected result

No errors.

Actual Result

cp: cannot create regular file '/usr/share/pki/ca-trust-source/anchors/ca.crt': Permission denied
cp: cannot create regular file '/usr/share/pki/ca-trust-source/anchors/cfssl_ca': Permission denied
Copied custom certificates to RHEL-compatible directory
p11-kit: couldn't create file: /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt: Unknown error 13
p11-kit: couldn't create file: /etc/pki/ca-trust/extracted/java/cacerts: Unknown error 13
p11-kit: couldn't create file: /etc/pki/ca-trust/extracted/edk2/cacerts.bin: Unknown error 13
Updated CA trust
Copied bundles into /etc/ssl/certs

Severity/Priority

low -> mid
I'm not sure the extent of the impact here given that most environments seem to be functional, but I could see this causing some odd behavior down the line if not addressed.

Additional Context

• This may be related to Pepr's user/group mutation on the init container
• This may be an issue specific to the registry1 image
• The OS is FIPS enabled? (not sure this is related.. but might be relevant)

[blancharda]

fwiw -- adjusting the init container's runtime user to match the registry1 image user with the following override did not resolve the behavior:

  - name: gitlab
    repository: ghcr.io/defenseunicorns/packages/uds/gitlab
    ref: 17.0.1-uds.1-registry1
    overrides:
      gitlab:
        gitlab:
          values:
            - path: gitlab.webservice.init.containerSecurityContext
              value:
                runAsNonRoot: true
                runAsUser: 65534

(Note that the GL chart doesn't appear to expose the security context for each individual init container.. Applying the above broke the dependencies initcontainer which had to be manually adjusted in-cluster in order to test a "successful" deployment)

[blancharda]

Dropping some links to various pieces of the helm chart here as well because they were a bit of pain to track down:

• webservice deployment
• certificates container helper template
• init container security context template
• webservice values file (the docs list the key in the wrong place 🙃)

[oates]

We could check to see if we see this in the staging env @defenseunicorns/swf

[Racer159]

Observed this in staging - it currently only affects the registry1 image - long term we would want to have a way (i.e. through trustmanager) to ensure that the certs are what we want them to be with no other certs in the truststore.

[Racer159]

Of note adding certs to GitLab works for webservice but not registry and the GitLab chart does not expose extra volume mounts in the registry...

[Racer159]

Blocking this on defenseunicorns/uds-core#464 for now - we could implement our own path / workarounds but it would be best to integrate with uds-core

[oates]

Keeping this in blocked for now. This doesn't affect uds.is env because we use public certs, but still something needed for those in airgap env.