Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

httpOci gives '401 on reattempt after auth: https://registry.gitlab.com/' #675

Closed
nickbroon opened this issue Nov 6, 2023 · 10 comments
Closed
Assignees
Labels
bug Something isn't working

Comments

@nickbroon
Copy link

While trying to follow the 'Working with GitLab CI' example at: https://containers.dev/guide/gitlab-ci

using this GitLab pipeline definition:

 $ cat .gitlab-ci.yml
---
default:
  image: docker:latest
  services:
    - docker:dind

variables:
  DOCKER_TLS_CERTDIR: "/certs"

before_script:
  - apk add --update nodejs npm python3 make g++
  - npm install -g @devcontainers/cli

build:
  stage: build
  script:
    - devcontainer features package src

deploy:
  stage: deploy
  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
  script:
    - echo "${CI_REGISTRY_PASSWORD}" | docker login ${CI_REGISTRY} --username ${CI_REGISTRY_USER} --password-stdin
    # - cat ~/.docker/config.json
    # - export DEVCONTAINERS_OCI_AUTH="${CI_REGISTRY}|${CI_REGISTRY_USER}|${CI_REGISTRY_PASSWORD}"
    # - echo ${DEVCONTAINERS_OCI_AUTH}
    # - devcontainer features info --log-level trace tags ${CI_REGISTRY}/${CI_REGISTRY_IMAGE}/graphnos
    - devcontainer features publish --log-level trace --registry ${CI_REGISTRY} --namespace ${CI_REGISTRY_IMAGE} src

I get the following 401 auth error from httpOci. As can be seen from the commented out additional line and addition of --log-level trace I tried to determine if there was problem. But unfortunately it's not obvious to me why the 401 is happening. The docker login succeeded, and httpOci reports finding auth docker populated in /root/.docker/config.json

Executing "step_script" stage of the job script
00:09
Using docker image sha256:5b912308540a3c49177b012c400f71cbb53ff56c4ee79f36323fdff7527f9f44 for docker:latest with digest docker@sha256:1dfc375736e448806602211e09a9b1390eb110548dcb839eef374da357ca5f5d ...
$ apk add --update nodejs npm python3 make g++
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/35) Upgrading musl (1.2.4-r1 -> 1.2.4-r2)
(2/35) Installing libgcc (12.2.1_git20220924-r10)
(3/35) Installing libstdc++ (12.2.1_git20220924-r10)
(4/35) Installing libstdc++-dev (12.2.1_git20220924-r10)
(5/35) Installing binutils (2.40-r7)
(6/35) Installing libgomp (12.2.1_git20220924-r10)
(7/35) Installing libatomic (12.2.1_git20220924-r10)
(8/35) Installing gmp (6.2.1-r3)
(9/35) Installing isl26 (0.26-r1)
(10/35) Installing mpfr4 (4.2.0_p12-r0)
(11/35) Installing mpc1 (1.3.1-r1)
(12/35) Installing gcc (12.2.1_git20220924-r10)
(13/35) Installing musl-dev (1.2.4-r2)
(14/35) Installing libc-dev (0.7.2-r5)
(15/35) Installing g++ (12.2.1_git20220924-r10)
(16/35) Installing make (4.4.1-r1)
(17/35) Installing brotli-libs (1.0.9-r14)
(18/35) Installing c-ares (1.19.1-r0)
(19/35) Installing icu-data-en (73.2-r2)
Executing icu-data-en-73.2-r2.post-install
*
* If you need ICU with non-English locales and legacy charset support, install
* package icu-data-full.
*
(20/35) Installing icu-libs (73.2-r2)
(21/35) Installing nghttp2-libs (1.57.0-r0)
(22/35) Installing nodejs (18.18.2-r0)
(23/35) Installing npm (9.6.6-r0)
(24/35) Installing libbz2 (1.0.8-r5)
(25/35) Installing libexpat (2.5.0-r1)
(26/35) Installing libffi (3.4.4-r2)
(27/35) Installing gdbm (1.23-r1)
(28/35) Installing mpdecimal (2.5.1-r2)
(29/35) Installing libpanelw (6.4_p20230506-r0)
(30/35) Installing readline (8.2.1-r1)
(31/35) Installing sqlite-libs (3.41.2-r2)
(32/35) Installing python3 (3.11.6-r0)
(33/35) Installing python3-pycache-pyc0 (3.11.6-r0)
(34/35) Installing pyc (0.1-r0)
(35/35) Installing python3-pyc (3.11.6-r0)
Executing busybox-1.36.1-r2.trigger
OK: 358 MiB in 89 packages
$ npm install -g @devcontainers/cli
added 1 package in 2s
$ echo "${CI_REGISTRY_PASSWORD}" | docker login ${CI_REGISTRY} --username ${CI_REGISTRY_USER} --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ devcontainer features publish --log-level trace --registry ${CI_REGISTRY} --namespace ${CI_REGISTRY_IMAGE} src
[2023-11-06T09:02:30.141Z] @devcontainers/cli 0.52.1. Node.js v18.18.2. linux 5.4.109+ x64.
[2023-11-06T09:02:30.141Z] Packaging feature collection...
[2023-11-06T09:02:30.143Z] Processing feature: feature1...
[2023-11-06T09:02:30.161Z] Packaged 1 features!
[2023-11-06T09:02:30.161Z] Processing feature: feature1...
[2023-11-06T09:02:30.162Z] > input: registry.gitlab.com/registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1
[2023-11-06T09:02:30.162Z] >
[2023-11-06T09:02:30.162Z] > resource: registry.gitlab.com/registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1
[2023-11-06T09:02:30.162Z] > id: feature1
[2023-11-06T09:02:30.162Z] > owner: registry.gitlab.com
[2023-11-06T09:02:30.162Z] > namespace: registry.gitlab.com/nickbroon/my-devcontainer-feature
[2023-11-06T09:02:30.162Z] > registry: registry.gitlab.com
[2023-11-06T09:02:30.162Z] > path: registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1
[2023-11-06T09:02:30.163Z] >
[2023-11-06T09:02:30.163Z] > version: latest
[2023-11-06T09:02:30.163Z] > tag?: latest
[2023-11-06T09:02:30.163Z] > digest?: undefined
[2023-11-06T09:02:30.163Z] Feature Annotations: {"dev.containers.metadata":"{\"id\":\"feature1\",\"version\":\"1.0.0\",\"name\":\"GraphNOS Development\",\"description\":\"GraphNOS APT repository configuration\",\"documentationURL\":\"https://gitlab.com/graphiant/tools/my-devcontainer-feature/-/blob/main/src/feature1\",\"licenseURL\":\"https://gitlab.com/graphiant/tools/my-devcontainer-feature/-/blob/main/LICENSE\",\"options\":{\"feature1_release\":{\"type\":\"string\",\"description\":\"GraphNOS Release\",\"proposals\":[\"main\",\"2.x\",\"3.x\"],\"default\":\"main\"}},\"postCreateCommand\":\"/usr/local/bin/feature1-add-apt-repo.sh\",\"containerEnv\":{\"GRAPHNOS_ENTITLEMENT_TOKEN\":\"${localEnv:GRAPHNOS_ENTITLEMENT_TOKEN}\"}}"}
[2023-11-06T09:02:30.163Z] Fetching published versions...
[2023-11-06T09:02:30.204Z] [httpOci] Attempting to authenticate via 'Bearer' auth.
[2023-11-06T09:02:30.207Z] [httpOci] Found auths entry in '/root/.docker/config.json' for registry 'registry.gitlab.com'
[2023-11-06T09:02:30.207Z] [httpOci] Attempting to fetch bearer token from:  https://gitlab.com/jwt/auth?service=container_registry&scope=repository:registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1:pull
[2023-11-06T09:02:30.358Z] [httpOci] 401 on reattempt after auth: https://registry.gitlab.com/v2/registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1/tags/list
[2023-11-06T09:02:30.358Z] (!) ERR: Could not fetch published tags for 'registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1' : {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1","ProjectPath":"","Action":"pull"}]}]}
 
[2023-11-06T09:02:30.358Z] (!) ERR: Failed to publish 'registry.gitlab.com/registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1'
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

Why would httpOci fail to auth and get a 401 here? Any suggestions on further debugging/logging to attempt?

@joshspicer
Copy link
Member

joshspicer commented Nov 6, 2023

Thanks for sharing the detailed trace logs.

To help debug this, would you be able to create a gitlab PAT manually (ensuring the permissions are permissive enough to push/pull from the gitlab registry) and use that in instead of CI_REGISTRY_PASSWORD? That would be helpful for me to see if it's a detail we should update in the guide, or perhaps an implementation issue in the CLI.

I'm admittedly not too familiar with gitlab registry auth, it may be an issue in the CLI's implementation here.

edit: following https://docs.gitlab.com/ee/user/packages/container_registry/authenticate_with_container_registry.html

@joshspicer joshspicer self-assigned this Nov 6, 2023
@nickbroon
Copy link
Author

I created a PAT with all permissions granted and tried using that instead, but got the same error.

I made this change, and had saved the PAT in a CI variable PERSONAL_ACCCESS_TOKEN for the job to use.

--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -21,7 +21,7 @@ deploy:
   rules:
     - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
   script:
-    - echo "${CI_REGISTRY_PASSWORD}" | docker login ${CI_REGISTRY} --username ${CI_REGISTRY_USER} --password-stdin
+    - echo "${PERSONAL_ACCCESS_TOKEN}" | docker login ${CI_REGISTRY} --username ${GITLAB_USER_LOGIN} --password-stdin
     # - cat ~/.docker/config.json
     # - export DEVCONTAINERS_OCI_AUTH="${CI_REGISTRY}|${CI_REGISTRY_USER}|${CI_REGISTRY_PASSWORD}"
     # - echo ${DEVCONTAINERS_OCI_AUTH}

Docker could still login using the PAT, and as before httpOci found auth details in config.json but failed.

$ npm install -g @devcontainers/cli
added 1 package in 1s
$ echo "${PERSONAL_ACCCESS_TOKEN}" | docker login ${CI_REGISTRY} --username ${GITLAB_USER_LOGIN} --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ devcontainer features publish --log-level trace --registry ${CI_REGISTRY} --namespace ${CI_REGISTRY_IMAGE} src
[2023-11-06T17:33:07.243Z] @devcontainers/cli 0.52.1. Node.js v18.18.2. linux 5.4.109+ x64.
[2023-11-06T17:33:07.243Z] Packaging feature collection...
[2023-11-06T17:33:07.245Z] Processing feature: feature1...
[2023-11-06T17:33:07.263Z] Packaged 1 features!
[2023-11-06T17:33:07.264Z] Processing feature: feature1...
[2023-11-06T17:33:07.265Z] > input: registry.gitlab.com/registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1
[2023-11-06T17:33:07.265Z] >
[2023-11-06T17:33:07.265Z] > resource: registry.gitlab.com/registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1
[2023-11-06T17:33:07.265Z] > id: feature1
[2023-11-06T17:33:07.265Z] > owner: registry.gitlab.com
[2023-11-06T17:33:07.265Z] > namespace: registry.gitlab.com/nickbroon/my-devcontainer-feature
[2023-11-06T17:33:07.265Z] > registry: registry.gitlab.com
[2023-11-06T17:33:07.265Z] > path: registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1
[2023-11-06T17:33:07.266Z] >
[2023-11-06T17:33:07.266Z] > version: latest
[2023-11-06T17:33:07.266Z] > tag?: latest
[2023-11-06T17:33:07.266Z] > digest?: undefined
[2023-11-06T17:33:07.266Z] Feature Annotations: {"dev.containers.metadata":"{\"id\":\"feature1\",\"version\":\"1.0.0\",\"name\":\"GraphNOS Development\",\"description\":\"GraphNOS APT repository configuration\",\"documentationURL\":\"https://gitlab.com/graphiant/tools/my-devcontainer-feature/-/blob/main/src/feature1\",\"licenseURL\":\"https://gitlab.com/graphiant/tools/my-devcontainer-feature/-/blob/main/LICENSE\",\"options\":{\"feature1_release\":{\"type\":\"string\",\"description\":\"GraphNOS Release\",\"proposals\":[\"main\",\"2.x\",\"3.x\"],\"default\":\"main\"}},\"postCreateCommand\":\"/usr/local/bin/feature1-add-apt-repo.sh\",\"containerEnv\":{\"GRAPHNOS_ENTITLEMENT_TOKEN\":\"${localEnv:GRAPHNOS_ENTITLEMENT_TOKEN}\"}}"}
[2023-11-06T17:33:07.266Z] Fetching published versions...
[2023-11-06T17:33:07.307Z] [httpOci] Attempting to authenticate via 'Bearer' auth.
[2023-11-06T17:33:07.310Z] [httpOci] Found auths entry in '/root/.docker/config.json' for registry 'registry.gitlab.com'
[2023-11-06T17:33:07.310Z] [httpOci] Attempting to fetch bearer token from:  https://gitlab.com/jwt/auth?service=container_registry&scope=repository:registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1:pull
[2023-11-06T17:33:07.481Z] [httpOci] 401 on reattempt after auth: https://registry.gitlab.com/v2/registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1/tags/list
[2023-11-06T17:33:07.481Z] (!) ERR: Could not fetch published tags for 'registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1' : {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1","ProjectPath":"","Action":"pull"}]}]}
 
[2023-11-06T17:33:07.481Z] (!) ERR: Failed to publish 'registry.gitlab.com/registry.gitlab.com/nickbroon/my-devcontainer-feature/feature1'
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

@joshspicer
Copy link
Member

joshspicer commented Nov 6, 2023

Thanks! Every registry seems to do auth a little differently (a lot follow a specification set out by docker, but nearly every registry i've seen so far has some exception). We're implementing this ourselves in the CLI so we need to account for these differences.

I think we'll need to dig in more to understand the differences. Afterwards a regression test can be placed here to ensure continued support: https://github.com/devcontainers/cli/blob/main/src/test/container-features/registryCompatibilityOCI.test.ts#L40-L69

@VincentDondain VincentDondain added the bug Something isn't working label Nov 22, 2023
@joshspicer
Copy link
Member

Upon closer review, it looks like GitLab does not yet support the arifacts specification required to publish dev container Features.

I'd recommend raising an issue with gitlab, or instead publishing your Features to one of the many registries that do currently support the spec (Google CR, GitHub CR, Azure CR, docker hub, etc...)

Please do re-open this if/when the support is implemented by gitlab :)

@joshspicer joshspicer closed this as not planned Won't fix, can't repro, duplicate, stale Nov 27, 2023
@nickbroon
Copy link
Author

@joshspicer can you detail or link the information you reviewed that led you to conclude that Gitlab did not support the required Artifacts Specification? I can use this as part of opening an issue with Gitlab.

@joshspicer
Copy link
Member

The "oras" project I linked above outlines all the implementors. I couldn't find any evidence online that this data is out-of-date, please correct me if i'm wrong: https://oras.land/docs/compatible_oci_registries#registries-supporting-oci-artifacts

@nickbroon
Copy link
Author

I found https://gitlab.com/gitlab-org/gitlab/-/issues/385277 that requests that GitLab add support for OCI artifacts

@nickbroon
Copy link
Author

I guess https://containers.dev/guide/gitlab-ci should be marked as not working/supported (assuming it ever did), and linked to this issues.

@joshspicer
Copy link
Member

I guess https://containers.dev/guide/gitlab-ci should be marked as not working/supported (assuming it ever did), and linked to this issues.

This guide is for publishing a dev container config into a docker image hosted on gitlab CR (which does work, doesn't need the OCI aritfacts spec implemented as it's a standard docker image with some custom labels). What I saw you trying to do here was publish Features, which is different and does requires that spec.

@joshspicer
Copy link
Member

(For any future readers) I think that this issue may still be valid (we may need to do a little fixing of the auth implementation to support gitlab), but doing so today would provide no value without the container registry supporting the aforementioned spec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants