Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using personal access token with Dockerhub API Beta #2127

Closed
qbiqing opened this issue Jul 23, 2021 · 35 comments
Closed

Using personal access token with Dockerhub API Beta #2127

qbiqing opened this issue Jul 23, 2021 · 35 comments
Assignees

Comments

@qbiqing
Copy link

qbiqing commented Jul 23, 2021

Problem description

When trying to use the Dockerhub beta API, such as this endpoint: /v2/namespaces/{namespace}/repositories/{repository}/images,

I used the authentication method through Personal Access Tokens as introduced in the docs. That is, I created a JWT using the authentication endpoint in this section and with the personal access token instead of the password in the password field. I was able to get a JWT in return, however, when I use it to get details about the images, I got this error:

{
    "message": "access to the resource is forbidden with personal access token"
}

I'm confused if the personal access token is an acceptable mode of authentication? This is not clear in the docs.

@mikeparker
Copy link

Apologies, that sounds like a gap in the docs. PATs are not currently supported for API operations, but this is something we are looking to implement in future.

@qbiqing
Copy link
Author

qbiqing commented Jul 26, 2021

@mikeparker thanks, good to know! It would be really useful for apps using the Dockerhub API.

@qbiqing
Copy link
Author

qbiqing commented Jul 30, 2021

To add on, the docs now are even more misleading, because this is under the section of "Create an authentication token":
image
It says PAT can be used to authenticate with but I still get forbidden access to resource.

@mikeparker
Copy link

@qbiqing you're absolutely right, the docs are misleading here. I'll get this fixed. Sorry again.

@modem7
Copy link

modem7 commented Aug 14, 2021

Apologies, that sounds like a gap in the docs. PATs are not currently supported for API operations, but this is something we are looking to implement in future.

@mikeparker I thought that docker/roadmap#115 was supposed to have been rolled out for this very purpose?

@modem7
Copy link

modem7 commented Aug 31, 2021

Is there any updates on this?

There doesn't seem to be much of a point to 2fa if you're unable to upload something as simple as a readme file.

@srbala
Copy link

srbala commented Sep 19, 2021

PATs are not currently supported for API operations, but this is something we are looking to implement in future.
@mikeparker Any tentative timeline for this to be implemented

@github-actions
Copy link

We are clearing up our old issues and your ticket has been open for 6 months with no activity. Remove stale label or comment or this will be closed in 15 days.

@github-actions github-actions bot added the Stale label Mar 19, 2022
@modem7
Copy link

modem7 commented Mar 19, 2022

Stalebot bump

@technicallyjosh
Copy link

technicallyjosh commented Mar 19, 2022

I think something to help this confusion on the error message is to return something like "access to the resource is forbidden with an access token generated from a PAT". As far as the API access, this was bumped my way and should be fairly simple to patch in since it is a repo scoped call. (Right now PATs only have repo scopes). We can definitely look into allowing for PAT generated JWTs to access this resource with the repo:read scope.

@technicallyjosh
Copy link

I've opened a PR to allow for image management route usage with PAT generated JWTs in our APIs. We can address the verbiage on the 403 this week.

@github-actions github-actions bot removed the Stale label Mar 20, 2022
@technicallyjosh
Copy link

👋🏼 You can now use a properly scoped JWT generated from a PAT on these image management routes. Changes to the wording should be updated within the next week.

@technicallyjosh
Copy link

I'll close this out. Thank you for leaving the feedback! Sorry it took a while to get to!

@modem7
Copy link

modem7 commented Jun 2, 2022

👋🏼 You can now use a properly scoped JWT generated from a PAT on these image management routes. Changes to the wording should be updated within the next week.

Heya @technicallyjosh

Lots of people are still having issues with this.

Do you have a link to the appropriate documentation to get this to work as intended?

Thanks!

@christian-korneck
Copy link

christian-korneck commented Jun 11, 2022

@technicallyjosh Thanks for this. Unfortunately I don't seem to be able to get this to work.

What I try is:

  • generate JWT for PAT (GET /v2/users/login) - works
    • note: the PAT has repo:admin scope
  • try to update a repo description (use JWT as bearer token for PATCH /v2/repositories/my-user/my-repo/) - fails

Result is a 403 FORBIDDEN with message:

access is forbidden with a JWT issued from a personal access token

Am I doing anything incorrectly? Would appreciate any pointer into the right direction. Thanks a lot in advance!

@technicallyjosh
Copy link

technicallyjosh commented Jun 11, 2022

@modem7 @christian-korneck Hey thanks for the ping on this not working still. It seems we might have some sort of regression here. I do know we've been slaying tech debt and moving APIs around specifically with registry/repos, so I'm wondering if something got lost in translation behind the scenes. I'll have some bandwidth to look at this this weekend and early next week and keep you updated.

@modem7 Because of some of these services being under refactoring, etc, we've been very selective as to what we publicly document. I'll ping the team and see if they are comfortable with documenting these APIs in the given state.

@technicallyjosh
Copy link

technicallyjosh commented Jun 12, 2022

Okay so it seems like there were some routes missed such as the one you mentioned: PATCH /v2/repositories/{namespace}/{repo}. I'll put a PR in to see if I can get the rest of these to work with the tokens.

Edit: I have a PR that should take care of any missing repository management routes allowing for PAT usage there. I'll still discuss with the team what we want to do about public API docs.

@technicallyjosh
Copy link

technicallyjosh commented Jun 14, 2022

👋🏼 These fixes are now live on production. Let us know if there are any issues 😄. I can work with the team on updating the proper docs for these. If your PAT doesn't have the right scopes, you will receive a message with something like "insufficient scope" or something of the like. If you get that, you may need to tweak the scopes for the token used to generate the JWT.

@christian-korneck
Copy link

christian-korneck commented Jun 14, 2022

@technicallyjosh thanks a lot for looking into this so quickly, much appreciated!

Unfortunately I still don't seem to get my use case to be working. Here's what I do:

1.) create a PAT via the DockerHub WebUI with read, write, delete scope.
2.) generate JWT. The password is a personal access token.

curl --location --request POST 'https://hub.docker.com/v2/users/login' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data-raw '{
  "username": "my-docker-username",
  "password": "00000000-0000-0000-0000-000000000000"
}'

the result body contains a JWT in the style of

{"token":"ey...pg"}}

3.) Use the token in a request to update a repo's description:

curl --location --request PATCH 'https://hub.docker.com/v2/repositories/my-docker-user/my-docker-repo/' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer ey...pg' \
--data-raw '{
  "description": "Hello World!"
}'

The result is a 403 FORBIDDEN:

{
    "detail": "access is forbidden with a JWT issued from a personal access token"
}

(The same example works if I use a password to create the JWT instead of a PAT)

Do you have any idea what I might be doing wrong? I highly appreciate any pointer into the right direction. (I assume it must be some stupid mistake on my side)?

Again, thanks a lot for your time and efforts!!

@technicallyjosh
Copy link

technicallyjosh commented Jun 14, 2022

Of course! The JWT generation is fine.

Interesting... I'll have to look into that as I know I allowed the PATCH to be used with a PAT generated token. It's possible that that route is going to a service that I was not aware of (we are migrating from an older service and it may still be used there).

@modem7
Copy link

modem7 commented Jun 15, 2022

Of course! The JWT generation is fine.

Interesting... I'll have to look into that as I know I allowed the PATCH to be used with a PAT generated token. It's possible that that route is going to a service that I was not aware of (we are migrating from an older service and it may still be used there).

Heya @technicallyjosh,

Would it make sense keeping this issue open until we can do community verification that all works as expected as this is still not working as expected.

@zigarn
Copy link

zigarn commented Jul 3, 2022

Hi @technicallyjosh, any news about the problem raised by @christian-korneck (#2127 (comment)) that is still existant.
It is generating failure in CI jobs trying to update repo's description.

@zhangguanzhang
Copy link

zhangguanzhang commented Jul 13, 2022

@technicallyjosh thanks a lot for looking into this so quickly, much appreciated!

Unfortunately I still don't seem to get my use case to be working. Here's what I do:

1.) create a PAT via the DockerHub WebUI with read, write, delete scope. 2.) generate JWT. The password is a personal access token.

curl --location --request POST 'https://hub.docker.com/v2/users/login' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data-raw '{
  "username": "my-docker-username",
  "password": "00000000-0000-0000-0000-000000000000"
}'

the result body contains a JWT in the style of

{"token":"ey...pg"}}

3.) Use the token in a request to update a repo's description:

curl --location --request PATCH 'https://hub.docker.com/v2/repositories/my-docker-user/my-docker-repo/' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer ey...pg' \
--data-raw '{
  "description": "Hello World!"
}'

The result is a 403 FORBIDDEN:

{
    "detail": "access is forbidden with a JWT issued from a personal access token"
}

(The same example works if I use a password to create the JWT instead of a PAT)

Do you have any idea what I might be doing wrong? I highly appreciate any pointer into the right direction. (I assume it must be some stupid mistake on my side)?

Again, thanks a lot for your time and efforts!!

same issue @milosgajdos PATL 👀

@zappy-shu
Copy link

zappy-shu commented Jul 13, 2022

Good afternoon @zhangguanzhang

We are rolling out personal access token support across endpoints as we speak. The PATCH repository endpoint was rolled out today with support for admin scoped PATs. Could you ensure your PAT has admin scope and test again for me?

Thanks

@zhangguanzhang
Copy link

zhangguanzhang commented Jul 13, 2022

Good afternoon @zhangguanzhang

We are rolling out personal access token support across endpoints as we speak. The PATCH repository endpoint was rolled out today with support for admin scoped PATs. Could you ensure your PAT has admin scope and test again for me?

Thanks

image
I used the Read, Write, Delete, but does not work for me.

repo_user=zhangguanzhang
repo_token=xxx-xxx-xxx-xxx-xxx
token=$(curl -s -H "Content-Type: application/json" -X POST \
    -d '{"username": "'${repo_user}'", "password": "'${repo_token}'"}' \
    https://hub.docker.com/v2/users/login/ | jq -r .token)
$ curl -iLH "Authorization: Bearer ${token}" -X DELETE \
    https://hub.docker.com/v2/repositories/zhangguanzhang/xxx/tags/test
HTTP/1.1 301 MOVED PERMANENTLY
date: Wed, 13 Jul 2022 12:05:09 GMT
content-type: text/html; charset=utf-8
transfer-encoding: chunked
x-ratelimit-limit: 600
x-ratelimit-reset: 1657713969
x-ratelimit-remaining: 599
x-frame-options: deny
location: https://hub.docker.com/v2/repositories/zhangguanzhang/xxx/tags/test/
server: nginx
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000

HTTP/1.1 403 FORBIDDEN
date: Wed, 13 Jul 2022 12:05:10 GMT
content-type: application/json
transfer-encoding: chunked
x-ratelimit-limit: 600
x-ratelimit-reset: 1657713970
x-ratelimit-remaining: 598
x-frame-options: deny
allow: GET, DELETE, HEAD, OPTIONS
server: nginx
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000

{"detail": "access is forbidden with a JWT issued from a personal access token"}

or used the jwt

$ curl -iLH "Authorization: JWT ${token}" -X DELETE \
    https://hub.docker.com/v2/repositories/zhangguanzhang/xxx/tags/test
HTTP/1.1 301 MOVED PERMANENTLY
date: Wed, 13 Jul 2022 12:05:46 GMT
content-type: text/html; charset=utf-8
transfer-encoding: chunked
x-ratelimit-limit: 600
x-ratelimit-reset: 1657714006
x-ratelimit-remaining: 597
x-frame-options: deny
location: https://hub.docker.com/v2/repositories/zhangguanzhang/xxx/tags/test/
server: nginx
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000

HTTP/1.1 403 FORBIDDEN
date: Wed, 13 Jul 2022 12:05:46 GMT
content-type: application/json
transfer-encoding: chunked
x-ratelimit-limit: 600
x-ratelimit-reset: 1657714006
x-ratelimit-remaining: 596
x-frame-options: deny
allow: GET, DELETE, HEAD, OPTIONS
server: nginx
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000

{"detail": "access is forbidden with a JWT issued from a personal access token"}

if I used the password for repo_token, It works

@zappy-shu
Copy link

@zhangguanzhang could you do a verbose curl and list the response headers for me? And give me the time/date that you make the call. Thank you

@zhangguanzhang
Copy link

@zhangguanzhang could you do a verbose curl and list the response headers for me? And give me the time/date that you make the call. Thank you

I updated the previous comment with more details

@zappy-shu
Copy link

@zhangguanzhang that appears to be a DELETE tag operation rather than a PATCH repo. Deleting a tag has not yet been migrated over so does not currently have PAT support (it's next on the list). Does PATCH repo work for you now?

@zhangguanzhang
Copy link

@zhangguanzhang that appears to be a DELETE tag operation rather than a PATCH repo. Deleting a tag has not yet been migrated over so does not currently have PAT support (it's next on the list). Does PATCH repo work for you now?

How to delete a tag use the api 😢,I just want to delete a tag

@zigarn
Copy link

zigarn commented Jul 13, 2022

My test executing the code from https://circleci.com/developer/orbs/orb/circleci/docker#commands-update-description to update description:

JWT=$(curl -s -d "username=$DOCKERHUB_USERNAME&password=$DOCKERHUB_PAT" https://hub.docker.com/v2/users/login/ | jq -r .token)

$ curl -v -H "Authorization: Bearer ${JWT}" -X PATCH --data-urlencode full_description@README.md https://hub.docker.com/v2/repositories/zenika/ztraining2strigo/
*   Trying 2600:1f18:2148:bc02:ab79:9619:af3f:8e4b:443...
* Connected to hub.docker.com (2600:1f18:2148:bc02:ab79:9619:af3f:8e4b) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.docker.com
*  start date: Jun 12 00:00:00 2022 GMT
*  expire date: Jul 11 23:59:59 2023 GMT
*  subjectAltName: host "hub.docker.com" matched cert's "*.docker.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> PATCH /v2/repositories/zenika/ztraining2strigo/ HTTP/1.1
> Host: hub.docker.com
> User-Agent: curl/7.81.0
> Accept: */*
> Authorization: Bearer ...
> Content-Length: 10307
> Content-Type: application/x-www-form-urlencoded
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 415 Unsupported Media Type
< date: Wed, 13 Jul 2022 14:04:11 GMT
< content-type: application/json
< content-length: 42
< x-ratelimit-limit: 600
< x-ratelimit-reset: 1657721111
< x-ratelimit-remaining: 600
< x-trace-id: 09ccff23d18e55e9301e22d6981adf39
< server: nginx
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< strict-transport-security: max-age=31536000
< 
{"message":"415: Unsupported Media Type"}
* Connection #0 to host hub.docker.com left intact

Looking at documentation, looks like x-www-form-urlencoded is not supported (anymore?).

$ curl -v -H "Authorization: Bearer ${JWT}" -H "Content-type: application/json" -X PATCH --data '{"full_description": "README.md"}' https://hub.docker.com/v2/repositories/zenika/ztraining2strigo/
*   Trying 2600:1f18:2148:bc01:33c8:2bac:60fa:e905:443...
* Connected to hub.docker.com (2600:1f18:2148:bc01:33c8:2bac:60fa:e905) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.docker.com
*  start date: Jun 12 00:00:00 2022 GMT
*  expire date: Jul 11 23:59:59 2023 GMT
*  subjectAltName: host "hub.docker.com" matched cert's "*.docker.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> PATCH /v2/repositories/zenika/ztraining2strigo/ HTTP/1.1
> Host: hub.docker.com
> User-Agent: curl/7.81.0
> Accept: */*
> Authorization: Bearer ...
> Content-type: application/json
> Content-Length: 33
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Wed, 13 Jul 2022 14:13:58 GMT
< content-type: application/json
< content-length: 512
< x-ratelimit-limit: 600
< x-ratelimit-reset: 1657721698
< x-ratelimit-remaining: 600
< x-trace-id: 738f0a10a317d746025b6cb28d0b7426
< server: nginx
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< strict-transport-security: max-age=31536000
< 
{"user":"zenika","name":"ztraining2strigo","namespace":"zenika","repository_type":"image","status":1,"description":"Create Strigo class for Zenika training.","is_private":false,"is_automated":false,"can_edit":false,"star_count":0,"pull_count":2277,"last_updated":"2022-07-13T13:56:40.069147Z","date_registered":"2021-03-03T16:01:09.749645Z","collaborator_count":0,"affiliation":null,"hub_user":"zigarn","has_starred":false,"full_description":"README.md","permissions":{"read":false,"write":false,"admin":false}}
* Connection #0 to host hub.docker.com left intact

=> Works 🚀

@zappy-shu
Copy link

@zhangguanzhang unfortunately for now you would have to use a password. I will prioritize getting the DELETE tag flow migrated over to use the new authentication flows and support PATs ASAP. Sorry for the inconvenience

@zappy-shu
Copy link

@zigarn that's correct. URL data encoding is no longer supported. If this is a feature you want please could you open a new ticket? This one is getting a little full. Thanks

@zigarn
Copy link

zigarn commented Jul 13, 2022

@zappy-shu It's OK for me, just need to update https://circleci.com/developer/orbs/orb/circleci/docker#commands-update-description

@christian-korneck
Copy link

@zappy-shu thanks 🙇 a lot also for this also from my side! Updating the repo settings now seems to work fine. 🚀 (At a first look my "Docker Push Readme" Tool now seems to work with PATs without any change necessary).

Btw. I didn't need to create an admin scope PAT. Creating a read/write PAT from the Docker Hub WebUI seems to be sufficient.

@reitzig
Copy link

reitzig commented Jul 2, 2023

Extending on what @zigarn posted (thank you!), I needed to upload Markdown instead of a file name. I used to use this:

curl -siX PATCH "https://hub.docker.com/v2/repositories/${dh_repo}/" \
     -H "Authorization: JWT ${dh_token}" \
     -o "${CURL_OUT}" \
     --data-urlencode description="${gh_description}" \
     --data-urlencode full_description@"${readme_profile_filepath}"

which does not work anymore. Now I need to encode the file content as JSON:

curl -siX PATCH "https://hub.docker.com/v2/repositories/${dh_repo}/" \
     -H "Authorization: JWT ${dh_token}" \
     -H "Content-Type: application/json" \
     -o "${CURL_OUT}" \
     --data @- << PAYLOAD
{
    "description": "${gh_description}",
    "full_description": $(jq -Rsa . "${readme_profile_filepath}")
}
PAYLOAD

Seems to work!

Credits for the usage of jq: SO#50380697

reitzig added a commit to reitzig/texlive-docker that referenced this issue Jul 2, 2023
We're apparently using unsupported API, and
it has changed.

See also:
 - docker/hub-feedback#2127
 - docker/hub-tool#172
 - docker/build-push-action#21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants