Document uploading a package to ghcr.io

[dhirschfeld]

My first attempt at uploading a package failed:

I'm stuck on a failed upload:

❯ powerloader upload -vvv ./dist/conda/linux-64/mypackage-1.2.3-py39h2e92cda_0.conda:org/repo/mypackage:1.2.3 -m oci://ghcr.io
[2022-12-20 00:12:50.011] [info] URL: https://ghcr.io/
[2022-12-20 00:12:50.012] [critical] Could not upload: map::at

Originally posted by @dhirschfeld in #135 (comment)

...but I persevered (and grepped the source) and finally succeeded! 🎉

I'm just opening this issue to document the steps I had to take in case it is useful to others...

[dhirschfeld]

My first attempt failed with a "Could not upload" message:

❯ powerloader upload ./dist/boa/noarch/eq-utils-0.0.1-hcdf0012_0.conda:energy-quants/eq-utils:0.0.1-hcdf0012_0 -m oci://ghcr.io
[2023-02-20 21:17:11.912] [info] URL: https://ghcr.io/
Could not upload ./dist/boa/noarch/eq-utils-0.0.1-hcdf0012_0.conda:energy-quants/eq-utils:0.0.1-hcdf0012_0 to OCI Registry at https://ghcr.io/

Fortunately turning on verbose logging gave the clue:

❯ powerloader upload -vvv ./dist/boa/noarch/eq-utils-0.0.1-hcdf0012_0.conda:energy-quants/eq-utils:0.0.1 -m oci://ghcr.io
[2023-02-20 21:25:58.530] [info] URL: https://ghcr.io/
[2023-02-20 21:25:58.531] [warning] SSL verification is ENABLED
[2023-02-20 21:25:58.531] [warning] SSL verification is ENABLED
* Couldn't find host ghcr.io in the (nil) file; using defaults
*   Trying 20.248.137.52:443...
* Connected to ghcr.io (20.248.137.52) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /opt/mambaforge/envs/quantdev/ssl/cacert.pem
*  CApath: none
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.ghcr.io
*  start date: Jul 19 00:00:00 2022 GMT
*  expire date: Jul 19 23:59:59 2023 GMT
*  subjectAltName: host "ghcr.io" matched cert's "ghcr.io"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: POST]
* h2h3 [:path: /v2/energy-quants/eq-utils/blobs/uploads/]
* h2h3 [:scheme: https]
* h2h3 [:authority: ghcr.io]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x55d76d5ba1b0)
> POST /v2/energy-quants/eq-utils/blobs/uploads/ HTTP/2
Host: ghcr.io
accept: */*

* old SSL session ID is stale, removing
< HTTP/2 401 
< content-type: application/json
< www-authenticate: Bearer realm="https://ghcr.io/token",service="ghcr.io",scope="repository:energy-quants/eq-utils:pull"
< date: Mon, 20 Feb 2023 11:25:59 GMT
< content-length: 73
< x-github-request-id: C8AA:41C9:0EFC:1433:63F358C7
< 
* Connection #0 to host ghcr.io left intact
[2023-02-20 21:25:59.054] [error] Received 401: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]}

Could not upload ./dist/boa/noarch/eq-utils-0.0.1-hcdf0012_0.conda:energy-quants/eq-utils:0.0.1 to OCI Registry at https://ghcr.io/

So, for some reason it's failing authentication. I have an admin GitHub PAT in my GITHUB_TOKEN environment variable which is picked up by the gh CLI so the next reason was to figure out why it wasn't working for powerloader.

Grepping the source, I found out that powerloader expects the environment variable to be named GHA_PAT. The name can be easily documented, however, it would be nice to also support the GITHUB_TOKEN env var as does the gh CLI:

❯ gh help environment
GH_TOKEN, GITHUB_TOKEN (in order of precedence): an authentication token for github.com API requests. Setting this avoids being prompted to authenticate and takes precedence over previously stored credentials.

GH_ENTERPRISE_TOKEN, GITHUB_ENTERPRISE_TOKEN (in order of precedence): an authentication token for API requests to GitHub Enterprise. When setting this, also set GH_HOST.

Having configured the GHA_PAT env var I still couldn't upload, getting the map::at error I'd seen previously:

❯ powerloader upload -vvv ./dist/boa/noarch/eq-utils-0.0.1-hcdf0012_0.conda:energy-quants/eq-utils:0.0.1 -m oci://ghcr.io
[2023-02-20 22:04:21.503] [info] URL: https://ghcr.io/
[2023-02-20 22:04:21.505] [warning] SSL verification is ENABLED
[2023-02-20 22:04:21.505] [warning] SSL verification is ENABLED
[2023-02-20 22:04:21.505] [critical] Could not upload: map::at

i.e. I was attempting to incorporate the org/user name in the package name. It wasn't too difficult to figure out that powerloader wanted the org/user name in the GHA_USER env var.

After setting export GHA_USER='energy-quants' the upload finally worked! 🎉

❯ powerloader upload -vvv eq-utils-0.0.1-hcdf0012_0.conda:eq-utils:0.0.1 -m oci://ghcr.io
<snip>
* Connection #0 to host ghcr.io left intact
[2023-02-20 22:08:17.888] [info] Uploaded 1 layers to energy-quants/eq-utils:0.0.1
Finished upload for eq-utils-0.0.1-hcdf0012_0.conda:eq-utils:0.0.1 to OCI Registry at https://ghcr.io/

[dhirschfeld]

TL;DR

export GHA_PAT=<github token>
export GHA_USER=<package owner>  # org/user to publish to

powerloader upload <filepath to conda package>:<package name>:<package version> -m oci://ghcr.io

[dhirschfeld]

It seems a little strange to have the location the package is published to be controlled by hidden state (an env var).

It might be more obvious to specify that as a CLI argument - e.g.

powerloader upload eq-utils-0.0.1.conda:eq-utils:0.0.1 -m oci://ghcr.io --owner energy-quants

I guess that might not make sense for all OCI registries?

[dhirschfeld]

Anyway, it works - I've uploaded a conda package as an OCI artifact to the GitHub Container Registry! That's awesome! 🚀🎉

Figuring out how to install such a package will have to be a problem for another evening... 🤔