New source-dockerhub connector from docker hub registry API

[ChristopheDuong]

Tell us about the new connector you’d like to have

A source-dockerhub to pull data from https://docs.docker.com/registry/spec/api/#listing-repositories to know what docker images are being published by a docker account.

If we can have extra metadata like statistics on:

• docker image size
• supported architecture (m1 compatible?)
• how many docker pull over time
• how many versions per docker images and when were they published
• etc

That's even better!

Describe the context around this new connector

• Why do you need this integration? How does your team intend to use the data? This helps us understand the use case.

We need a data source to know what has actually been published on airbyte's docker registry and thus available to OSS users.

[evantahler]

This information is going to be required for Connector incident management (https://github.com/airbytehq/airbyte-cloud/issues/1506) as well, so we can attribute errors to a specific version. e.g. If failures are attributed to V4, and we fix it and release v5, we can ignore all v4 errors.

Moved to the DX team backlog.

[evantahler]

@Phlair - Can you check with Docker inc if we can get this additional data being a Verified Publisher? I think "number of downloads per day per version" would be a great secondary stream for this connector.

[evantahler]

I've updated the title to be soruce-dockerhub to clarify that we are pulling data from the docker registry, rather than a docker image itself

[swyxio]

sharing my research as i go along:

• i struggled with finding documentation for authentication for the dockerhub api. the docs have the audacity to say "While authentication and authorization support will influence this specification, details of the protocol will be left to a future specification." and i bet that hasnt been updated in years lol
• this stackoverflow answer seems to have the answer:
  • for unauthenticated/public repos, you can just get a JWT from https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/alpine:pull
  • and then use the JWT in a curl -s -H "Authorization: TOKEN" https://hub.docker.com/v2/repositories/USERNAME/ call
• if you need auth for private repos i think its just one step, but introduces concern for airbyte storing credentials because it's http basic auth: curl -u username:password https://auth.docker.io/token?service=registry.docker.io&scope=repository:<repo>:pull

[Phlair]

@evantahler I've sent Docker an email, will update here with response

[Phlair]

Answer from Docker about getting access to additional data that we saw previewed in a Verified Publisher pitch as a UI report:

The short answer to your question is: it is planned to be available via API for Docker Verified Publishers only - I'm anticipating pre-release testing for this in the next few weeks, so fingers crossed it will be very soon.

There is no way to get it programmatically other than that (^) and will be available only via the report until the API ships.

All that said, if you're interested, we'd be happy to connect you to our product manager once you get in the program if you'd like to do some pre-release testing.

[ChristopheDuong]

Dockerhub alpha is available and even in cloud, closing this ticket, thanks @sw-yx!