Package Registry related ToDo list

[KN4CK3R]

Now that #16510 got merged there are still some ToDos.

General:

• Add a webhook for packages without repository relation (Decouple HookTask from Repository #17940, Delete only HookTasks of the repository #17981)
• Enable/Disable packages on organisation level
• Enable/Disable packages on repository level (disable linking packages to that repository and hiding the packages tab)
• Fine-granular access model (Permissions for package repositories #20596)
• Deletion policy (prevent deletion after x hours or something, Add support for npm unpublish #20688 (comment))
• Proxy for remote package registries (Support proxy registries for each package type #21223)
• Automated deletion of packages (Automated deletion for images in the container registry #20514, Data retention settings for package registry #20766)
• npm search support (Package Registry for NPM doesn't support search API endpoints #20098, NPM Package Registry search API endpoint #20280)
• Set repository link from package metadata (npm packages: respect "repository" field to automatically set repository link of uploaded packages #20146, Automatically link NuGet package to repository based on RepositoryUrl #20912, Pushing docker image appears in organization packages, but not in sub project packages #26065, [Feature] Automatically connect a package release to a specific repositiory (specifically looking at container registries right now) #28540)
• Display if a repo has linked packages in repo lists (asked in Missing info about docker image size #20909)
• Support for scoped access tokens (Support scoped access tokens #20908)

Package Types:

• Alpine packages (Add Alpine package registry #23714)
• Ansible packages ([Feature] Ansible Galaxy package manager #29483)
• Cargo / Rust packages (Add Cargo package registry #21888)
• Chef (Add Chef package registry #22554)
• Conda packages (Add Conda package registry #22262)
• CRAN packages (Add CRAN package registry #22331)
• Debian packages (Debian apt package & yum package registry #20751, Add Debian package registry #22854)
• Flatpak (Flatpak package registry #22415)
• Go packages (GOPROXY protocol support #7608, Add Go package registry #24687)
• JSR packages (https://jsr.io/docs/api#jsr-registry-api)
• Pub packages (Support for pub packages #19421)
• RPM packages (Debian apt package & yum package registry #20751, Add RPM registry #23380)
• Swift (Add Swift package registry #22404)
• Terraform (requested by Add Package Registry #16510 (comment))
• Vagrant boxes (Add support for Vagrant packages #20930)

[lunny]

https://github.com/boskiv/go-open-registry is a cargo registry Golang implementation .

[lunny]

You can also consider add #7608 to todo list.

[KN4CK3R]

https://github.com/boskiv/go-open-registry is a cargo registry Golang implementation .

That project still uses a git repo as storage. We could use that approach too but at the moment we can't set a repository to "hidden" to hide it from the normal repository view pages etc.

[wxiaoguang]

It seems that packages/packages.go:353:Cleanup need to be polished.

Everytime I run Gitea, I see the error 😂

2022/04/01 10:48:27 ...packages/packages.go:353:Cleanup() [E] [624667fb-15] hier2

[KN4CK3R]

Everytime I run Gitea, I see the error 😂

😢 #19293

[wxiaoguang]

Some links in i18n are 404 (update: duplicate with justusbunsi's #19315 )

[mscherer]

One feature that I think would benefit public forges is adding a quota system (eg, make sure a user/org can't take all the storage). While the problem already exist for repositories (folks can just commit tarball after all), I think binary packages would make it worst and limits would be easier to reach by regular users.

[justusbunsi]

Not sure if this feedback should be in this issue. If not, let me know:

• Linking packages to a repository seem to not work; neither inside an organization nor for user related packages. The dropdown list cannot be opened.
• Docs reference pointing to https://docs.gitea.io/en-us/<type>/ should be https://docs.gitea.io/en-us/packages/<type>/ (Fix links to packages documentation #19315)

NPM/Yarn

• Support yarn add <package>
• Packages with dependencies not published along with package inside Gitea will prevent a successful npm install <package>

Container Images

• Pull command in package overview contains port 3000

[KN4CK3R]

Linking packages to a repository seem to not work; neither inside an organization nor for user related packages. The dropdown list cannot be opened.

That should work (for public repositories). There may be a bug which hides private repos but I could not verify that. Does the html source contains the repo names?

Support yarn add <package>

What do you mean? As addition to the npm install instruction?

Packages with dependencies not published along with package inside Gitea will prevent a successful npm install <package>

I don't think the registry should prevent that. Isn't that up to the user?

Pull command in package overview contains port 3000

That's correct and ok. If you use a "normal" port like 80 or 443 only the url is displayed. But if the registry uses a different port it needs to be specified.

gitea/modules/setting/packages.go

Lines 34 to 37 in be9ef15

	Packages.RegistryHost = Domain
	if (Protocol == HTTP && HTTPPort != "80") || (Protocol == HTTPS && HTTPPort != "443") {
	Packages.RegistryHost += ":" + HTTPPort
	}

[justusbunsi]

That should work (for public repositories). There may be a bug which hides private repos but I could not verify that. Does the html source contains the repo names?

I configured everything (org + repo) as private. That would explain it.
Regarding html source I need to check again. I remember there was a value for the drop down list. But nothing was visible in the UI.

What do you mean? As addition to the npm install instruction?

Yes. I had issues with using yarn add instead of npm install. I'll have another closer look at that part. Maybe I have some misconfiguration which also causes the dependency issue.

If you use a "normal" port like 80 or 443 only the url is displayed.

Thats odd. I ran the latest dev docker image of Gitea and had a reverse proxy in front of it. I even configured ROOT_URL. So my instance is available at https://git.example.com (port 443) and I am able to push the image to this endpoint. But inside the pull command port 3000 is shown.

[BaseCrusher]

I don't know if I am right with posting this here but I hope so.
Using the package registries (at least the container registry. I haven't tested the others) while the user is accessing gitea via LDAP or OIDC makes it unclear what docker/podman login should be used, since the gitea username does not have to be the same as the oidc username.
I for myself did not manage to push images with an oidc user. But this could be due to an error of mine, where I wasn't logged in correctly. Either way this case should be addressed in the docs if it works with OIDC/LDAP and if not there should be an option for those users to use the registries.

[KN4CK3R]

That's a good question. You can use a token to login in this case. Simply generate a token for that account and use docker login <Gitea username> <token>.

I will add it to the docs.

Edit: Here is the PR #20561 @BaseCrusher

[OverkillGuy]

Suggested feature request for package registry: Support of proxy registries for all package types.

These are package registries that proxy an external source of packages, hence configured with proxy URL, but are otherwise indistinguishable from local package registries.

Example: A local Pypi.org proxy. Local build system would use both the private package registry for "internal" (private) packages, but fetch dependencies on Pypi.org through Gitea too.

Advantages:

• Shorter round-trip to fetch packages = faster build times
• Improved auditability of dependencies
• Offline-able build systems helps with disaster recovery, privacy...
• Mitigate rogue updates

This feature in Docker repositories would remove any need for Dockerhub ECR mirror, which many have to set up to avoid Dockerhub's recent rate-limiting.

The canonical example of the feature is in JFrog's Artifactory.

Effectively, Gitea would, for these proxy repositories, become a local package cache. The biggest technical decision is about when to invalidate cache (docker image's "latest" tag moves pretty quickly, but if you already have a local copy, do you serve it as-is? even if you got it 2 years ago?)

I understand this feature can be a big investment, and acknowledge that there may be no particular need for it. I mostly envy the feature, and wish for Gitea to succeed by out-executing Artifactory, given the new Package Registry is already encroaching on that a bit.

Should I spin off this request into a standalone issue? Drop the request as too big? Happy either way, just want to surface this potential usecase that's now in reach.

EDIT: Just noticed the related ticket around "Go-proxy" which sounds like it touches on same vibes(?)

EDIT2: Standalone ticket is now: #21223

[lunny]

A standalone issue is better.

[kotovalexarian]

Rust (Currently no idea how to implement this one as a normal git repo is used as registry)

Are you sure? As I remember, it's just an archive. This page also doesn't mention Git as a necessity: https://doc.rust-lang.org/cargo/appendix/glossary.html

I can try to implement the support of Cargo packages. I'll look at the code soon. I see that no one tried yet.

[KN4CK3R]

Yep, I'm sure. Have a look here:
https://doc.rust-lang.org/cargo/reference/registries.html#index-format

The index is stored in a git repository so that Cargo can efficiently fetch incremental updates to the index.

But you don't have to spend time on that. I already implemented the Cargo registry but did not push it yet because I wait until #21658 is merged. I need the UI part for Cargo too and don't want to add that in the PR too.

Here is a screenshot (ignore the install instructions, that's c&p from NuGet):

[kotovalexarian]

I've tried RubyGems repository. Looks like it's incomplete. While Bundler succeeded to install gems when their exact versions were specified in Gemfile.lock, it was not able to install gems without a lockfile. I'll try to investigate this, create an issue with steps to reproduce, and maybe fix this.

[kotovalexarian]

Also I want to add Gitea configuration option for custom domain for packages to display it in the interface. It works well for both gem downloading and pushing with the following Nginx configuration:

server_name rubygems.example.com;

location / {
  # "gitea" is an upstream
  proxy_pass http://gitea/api/packages/rubygems/rubygems/;
  proxy_set_header Host 'git.example.com';
}

I don't want to create an issue. I'll try to implement this soon.

P.S. However I really should test if that's not the reason why Bundler doesn't work properly.

[KN4CK3R]

I added the Cargo registry with #21888.

[lafriks]

Would be nice to be able set specific package registry as public (without authorization) for private instances (always requiring authorization)

[lafriks]

Automatic container linking to repository would also be nice if container has source url label

[KN4CK3R]

That's already on the ToDo list (Set repository link from package metadata).

[dr-carrot]

Is there an api endpoint available to link the package to a repository? I don't see anything in swagger

[KN4CK3R]

Not yet. May come bundled with Set repository link from package metadata

[viceice]

I would like to have readonly virtual package registries (npm /nuget), with lists all accessable packages by suppplied credentials. so i don't need to configure multiple registries (eg one for each organization).

somehow related to:

• Support proxy registries for each package type #21223

[jcgruenhage]

I might be wrong here, is this the right place to suggest additional package types?

I'd be interested in Ansible Galaxy support (existing implementations are https://github.com/briantist/galactory, https://github.com/ansible/galaxy, https://github.com/ansible/galaxy_ng), primarily for hosting ansible collections. The ability to host stand-alone roles is more of a legacy feature anyway.

[ExplodingDragon]

Will the Alpine, Debian, and RPM repositories be available upon the release of version 1.19.3? I noticed that the pull request has been open for a month now.

[KN4CK3R]

No, we backport only fixes. As they are new features, they will available with 1.20 (if someone reviews them). You can help review the code and test the functionality.