Gitea: Group repositories under a directory

[vignesh-rajendran]

Dear Gitea team,
Is there any way to group repositories under a directory instead of hosting all the repositories at TOP of an organization?
Any help would be really appreciated.

Thanks and Regards,
Vignesh Rajendran

[lunny]

Maybe we could use topic to do that?

[bkcsoft]

You mean like sub-groups or topics?
Subgroups: #1872
Org/Project/Repo proposal: #363
Repo Topics: #219

[belliash]

Any chance to get this implemented? 👍

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[stale]

This issue has been automatically closed because of inactivity. You can re-open it if needed.

[belliash]

Please implement this.

[typeless]

I want this feature too. But there are still many unclear details about how this would be implemented.

Reminder: https://github.com/go-gitea/gitea/pull/3711/files?file-filters%5B%5D=#diff-d4e2553a87eb8b329efd7335f028903c

[typeless]

FWIW, Google has it that is very close to what I want https://android.googlesource.com/.

[typeless]

Just a brief note.

There are two simple approaches. The first is to allow the Name of models.Repository to contain slashes(/).
The second is to add a new field Namespace to models.Repository.

If we allow slashes in Repository.Name, I imgine that most of the code can be backward-compatible and remain unchanged with the old behivior. And user would not notice the difference when they do not have repositories with slashes.

To emulate the appearance of hierarchical directories, we can filter the repo names with the prefix path when a repo list has to be generated.

The git command can still use the Repository.Name verbatim along with how models.RepoPath currently works.

What remains to be clarified is how to distinguish a repository path from an endpoint when routing requests.
it's unclear how we should resolve the clashed names of a path and a repository too.
Say /a/b/c where c can be a URL path to view the repositories under the directory c or a repository named c under /a/b/.

I think the second approach (with an extra Namespace field) is harder to implement.
It can probably provide more flexibility which I can not clearly envision.

[lafriks]

@typeless allowing / in Name would not work as current routing does not allow that so it will simply not work

[lunny]

Another proposal is a repository could be a group /:org/<:groupname or :reponame>. When it's a group, we will list the group's repositories. The change is small and keep compatible with old version. The permission to a group will be applied to all the repositories on this group.

[typeless]

@lafriks yes, that is a part that requires effort. Hopefully, the ambiguity can be revolved given some supported rules of Macaron. Needs more investigations.

@lunny That sounds good. But I need it to support nested groups as well.
That permission-to-group thing seems to be useful too.
It's also desirable if we can map the group hierarchy to the local locations of the git repositories respectively (each group corresponds to a filesystem directory).

[lunny]

Did you mean subgroup of a group. That sounds complicated.

[typeless]

@lunny Yes.
I agree it would be complicated. Part of the reasons is that hierarchical structures are not very suitable for a relational database.
So, encoding the namespace information into repo names seems to be good enough. And that is how Gitiles (and probably Gitweb) works too.
It would make search operation be O(n). But I suppose that wouldn't be a big deal because we aren't going to handle millions of repositories.

[lunny]

@typeless No. Gitea.com will. :)

[typeless]

@lunny 😆
That is alright. Searching repositories by name, which is string comparisons by itself, is already O(n). Naming it with slashes or not wouldn't be different, I assume.

[typeless]

We would have to introduce a new arrangement of URL routes to avoid the ambiguity related to the repo names. Say, a separator to distinguish the repo name part and the rest.

By the way, slashes should be acceptable in branch names (by vanilla git) too. It appears that branch names are always at the end of the URLs.

[lafriks]

@typeless problem is that in current routing you can't have two wildcard arguments

[typeless]

@lafriks I see. You are absolutely right. So, I am thinking about something like /org/a/repo/path/~commits or /org/a/repo/path/~/commits, where ~ acts as a marker to distinguish endpoints from repo names. I don't know if there is a conventional way of dealing with such a situation.

Regarding routing, it could be making an enhancement to Macaron or migrating to other routers. But at the very first, we have to figure out how to make the paths unambiguous (or if possible at all).

Addendum: by 'umambiguous', I mean whether it's possible to make the routing rules codifiable in an acceptable complexity.

[evur]

A mockup of what this could look like:

[LexaGV]

I offer much better approach - avoid making additional entities w/o need. Organization itself can be "renamed" to smth like "folder" or "group" and we should extend this "group" to be HIERARCHY. So at top level you have tree of groups/folders and underneath - our good, old repos. Obviously, each company/project has own structure, so giving 'em hierarchy, we provide most flexible way to organize things.
As a real example, you name top folder as "my company", subfolder "IoT division", "Project #1" and underneath - repos, corresponding ONLY to this project.
I thought this simple idea had to come FIRST when you implement "organization"!

[rwdj]

@evur What would that look like if there were over a thousand repos in that folder/group?
Use-case: an organization with multiple projects sharing over a thousand libraries (some historical) shoved into a folder to indicate it's a group of libraries.
I'd say it's a good use-case to put these shared libraries into a folder than alongside the projects using them that users want to see more readily.

[evur]

@evur What would that look like if there were over a thousand repos in that folder/group?
Use-case: an organization with multiple projects sharing over a thousand libraries (some historical) shoved into a folder to indicate it's a group of libraries.
I'd say it's a good use-case to put these shared libraries into a folder than alongside the projects using them that users want to see more readily.

If an organization shares over a thousand repositories of a specific type (which is very rare), then you would mostly use the search functionality. With this many repositories, the organization should put them into a (sub)group.
A "folder" (not a little collapsible, but a full submenu) would practically be a subgroup. So I don't see any use case.

My proposal is to add little collapsible groups, where all repositories can belong to one organization (including the URL), but in a more organized way. The use case for that is, if an organization has multiple projects with 2+ repositories each, to keep them in groups so they don't spread over the whole organization.

PS: Here's the text from the issue I had created:

My proposal is to add the possibility to visually group repositories in the organization view.

This could be implemented as named dropdowns containing the repositories.

There could also be an option to automatically generate groups containing repositories with a specific prefix (e. g. "app-").

[kozmanbalint-ardinsys]

Adding the ability to assign labels to repos could do the job as well and then no need to touch urls / routing.

[melato]

The description of this issue is ambiguous and leads to multiple interpretations
of what "grouping" means, as shown by the comments.
There is no use case attached to the issue description to clarify the need and eliminate some of the interpretations. Some interpretations exist as separate issues.

I would like to focus on having hierarchical repositories,
similar to a hierarchical filesystem. This is what "directories" are.
Specifically, it should be possible for two different repositories in different directories to have the same base name.

Go itself groups code in hierarchical packages:
crypto/rand and math/rand are two different packages.
If these packages were git repositories, how would we name them?

The natural way to name them is "crypto/rand" and "math/rand".
But gitea does not allow "/" in repository names. Neither does github.com
Actually, both have hierarchical repositories, but with only two levels, the first one being the username.
But the issue is having hierarchical repositories in a single account, in a single gitea instance.

This is a problem for me. I have many git repositories with the same base name,
in different directories, sometimes by programming language and sometimes by project.
For example java/util.git, go/util.git, project1/doc.git, project2/doc.git

Plain old bare git repositories with ssh access handle this just fine:

	git clone git.example.org:go/util.git

gitolite handles them just as easily. This is what I currently use. I'm evaluating migrating to gitea.

Grouping repositories by labelling them does not solve the issue of having two different repositories with the same base name.

Using a prefix, such as "app-" or "go-" is cumbersome, because the "git clone" command would
include this prefix in the repository name, by default, unless you explicitly remember to specify
a different target directory name. It would basically require the user to escape/unescape repository names and create confusion in figuring out if repository "a-b" should be cloned to directory "b" or "a-b".

names with slashes (/)

Allowing repository names to contain slashes(/) seems a simple enough solution.
Such repositories should be stored in subdirectories in the git/{user}/ folder.
"Unadopted" repositories should be detected in subdirectories

The problem of routing can be addressed by using a wildcard or by escaping slashes in url arguments (when the wildcard is not enough). Common operations like clone should work without escaping slashes:

	git clone gitea.example.org:{user}/a/b/c.git

hierarchical repositories do not need to be listed hierarchically. They can be listed in a flat list.
The list can be searched as needed.

[lunny]

Firstly, the change introduces significant complexity because most parts of the system assume the repository URL follows the {user}/{repo} format.

Secondly, the traditional router format {user}/a/b/c.git can also be accessed as {user}/a/b/c. This creates potential conflicts when there are two repositories, such as {user}/a/b/c and {user}/a. In such cases, certain routes may overlap. This raises additional challenges related to storing these repositories on disk and managing the routing logic effectively, further complicating the implementation.

[wxiaoguang]

Allowing repository names to contain slashes(/) seems a simple enough solution.

I also agree that it is enough, I have some brief ideas to make it work, but we need to reserve some names (for example: issues) for sub-repo names.

[melato]

By "simple", I meant conceptually simple, not that it is trivial to implement.

I found a similar long-standing request about bitbucket: https://jira.atlassian.com/browse/BSERV-3008
Several comments mentioned this issue preventing them from migrating from gitolite (my use case).
One of the comments mentions a list of numerous chromium git repositories with slashes in their names: https://chromium.googlesource.com/

I couldn't find any discussion as to why github doesn't support slashes in repo names.

[wxiaoguang]

I couldn't find any discussion as to why github doesn't support slashes in repo names.

The situation applies to most softwares: when the system grows bigger and bigger, some rules have been hard-coded everywhere, it becomes more and more difficult to change these rules. It needs a lot of work and time, and needs some people who have time and ability to push it to move on .......

[melato]

I now think it's best to not allow slashes in repo names, for the following reasons:

• Compatibility with github and other systems. Why be different and introduce incompatibility, when the current rules are so entrenched?
• Too many changes. Besides the web UI, the API has numerous methods which put repository names in the url.
• There are other ways of achieving the original need.

For migrating from gitolite to gitea:

• The first level of a hierarchical repository name is best mapped to organizations. If everything was mapped to a single account (or org), repository a/b would be mapped to {user}/a/b in gitea, which would not match the original gitolite usage.
• For the remaining hierarchy levels, I would map the path separator (/) to "--", or something similar that does not appear in one's repositories. The allowed special characters "-", "_", and "." are all frequently used in repository names (look at gitea's go.mod).

One can write simple client tools to handle "--" as a path separator. I am writing such a tool that:

• Wraps "git clone" so that cloning git.example.com/x/a--b.git becomes:
  git clone git.example.com/x/a--b.git b
  This can also be easily done manually, without any tool.
  It could also handle git.example.com/x/a/b.git:
  git clone git.example.com/x/a--b.git b
  I could clone the same hierarchical paths that I'm used to.

• Flattens a hierarchy of bare git repositories by replacing path separators with "--". I can then have gitea adopt them.

• Replaces "/" with "--" in existing remote urls in local git repositories
  This can also be done manually, but it may be annoying
  to do repeatedly for many repositories.

Migrating from gitolite to gitea involves more than just handling slashes in repository names. Re-configuring access control is a more serious issue.