Allow for conflicting namespaces: introducing backend prefixes

[wk8]

Is your feature request related to a problem? Please describe.
I have 3 different backends, a, b and c. I'd like Kraken to provide a single registry for all three.

As I understand it, Kraken currently allows to do that as long as the image names (repos and tags) in all three backends obey to some pattern that allows them to be neatly divided into Kraken namespaces. However, for reasons outside of my control, that's not the case for me: any image name can be found in any backend.

I'd still like to be able to configure Kraken to be able to do docker pull localhost:8081/a/my/image:1.0 (assuming the Kraken agent is listening on port 8081), and have Kraken understand that I want to pull my/image:1.0 from backend a; and docker pull localhost:8081/b/my/image:1.0 would pull my/image:1.0 from backend b.

Describe the solution you'd like
A way to achieve that could be to add a new, optional prefix field in backend definitions, that Kraken would understand to be the artificial prefix that it needs to remove when talking to the underlying backend.

The example from the previous section could then be configured as:

backends:
  - namespace: '^a/.*'
    prefix: a/
    backend:
      registry_blob:
        ...

  - namespace: '^b/.*'
    prefix: b/
    backend:
      s3:
        ...

  - namespace: '^c/.*'
    prefix: c/
    backend:
      http:
        ...

That change would enable that new use-case while being completely seamless for existing deployments/configurations.

I would be happy to submit a PR for this as long as maintainers pre-approve the idea.

Thanks.

[evelynl94]

Let me try to rephrase to see if I understand correctly:
my/image:1.0 can exist in backend a, b, c, but you want to distribute the pull requests by adding a prefix from the client side. So localhost:8081/(a|b|c)/my/image:1.0 will pull from a, b, c respectively, correct?

Adding a prefix notion in the top level config can be tricky when one day there is a need to, instead of trimming prefix, adding a prefix for backends. To me, it sounds like the responsibility of the backend itself to decide how to interpret the namespace, and it is probably also much more flexible to do this outside of kraken.

I wonder if a proxy would solve your problem. i.e. kraken will treat the proxy as backend and pull from it. The proxy redirects requests based on path.

[wk8]

@evelynl94 : thanks for your reply! :)

Let me try to rephrase to see if I understand correctly:
my/image:1.0 can exist in backend a, b, c, but you want to distribute the pull requests by adding a prefix from the client side. So localhost:8081/(a|b|c)/my/image:1.0 will pull from a, b, c respectively, correct?

Couldn't have said it better :)

Adding a prefix notion in the top level config can be tricky when one day there is a need to, instead of trimming prefix, adding a prefix for backends.

I'm not sure I understand what you mean here, could you please elaborate?

To me, it sounds like the responsibility of the backend itself to decide how to interpret the namespace, and it is probably also much more flexible to do this outside of kraken.

I wonder if a proxy would solve your problem. i.e. kraken will treat the proxy as backend and pull from it. The proxy redirects requests based on path.

A proxy can indeed do the job, but that adds another moving piece, and another critical service to keep up at all times. Having several backends, with no obvious way to separate them into regex-based namespaces, seemed to me to be a generic enough problem, that enough teams are likely to encounter, that it would make sense to support this scenario in Kraken itself; especially considering that it adds little complexity to it.

[codygibb]

My 2c -- probably the easiest way to do this is by implementing a backend.Client (to be used by build-index to fetch tags) which strips the namespace prefix from the tag name and then calls into a real backend. Something like:

type trimPrefixClient struct {
  prefix string
  backend backend.Client
}

func (c *trimPrefixClient) Download(namespace, name string, dst io.Writer) error {
  namespace = strings.TrimPrefix(namespace, c.prefix)
  name = strings.TrimPrefix(name, c.prefix)
  return c.backend.Download(namespace, name, dst)
}

I'm hesitant to commit this upstream, since it's clearly targeting a very specific use-case. Given that backends are pluggable, this should be fairly easy to support in your own private repo (we do this for numerous internal backends at Uber).

[evelynl94]

Adding a prefix notion in the top level config can be tricky when one day there is a need to, instead of trimming prefix, adding a prefix for backends.

I'm not sure I understand what you mean here, could you please elaborate?

Having prefix in the top level config is probably not ideal because we would be enforcing all backends to interpret prefix the same way (i.e the prefix will be trimmed).

I agree with Cody that this sounds more specific to a custom backend (it could also serve as your proxy layer).

[wk8]

Thanks @codygibb and @evelynl94