oci/cas/dir: Load blob URI from oci-layout

[wking]

With this change, users can configure their blob storage once at init time with an optional --blob-uri. Most other commands (which do not need path → blob conversion) can load the blob location from the oci-layout layout file (the 1.1.0 format is in flight). The only other user-facing change is to umoci gc, which gains a --digest-regexp. Folks who customized their blob URI will need to supply --digest-regexp to reverse whichever blob URI they're using.

This seems like a more convenient interface to me than requiring all callers to provide the custom blob location (#190). And it is more powerful as well, allowing users to shard their blob storage, etc. if they feel moved to do so.

I've also swapped in my external CAS implementation, since that provides the interfaces we could use for read/write CAS distinctions (#185). Obviously you could do all these same things with a local-to-umoci CAS implementation as well. I'd rather put the time into a stand-alone implementaion that can also be used in the oci-discovery context. But it's all Apache 2.0, so we can copy/paste between repos if you see anything you like and aren't interested in an external dependency.

[cyphar]

Absolutely not, Trevor.

This sort of mis-use of "oci-discovery" is precisely why I didn't like it -- you've blended together the concepts of distribution and internal structure so much to the point where a PR like this turns up. What on earth does local (internal) structure of an OCI layout have to do with distribution? Is the plan to allow you to specify alternative places to fetch blobs inside an OCI image? But then how do we deal with mirroring and so on? Is mirroring going to require modifying an image's internal structure, or will there be multiple places you can get this information from (which now just increases the number of systems which have to be in place to do mirroring).

To be clear, I think that allowing for custom internal structures is probably a good thing.

However, I don't like that you're using "oci-discovery" for this purpose. I also don't understand why you've reimplemented oci/cas in the form of wking/casengine and now have made oci/cas just use that -- I don't understand the benefits and it just feels like reinventing the wheel for no real gain.

[wking]

On Fri, Nov 03, 2017 at 06:06:13AM +0000, Aleksa Sarai wrote: What on earth does local (internal) structure of an OCI layout have to do with distribution?

I think we want the ability to customise internal structure, and you seem to agree based see also your “probably a good thing”. Once you have that, clients can no longer hard-code the “where do I look up blob ${DIGEST}?” logic. For example, instead of [1], we'll need [2] or similar. And clients will need some way to figure out the configured custom structure. With the approach in #190, that's via a --shared-cas optionon all umoci commands that work with image layouts. With this PR, it's generally via a casEngines entry in oci-layout. The connection with oci-discovery and casEngines is that “where do I look up blob ${DIGEST}?” is a question that you need to answer when you fetch a discovered image, and it's a question you need to answer when you fetch blobs from a customized local image. I think we end up with a simpler ecosystem if we use the same answer in both cases, instead of inventing two separate answers to the same question.

Is the plan to allow you to specify alternative places to fetch blobs inside an OCI image?

Yes. Isn't that what #190 is about too?

But then how do we deal with mirroring and so on?

CAS mirroring should be really easy, and I don't think this makes it any more complicated. Publishers can drop their CAS blobs wherever they like, and declare casEngines *somewhere* so consumers can find them. With oci-discovery, those locations include, but are not limited to: * XDG configurations on the local box [3]. * The oci-host-ref-engines well-known URI [4]. * A particular ref-engine (e.g. [5]). * An oci-layout object [6,7] (like I'm doing in this PR). What's important about CAS mirroring is that the client can figure out how to retrieve a blob with a given digest. oci-discovery defines a negotiation framework for determining that protocol, it doesn't requiring everybody to use a single protocol.

Is mirroring going to require modifying an image's internal structure, or will there be multiple places you can get this information from (which now just increases the number of systems which have to be in place to do mirroring).

There is no need to modify images before you distribute them. In fact, with the oci-discovery approach, there's no need to hit oci-layout at all. Folks *will* need to have either their ref-engines object [8] or their ref-engine (e.g. [5]) declare useful casEngines, and those declarations may use the same casEngines entry that this PR is pulling from oci-layout. If/when my oci-layout restructuring [6,7] lands, then the oci-layout file will also be usable directly as a ref-engines object.

I also don't understand why you've reimplemented `oci/cas` in the form of `wking/casengine` and now have made `oci/cas` just use that -- I don't understand the benefits and it just feels like reinventing the wheel for no real gain.

Probably for the same reasons you created oci/cas based on opencontainers/image-tools#5 [9]. With image-tools apparently unwilling to adopt a CAS API (at least in the near future), having this live in its own external repo seemed like the easiest way to provide what I feel is the most usable API. And both image-tools and umoci have a broader scope than just: * Defining the CAS-engine interface [10], * Providing a CAS-engine registry [11], * Providing a URI-Template-based reader [12], and * Providing a directory-based read/writer [13]. By keeping that in a more CAS-focused repo, we give folks something they can consume without pulling in things they don't need. This is the same approach taken by other micro-projects like go-digest, and I'm aiming for similar boring stability once the initial dev work settles down. And it's all Apache 2, so folks who are not interested in another external dependency are free to copy/paste into their own Apache 2 projects as they see fit. [1]: https://github.com/openSUSE/umoci/blob/v0.3.1/oci/cas/dir/dir.go#L69 [2]: https://github.com/wking/casengine/blob/3ed08888a9365a2753ab8b809b7efb286566fe8d/dir/engine.go#L200-L209 [3]: https://github.com/xiekeyang/oci-discovery/blob/17aaa9ee7538d1db09b5f142ed319e06dee7407e/xdg-ref-engine-discovery.md [4]: https://github.com/xiekeyang/oci-discovery/blob/17aaa9ee7538d1db09b5f142ed319e06dee7407e/well-known-uri-ref-engine-discovery.md [5]: https://github.com/xiekeyang/oci-discovery/blob/17aaa9ee7538d1db09b5f142ed319e06dee7407e/index-template.md#example [6]: https://github.com/wking/image-spec/blob/ref-engine-discovery-layout/image-layout.md#layout-object [7]: xiekeyang/oci-discovery#20 [8]: https://github.com/xiekeyang/oci-discovery/blob/17aaa9ee7538d1db09b5f142ed319e06dee7407e/xdg-ref-engine-discovery.md#ref-engines-objects [9]: https://github.com/openSUSE/umoci/blob/v0.3.1/oci/cas/README.md [10]: https://godoc.org/github.com/wking/casengine [11]: https://godoc.org/github.com/wking/casengine/read [12]: https://godoc.org/github.com/wking/casengine/read/template [13]: https://godoc.org/github.com/wking/casengine/dir

[cyphar]

This is fairly stale by now, closing. Also now that umoci is being considered to be an OCI Reference Implementation, this sort of feature would be very strongly frowned against.