image-layout: move `refs` to `refs/tags` #219

philips · 2016-08-29T16:44:57Z

Based on the confusion from Antonio in the image-layout in
#173 we are going to
move everything that was in refs to refs/tags. This does a few
things:

It makes it clearer that this layout is for a single "image name" instead of allowing for lots of different refs.
It matches the layout of git so it likely matches existing expectations.

This was also discussed during the OCI F2F and decided to be a
reasonable idea.

Signed-off-by: Brandon Philips brandon.philips@coreos.com

Based on the confusion from Antonio in the image-layout in opencontainers#173 we are going to move everything that was in `refs` to `refs/tags`. This does a few things: 1. It makes it clearer that this layout is for a single "image name" instead of allowing for lots of different refs. 2. It matches the layout of git so it likely matches existing expectations. This was also discussed during the OCI F2F and decided to be a reasonable idea. Signed-off-by: Brandon Philips <brandon.philips@coreos.com>

philips · 2016-08-29T16:45:16Z

cc @runcom @s-urbaniak @vbatts @stevvooe

runcom · 2016-08-29T17:13:40Z

Great! I like refs/tags also, thx @philips. LGTM (IANAM)

wking · 2016-08-29T17:45:04Z

On Mon, Aug 29, 2016 at 09:44:58AM -0700, Brandon Philips wrote:

Based on the confusion from Antonio in the image-layout in
#173 we are going to
move everything that was in refs to refs/tags. This does a few
things:

It makes it clearer that this layout is for a single "image name"
instead of allowing for lots of different refs.

Previous discussion of this around 1. And the current spec has (and
I like) 3:

No semantic restriction is given for object names in the refs
subdirectory.

I see no reason to back away from that at the image-layout level. And
I don't see why there would be a problem with folks keeping all
their refs in a single image-layout (although they probably wouldn't
want to back it with tar ;). For example:

https://oci.example.com/refs/…
https://oci.example.com/blobs/…

could have multiple debian:… refs, multiple ubuntu:… refs, multiple
python:… refs, etc.

Folks with special per-blob auth rules might choose to use per-"image
name" image-layout instances, but and there's no reason you couldn't
do that now. But I think we want to keep support for multi-"image
name" image-layout instances too.

It matches the layout of git so it likely matches existing
expectations.

Git also has refs/heads and refs/remotes. Do we expect image-layout
to need those, or other kinds of refs? I don't see a use case for
them, excepting possibly a refs/HEAD working like Git's .git/HEAD to
provide a default entry point for folks running an image with multiple
refs but not specifying a ref directly.

 Subject: Misleading language in image-layout.md

sgotti · 2016-08-29T18:26:31Z

I was also thinking that the oci image layout could also work for multiple images and the ref is not tied to a name (or just a tag) but it was a level below image name. For example you can just create a simple image "registry" using an oci image layout using a filesystem backed by http, nfs, local fs access etc...
Instead retrieving which location and ref refers to example.com/app:tag will be an upper layer that does image discovery.

I also tried to explain some possibilities of this approach in the Distribution point concept for rkt (rkt/rkt#2953) where there are two different distribution types: a direct distribution point called oci-image-layout and one or more theoretical (not currently covered by the image spec) indirect (that will do location and ref discovery starting from an image name) oci distributions.

vbatts · 2016-08-29T19:35:07Z

LGTM

philips · 2016-08-29T20:52:06Z

@wking the problem is that we don't allow slashes or colons in ref names to ensure we don't confuse filesystems. Without those separators it is strange to imagine inventing some sort of serialization from the common debian/base:v0.3.0 pattern or gcr.io/foo/bar:v1.0 into the refs directory.

wking · 2016-08-29T21:06:28Z

On Mon, Aug 29, 2016 at 01:52:07PM -0700, Brandon Philips wrote:

@wking the problem is that we don't allow slashes or colons in ref
names to ensure we don't confuse filesystems. Without those
separators it is strange to imagine inventing some sort of
serialization from the common debian/base:v0.3.0 pattern or
gcr.io/foo/bar:v1.0 into the refs directory.

There's a straightforward fix for that in #174. Even without that
fix, there are still reasonably straightforward ways to put multiple
names in one image-layout (moby/moby#25779).

But regardless, “we're not sure how to name refs when there are
multiple image names in one image-layout” doesn't seem like it
directly motivates “let's add a tags/ directory”. It sounds more like
motivation for “let's add semantic restrictions to refs”, and I don't
think image-layout is the place to add those restrictions. I'm fine
with a higher-level policy section in image-spec that adds these
restrictions.

Folks who can find an ref naming scheme that they're comfortable with
([1], #174, moby/moby#25779) should be allowed to pack several
image-names into a single image-layout without breaking image-layout
compliance (although they may break image-spec compliance).

stevvooe · 2016-08-29T21:07:09Z

image-layout.md

@@ -1,13 +1,13 @@
 ## Open Container Initiative Image Layout Specification

-The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) references (refs).
+The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) reference tags (tags).


Does this mean we are dropping the concept of generic refs? I thought we were just calling out a "Reserved" structure within refs?

On Mon, Aug 29, 2016 at 02:07:10PM -0700, Stephen Day wrote:

@@ -1,13 +1,13 @@

Open Container Initiative Image Layout Specification

-The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and location-addressable references (refs).
+The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and location-addressable reference tags (tags).

Does this mean we are dropping the concept of generic refs? I
thought we were just calling out a "Reserved" structure within
refs?

Ugh. In order of my decreasing preference:

a. Keep our current generic refs 1.
b. Replace generic refs with refs that have some required tag
semantics (this PR?).
c. Support both generic refs and refs with required tag semantics
(what you seem to be suggesting with the “Reserved structure”
comment).

Any type information about refs should go in the descriptor media
type field. I see no reason to bake it into the bucket that holds the
descriptors.

What is the use case of "generic refs"? I thought we were just trying to clear up the use case of the refs directory by making the tags use case explicit through naming, as git does.

The alternative proposal we discussed was just renaming "refs/" to "tags/".

On Mon, Aug 29, 2016 at 02:24:21PM -0700, Brandon Philips wrote:

What is the use case of "generic refs"?

It allows image-layout users to link link other content if they see a
use case for it (vs. requiring refs to point to a manifest(-list) with
some standard name mapping). For example, Git has a tagged blob 1,
because that was a convenient way to distribute Junio's public key.

Having multiple image-names in one image-layout is a similar use-case
that I think we want to allow at the image-layout level.

How is calling this a "tag" stopping you from having multiple image-names? I view this as similar to how GitHub has the gh-pages branch. We aren't restricting the use cases but trying to clearup the intended use case.

On Mon, Aug 29, 2016 at 03:02:46PM -0700, Brandon Philips wrote:

I don't know. I simply don't understand why having a directory
called refs/tags is confusing.

You'll have people talking about refs and people talking about tags
and people wondering if there's some distinction between them (which
it sounds like there isn't).

What is your use case for having lots of refs pointing to different
images in a single image-layout?

I'm publishing a bunch of images on example.com, and don't want to be
bothered with maintaining multiple image-layout instances 1. Or I
want a convenient way to distribute my public key along with the
images it signs 2. And supporting generic refs protects
image-layout implementations from having to worry about scope creep
and such if a more opinionated refs spec decided that it needed to
support something different from the narrow use case it initially
picked.

rkt image import --name mystuff.com/foo/bar /path/to/oci-image-layout
docker import --name mystuff.com/foo/bar /path/to/oci-image-layout

In these cases the name and tags make sense. What is your use case?

I'd rather keep both explicit:

$ my-tool import LAYOUT-PATH REF [NAME]

unless we decide to support something like HEAD 1, in which case REF
would also be optional. If name was not specified on the command
line, the name used for the imported object would come from some
verifiable name (e.g. using the Merkle objects discussed in #176, see
3).

On Mon, Aug 29, 2016 at 3:16 PM W. Trevor King notifications@github.com
wrote:

I'd rather keep both explicit: $ my-tool import LAYOUT-PATH REF [NAME]
unless we decide to support something like HEAD [1], in which case REF
would also be optional. If name was not specified on the command line, the
name used for the imported object would come from some verifiable name
(e.g. using the Merkle objects discussed in #176
#176, see [3]). [1]: #219
(comment)
#219 (comment)
[2]: #219 (comment)
#219 (comment)
[3]: #173 (comment)
#173 (comment)

Eh, I think that it is going to be more common for someone to download "an
entire image named foo" then wanting to import the whole thing and expect
the tags to show back up rather than forcing the user to specify every ref.
For example you would actually need to do:

my-tool import /path/to/layout my-img-foo-bar-v1.0.0 my.img/foo/bar v1.0.0
my-tool run my.img/foo/bar:v1.0.0

vs

my-tool import /path/to/layout my.img/foo/bar
my-tool run my.img/foo/bar:v1.0.0

I feel there is a usability cost to supporting lots of imgs in a single
layout on this common import UX.

On Mon, Aug 29, 2016 at 03:24:32PM -0700, Brandon Philips wrote:

Eh, I think that it is going to be more common for someone to download "an
entire image named foo" then wanting to import the whole thing and expect
the tags to show back up rather than forcing the user to specify every ref.
For example you would actually need to do:

my-tool import /path/to/layout my-img-foo-bar-v1.0.0 my.img/foo/bar v1.0.0
my-tool run my.img/foo/bar:v1.0.0

vs

my-tool import /path/to/layout my.img/foo/bar
my-tool run my.img/foo/bar:v1.0.0

I feel there is a usability cost to supporting lots of imgs in a single
layout on this common import UX.

So add an:

$ my-tool import-all LAYOUT-PATH

that imports all the refs (still getting names from the CAS blobs
1). Or, without a HEAD spec, make:

$ my-tool import LAYOUT-PATH

(without an explicit REF) import all the refs.

Specifying an unverified name on the command line or extracting
information from the ref name (vs. the referenced CAS content)
should not be part of your main UX.

How would you get a name from a CAS blob?

On Mon, Aug 29, 2016 at 04:19:33PM -0700, Brandon Philips wrote:

How would you get a name from a CAS blob?

See #176 and 1.

philips · 2016-08-29T23:33:41Z

I was perfectly fine with the status quo of having refs be flat with no implied use case. If we can just add some clarifying language about them mapping to tags in a general case for import/export from docker engines would that be sufficient @runcom ?

runcom · 2016-08-29T23:41:29Z

@philips, yes that was my plan at the beginning (just a language issue)

philips · 2016-08-31T02:02:52Z

I am going to close this out and open something that tries to clear up the language issue instead.

vbatts · 2016-09-15T14:00:38Z

@philips was a new issue opened for this yet?

wking · 2016-09-15T14:18:34Z

On Thu, Sep 15, 2016 at 07:00:39AM -0700, Vincent Batts wrote:

@philips was a new issue opened for this yet?

#225, since merged.

vbatts · 2016-09-15T14:30:55Z

nah. the use of ./refs/tags/foo

wking · 2016-09-15T14:42:52Z

On Thu, Sep 15, 2016 at 07:30:55AM -0700, Vincent Batts wrote:

nah. the use of ./refs/tags/foo

Why would you do that? Can't higher levels pick a convention like
tag-foo just like 1 if they're worried about mixing different sorts
of refs?

vbatts · 2016-09-15T14:49:48Z

like has been mentioned before, for the same reason git does it. a tag is just one reference type. there could be others. this is the same thing i said yesterday on the call.

wking · 2016-09-15T14:54:09Z

On Thu, Sep 15, 2016 at 07:49:49AM -0700, Vincent Batts wrote:

like has been mentioned before, for the same reason git does it. a
tag is just one reference type. there could be others.

And higher levels can namespace as many as they like using the
existing set of allowed characters. Why special case a slash for this
one class of refs?

vbatts · 2016-09-15T14:57:27Z

sometimes i'm not even sure the point you're trying to make. Honestly.

wking · 2016-09-15T15:15:32Z

On Thu, Sep 15, 2016 at 07:57:28AM -0700, Vincent Batts wrote:

sometimes i'm not even sure the point you're trying to make. Honestly.

We currently have one class for all refs. Higher levels which want to
divide that into multiple classs can already namespace however they
wish as suggested here 1. That approach works for “I want a
subclass for just base-64-encoded refs” (you prefix with
‘org.prefix.base64-’ or ‘base64-’ or whatever). It also works for “I
want a subclass for just tags” (you prefix with ‘org.prefix.tag-’ or
‘tag-’ or whatever). This is the same approach taken by RFC 5890 with
A labels (prefixed with ‘xn--’) 2.

If you instead bless a subclass with a slashed name (like ‘tags/’)
associated tooling and APIs need to grow to handle the additional
class. That seems like more complication then it's worth, when you
could accomplish the same goal by defining a ‘tag-’ prefix and make it
easy for tooling to handle your refs as opaque bytes without having to
grow special slash handling.

So I'm suggesting we either leave the spec as it is on this point or
define a ‘tag-’ prefix, but that we don't define a ‘tag/’ prefix.

philips added this to the v0.5.0 milestone Aug 29, 2016

stevvooe reviewed Aug 29, 2016
View reviewed changes

philips closed this Aug 31, 2016

wking mentioned this pull request Oct 10, 2016

reference management in image layout #374

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

image-layout: move `refs` to `refs/tags` #219

image-layout: move `refs` to `refs/tags` #219

philips commented Aug 29, 2016

philips commented Aug 29, 2016

runcom commented Aug 29, 2016

wking commented Aug 29, 2016

sgotti commented Aug 29, 2016

vbatts commented Aug 29, 2016 •

edited by caniszczyk

Loading

philips commented Aug 29, 2016

wking commented Aug 29, 2016

stevvooe Aug 29, 2016

wking Aug 29, 2016

philips Aug 29, 2016

wking Aug 29, 2016 •

edited

Loading

philips Aug 29, 2016

wking Aug 29, 2016

philips Aug 29, 2016

wking Aug 29, 2016

philips Aug 29, 2016

wking Aug 29, 2016

philips commented Aug 29, 2016

runcom commented Aug 29, 2016

philips commented Aug 31, 2016

vbatts commented Sep 15, 2016

wking commented Sep 15, 2016

vbatts commented Sep 15, 2016

wking commented Sep 15, 2016 •

edited

Loading

vbatts commented Sep 15, 2016

wking commented Sep 15, 2016

vbatts commented Sep 15, 2016

wking commented Sep 15, 2016

image-layout: move refs to refs/tags #219

image-layout: move refs to refs/tags #219

Conversation

philips commented Aug 29, 2016

philips commented Aug 29, 2016

runcom commented Aug 29, 2016

wking commented Aug 29, 2016

sgotti commented Aug 29, 2016

vbatts commented Aug 29, 2016 • edited by caniszczyk Loading

philips commented Aug 29, 2016

wking commented Aug 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Open Container Initiative Image Layout Specification

Choose a reason for hiding this comment

wking Aug 29, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philips commented Aug 29, 2016

runcom commented Aug 29, 2016

philips commented Aug 31, 2016

vbatts commented Sep 15, 2016

wking commented Sep 15, 2016

vbatts commented Sep 15, 2016

wking commented Sep 15, 2016 • edited Loading

vbatts commented Sep 15, 2016

wking commented Sep 15, 2016

vbatts commented Sep 15, 2016

wking commented Sep 15, 2016

image-layout: move `refs` to `refs/tags` #219

image-layout: move `refs` to `refs/tags` #219

vbatts commented Aug 29, 2016 •

edited by caniszczyk

Loading

wking Aug 29, 2016 •

edited

Loading

wking commented Sep 15, 2016 •

edited

Loading