Skip to content

Commit

Permalink
WIP: Use local ref-engine discovery for layouts
Browse files Browse the repository at this point in the history
Bump the layout to v1.1 to support this.  This makes it possible to
distribute layouts that use other protocols, for example new
ref-engine protocols or a sharded blob store [1].  You can also
reference external ref- and CAS-engines, although obviously the
utility of such depends on the availability of those external engines.

[1]: opencontainers#449

Signed-off-by: W. Trevor King <wking@tremily.us>
  • Loading branch information
wking committed Sep 18, 2017
1 parent ebd93fd commit 8238f0f
Show file tree
Hide file tree
Showing 4 changed files with 148 additions and 176 deletions.
3 changes: 2 additions & 1 deletion annotations.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ This specification defines the following annotation keys, intended for but not l
* **org.opencontainers.image.vendor** Name of the distributing entity, organization or individual.
* **org.opencontainers.image.licenses** License(s) under which contained software is distributed as an [SPDX License Expression][spdx-license-expression].
* **org.opencontainers.image.ref.name** Name of the reference for a target (string).
* SHOULD only be considered valid when on descriptors on `index.json` within [image layout](image-layout.md).
* SHOULD only be considered valid when on descriptors used for [reference resolution][ref-engines].
* Character set of the value SHOULD conform to alphanum of `A-Za-z0-9` and separator set of `-._:@/+`
* The reference must match the following [grammar](considerations.md#ebnf):
```
Expand Down Expand Up @@ -64,4 +64,5 @@ While users are encouraged to use the **org.opencontainers.image** keys, tools M
| | `schema-version`| No equivalent in the OCI Image Spec |
| | `docker.*`, `rkt.*` | No equivalent in the OCI Image Spec |
[ref-engines]: https://github.com/xiekeyang/oci-discovery/blob/44ec3cf3113e29a743ad04220ccb7ff5197dab2a/ref-engine-protocols.md
[spdx-license-expression]: https://spdx.org/spdx-specification-21-web-version#h.jxpfx0ykyb60
211 changes: 37 additions & 174 deletions image-layout.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
## OCI Image Layout Specification

This is version 1.1.0 of this specification.

* The OCI Image Layout is a slash separated layout of OCI content-addressable blobs and [location-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage#Content-addressed_vs._location-addressed) references (refs).
* This layout MAY be used in a variety of different transport mechanisms: archive formats (e.g. tar, zip), shared filesystem environments (e.g. nfs), or networked file fetching (e.g. http, ftp, rsync).

Expand All @@ -11,195 +13,56 @@ Given an image layout and a ref, a tool can create an [OCI Runtime Specification

# Content

The image layout is as follows:

- `blobs` directory
- Contains content-addressable blobs
- A blob has no schema and SHOULD be considered opaque
- Directory MUST exist and MAY be empty
- See [blobs](#blobs) section
- `oci-layout` file
- It MUST exist
- It MUST be a JSON object
- It MUST contain an `imageLayoutVersion` field
- See [oci-layout file](#oci-layout-file) section
- It MAY include additional fields
- `index.json` file
- It MUST exist
- It MUST be an [image index](image-index.md) JSON object.
- See [index.json](#indexjson-file) section

## Example Layout

This is an example image layout:

```
$ cd example.com/app/
$ find . -type f
./index.json
./oci-layout
./blobs/sha256/3588d02542238316759cbf24502f4344ffcc8a60c803870022f335d1390c13b4
./blobs/sha256/4b0bc1c4050b03c95ef2a8e36e25feac42fd31283e8c30b3ee5df6b043155d3c
./blobs/sha256/7968321274dc6b6171697c33df7815310468e694ac5be0ec03ff053bb135e768
```
An image layout consists of an `oci-layout` file which MUST exist and be a valid [layout object](#layout-object).

Blobs are named by their contents:
## Layout object

```
$ shasum -a 256 ./blobs/sha256/afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51
afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51 ./blobs/sha256/afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51
```

## Blobs

* Object names in the `blobs` subdirectories are composed of a directory for each hash algorithm, the children of which will contain the actual content.
* The content of `blobs/<alg>/<encoded>` MUST match the digest `<alg>:<encoded>` (referenced per [descriptor](descriptor.md#digests)). For example, the content of `blobs/sha256/da39a3ee5e6b4b0d3255bfef95601890afd80709` MUST match the digest `sha256:da39a3ee5e6b4b0d3255bfef95601890afd80709`.
* The character set of the entry name for `<alg>` and `<encoded>` MUST match the respective grammar elements described in [descriptor](descriptor.md#digests).
* The blobs directory MAY contain blobs which are not referenced by any of the [refs](#indexjson-file).
* The blobs directory MAY be missing referenced blobs, in which case the missing blobs SHOULD be fulfilled by an external blob store.

### Example Blobs

```
$ cat ./blobs/sha256/9b97579de92b1c195b85bb42a11011378ee549b02d7fe9c17bf2a6b35d5cb079 | jq
{
"schemaVersion": 2,
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 7143,
"digest": "sha256:afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51",
"platform": {
"architecture": "ppc64le",
"os": "linux"
}
},
...
```

```
$ cat ./blobs/sha256/afff3924849e458c5ef237db5f89539274d5e609db5db935ed3959c90f1f2d51 | jq
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"size": 7023,
"digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270"
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 32654,
"digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0"
},
...
```
This section defines the `application/vnd.oci.layout.header.v1+json` [media type](media-types.md).

```
$ cat ./blobs/sha256/5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270 | jq
{
"architecture": "amd64",
"author": "Alyssa P. Hacker <alyspdev@example.com>",
"config": {
"Hostname": "8dfe43d80430",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "sha256:6986ae504bbf843512d680cc959484452034965db15f75ee8bdd1b107f61500b",
...
```
Content of this type MUST be a JSON object.

```
$ cat ./blobs/sha256/9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0
[gzipped tar stream]
```
The object MUST include an `imageLayoutVersion` entry to provide the version of the image-layout in use.
The value will align with the OCI Image Specification version at the time changes to the layout are made, and will pin a given version until changes to the image layout are required.

## oci-layout file
The object MUST include a `refEngines` entry, [as defined by version 0.1 of the OCI Ref-Engine Discovery specification][ref-engines-objects].

This JSON object serves as a marker for the base of an Open Container Image Layout and to provide the version of the image-layout in use.
The `imageLayoutVersion` value will align with the OCI Image Specification version at the time changes to the layout are made, and will pin a given version until changes to the image layout are required.
This section defines the `application/vnd.oci.layout.header.v1+json` [media type](media-types.md).
The object MAY include a `casEngines` entry, [as defined by version 0.1 of the OCI Ref-Engine Discovery specification][ref-engines-objects].

### oci-layout Example
### Example Layout Object

```json,title=OCI%20Layout&mediatype=application/vnd.oci.layout.header.v1%2Bjson
{
"imageLayoutVersion": "1.0.0"
"imageLayoutVersion": "1.1.0",
"refEngines": [
{
"protocol": "oci-index-template-v1",
"uri": "index.json"
}
],
"casEngines": [
{
"protocol": "oci-cas-template-v1",
"uri": "blobs/{algorithm}/{encoded}"
}
]
}
```

## index.json file

This REQUIRED file is the entry point for references and descriptors of the image-layout.
The [image index](image-index.md) is a multi-descriptor entry point.

This index provides an established path (`/index.json`) to have an entry point for an image-layout and to discover auxiliary descriptors.

* No semantic restriction is given for the "org.opencontainers.image.ref.name" annotation of descriptors.
* In general the `mediaType` of each [descriptor][descriptors] object in the `manifests` field will be either `application/vnd.oci.image.index.v1+json` or `application/vnd.oci.image.manifest.v1+json`.
* Future versions of the spec MAY use a different mediatype (i.e. a new versioned format).
* An encountered `mediaType` that is unknown SHOULD be safely ignored.

Where the above `refEngines` and `casEngines` entries are used, the remainder of a 1.1.0 layout will be identical to [a 1.0.0 layout][layout-1.0.0].

**Implementor's Note:**
A common use case of descriptors with a "org.opencontainers.image.ref.name" annotation is representing a "tag" for a container image.
For example, an image may have a tag for different versions or builds of the software.
In the wild you often see "tags" like "v1.0.0-vendor.0", "2.0.0-debug", etc.
Those tags will often be represented in an image-layout repository with matching "org.opencontainers.image.ref.name" annotations like "v1.0.0-vendor.0", "2.0.0-debug", etc.
### Example Layout

This is an example image layout corresponding to the [example layout object](#example-layout-object)

### Index Example

```json,title=Image%20Index&mediatype=application/vnd.oci.image.index.v1%2Bjson
{
"schemaVersion": 2,
"manifests": [
{
"mediaType": "application/vnd.oci.image.index.v1+json",
"size": 7143,
"digest": "sha256:0228f90e926ba6b96e4f39cf294b2586d38fbb5a1e385c05cd1ee40ea54fe7fd",
"annotations": {
"org.opencontainers.image.ref.name": "stable-release"
}
},
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 7143,
"digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f",
"platform": {
"architecture": "ppc64le",
"os": "linux"
},
"annotations": {
"org.opencontainers.image.ref.name": "v1.0"
}
},
{
"mediaType": "application/xml",
"size": 7143,
"digest": "sha256:b3d63d132d21c3ff4c35a061adf23cf43da8ae054247e32faa95494d904a007e",
"annotations": {
"org.freedesktop.specifications.metainfo.version": "1.0",
"org.freedesktop.specifications.metainfo.type": "AppStream"
}
}
],
"annotations": {
"com.example.index.revision": "r124356"
}
}
```
$ cd example.com/app/
$ find . -type f
./blobs/sha256/3588d02542238316759cbf24502f4344ffcc8a60c803870022f335d1390c13b4
./blobs/sha256/4b0bc1c4050b03c95ef2a8e36e25feac42fd31283e8c30b3ee5df6b043155d3c
./blobs/sha256/7968321274dc6b6171697c33df7815310468e694ac5be0ec03ff053bb135e768
./index.json
./oci-layout
```

This illustrates an index that provides two named manifest references and an auxiliary mediatype for this image layout.


[descriptors]: ./descriptor.md
[layout-1.0.0]: https://github.com/opencontainers/image-spec/blob/v1.0.0/image-layout.md
[ref-engines-objects]: https://github.com/xiekeyang/oci-discovery/blob/44ec3cf3113e29a743ad04220ccb7ff5197dab2a/ref-engine-discovery.md#ref-engines-objects
51 changes: 50 additions & 1 deletion specs-go/v1/layout.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,64 @@

package v1

import (
"encoding/json"
"fmt"
)

const (
// ImageLayoutFile is the file name of oci image layout file
ImageLayoutFile = "oci-layout"
// ImageLayoutVersion is the version of ImageLayout
ImageLayoutVersion = "1.0.0"
ImageLayoutVersion = "1.1.0"
)

type EngineConfig struct {
Protocol string `json:"protocol"`
Data map[string]interface{}
}

// ImageLayout is the structure in the "oci-layout" file, found in the root
// of an OCI Image-layout directory.
type ImageLayout struct {
Version string `json:"imageLayoutVersion"`

RefEngines []EngineConfig `json:"refEngines,omitempty"`
CasEngines []EngineConfig `json:"casEngines,omitempty"`
}

func (c *EngineConfig) UnmarshalJSON(b []byte) (err error) {
var dataInterface interface{}
if err := json.Unmarshal(b, &dataInterface); err != nil {
return err
}

data, ok := dataInterface.(map[string]interface{})
if !ok {
return fmt.Errorf("engine config is not a JSON object: %v", dataInterface)
}

protocolInterface, ok := data["protocol"]
if !ok {
return fmt.Errorf("engine config missing required 'protocol' entry: %v", data)
}

c.Protocol, ok = protocolInterface.(string)
if !ok {
return fmt.Errorf("engine config protocol is not a string: %v", protocolInterface)
}

delete(data, "protocol")
c.Data = data
return nil
}

func (c EngineConfig) MarshalJSON() ([]byte, error) {
var data map[string]interface{}
data = make(map[string]interface{})
for key, value := range c.Data {
data[key] = value
}
data["protocol"] = c.Protocol
return json.Marshal(data)
}
59 changes: 59 additions & 0 deletions specs-go/v1/layout_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
// Copyright 2016 The Linux Foundation
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package v1

import (
"encoding/json"
"testing"

"github.com/stretchr/testify/assert"
)

func TestEngineConfigGood(t *testing.T) {
for _, testcase := range []struct {
JSON string
Expected EngineConfig
}{
{
JSON: `{"protocol":"oci-image-template-v1","uri":"index.json"}`,
Expected: EngineConfig{
Protocol: "oci-image-template-v1",
Data: map[string]interface{}{
"uri": "index.json",
},
},
},
{
JSON: `{"protocol":"nested-array","x-array":[1.2,3.4]}`,
Expected: EngineConfig{
Protocol: "nested-array",
Data: map[string]interface{}{
"x-array": []interface{}{1.2, 3.4},
},
},
},
} {
t.Run(testcase.JSON, func(t *testing.T) {
var config EngineConfig
json.Unmarshal([]byte(testcase.JSON), &config)
assert.Equal(t, config, testcase.Expected)
marshaled, err := json.Marshal(config)
if err != nil {
t.Fatal(err)
}
assert.Equal(t, string(marshaled), testcase.JSON)
})
}
}

0 comments on commit 8238f0f

Please sign in to comment.