proto: initial protobuf for config

[vbatts]

To have a less source based, and more consumable example of structures,
this is an initial pass at protobuf structures for the structures from all the current golang source.

There is some exercise needed in platform specific structures.

The Makefile defaults to golang source output, but also has a c target for C source and py target for python as well.

Signed-off-by: Vincent Batts vbatts@hashbangbash.com

[laijs]

Don't put a refactor patch which refactors another patch inside the same pr.

[vbatts]

@laijs eh?

[laijs]

I found you pr has a style-related patch and an indent patch, they are applied for an earlier patch in the same pr, it is bad, the earlier patch should be updated directly to resolve these problem.

[vbatts]

Oh I see, I'll flatten the indentation fixes. Not sure why that wasn't
visible earlier. The style change was more for commentary.
As this PR is not against master but for conversation on the whether this
approach with protobuf is sufficient.
On Sep 14, 2015 22:37, "Lai Jiangshan" notifications@github.com wrote:

I found you pr has a style-related patch and an indent patch, they are
applied for an earlier patch in the same pr, it is bad, the earlier patch
should be updated directly to resolve these problem.

—
Reply to this email directly or view it on GitHub
#185 (comment).

[vbatts]

@laijs I squashed some and moved the indentation fixes to the initial commit

[philips]

The idea is that we use protobuf 3 JSON encodings?

Update: this is what I mean by protobuf encoding: https://developers.google.com/protocol-buffers/docs/proto3?hl=en#json

[vbatts]

https://github.com/google/protobuf/releases/tag/v3.0.0-beta-1 beta was only 28 days ago...

[vbatts]

@philips here is the output with this most recent push:

vbatts@valse ~/src/opencontainers/specs/proto (protobuf) $ make 
protoc --go_out=./go config.proto
protoc --go_out=./go runtime_config.proto
go run ./example.go
{
  "spec": {
    "platform": {
      "os": "linux",
      "arch": "x86_64"
    },
    "process": {
      "env": [
        "TERM=linux"
      ],
      "cwd": "/"
    }
  }
}

[mrunalp]

@vbatts The json looks good 👍

[wking]

On Thu, Sep 24, 2015 at 10:21:17AM -0700, Brandon Philips wrote:

The idea is that we use protobuf 3 JSON encodings?

I've pushed some work along this lines to vbatts#2, which turns
up some problems with Any support in Go.

I've also pushed some proto2 work fleshing out Process to
vbatts#1 which turns up some problems with the proto2 JSON
serialization.

I'll leave it there for now, but I'm happy to help push this forward
in either direction if folks see a way around those issues.

[wking]

On Sat, Sep 26, 2015 at 01:12:07PM -0700, W. Trevor King wrote:

I've pushed some work along this lines to vbatts#2, which
turns up some problems with Any support in Go.

I've also pushed some proto2 work fleshing out Process to
vbatts#1 which turns up some problems with the proto2 JSON
serialization.

I'll leave it there for now, but I'm happy to help push this forward
in either direction if folks see a way around those issues.

One way around the Any+Go issue is to use C++ instead of Go ;), so
I've gone that way with vbatts#3. Differences vs. the current
Markdown-defined spec are the process.user @type (used by Any) 1 and
the runtime.json differences I talk through in
wking/opencontainer-runtime-spec@5f422ee6ae.

[vbatts]

one approach that could work, regardless of protobuf, but could facilitate this kind of multi-platform accommodation, would be to have the canonical configuration be config.json and platform specific be held in config-$keyword.json. That keyworded filename could be a referenced from the config.json, such that tooling could implement repeatable patterns, but something like the spec would not need to define up front config-{linux,windows,etc}.json.

[conqerAtapple]

How about concept of overlaid configurations?

-jojy

On Sep 28, 2015, at 7:51 AM, Vincent Batts notifications@github.com wrote:

one approach that could work, regardless of protobuf, but could facilitate this kind of multi-platform accommodation, would be to have the canonical configuration be config.json and platform specific be held in config-$keyword.json. That keyworded filename could be a referenced from the config.json, such that tooling could implement repeatable patterns, but something like the spec would not need to define up front config-{linux,windows,etc}.json.

—
Reply to this email directly or view it on GitHub.

[wking]

On Mon, Sep 28, 2015 at 08:30:23AM -0700, Jojy George Varghese wrote:

How about concept of overlaid configurations?

Previous discussion along these lines in #73, #74, and #76. My
current feeling on this front is that this spec shouldn't talk about
that sort of thing, which we can always punt to an external
preprocessor 1.

[wking]

On Mon, Sep 28, 2015 at 07:51:04AM -0700, Vincent Batts wrote:

one approach that could work, regardless of protobuf, but could
facilitate this kind of multi-platform accommodation, would be to
have the canonical configuration be config.json and platform
specific be held in config-$keyword.json…

I don't think that works for multi-platform bundles. For example,
process isn't in the platform-specific Linux section, but process.user
is Linux specific. If you process (platform-specific user, likely a
platform specific command), platform, root (likely pointing at a
platform-specific directory), and mounts (likely pointing at
platform-specific target paths) into the platform-specific section,
the only things left in config.json are version and hostname. That's
not much of a win for sharing a cross-platform config ;).

If you're just suggesting it as a way to work around the platform.user
typing, I think protobuf already gives us that.

[conqerAtapple]

Thanks for the history Trevor. If there has to be a way to "include" other configs to form a configuration "union", shouldnt there be a spec so that its uniform and unambiguous?

-jojy

On Sep 28, 2015, at 9:15 AM, W. Trevor King notifications@github.com wrote:

On Mon, Sep 28, 2015 at 08:30:23AM -0700, Jojy George Varghese wrote:

How about concept of overlaid configurations?

Previous discussion along these lines in #73, #74, and #76. My
current feeling on this front is that this spec shouldn't talk about
that sort of thing, which we can always punt to an external
preprocessor 1.

—
Reply to this email directly or view it on GitHub.

[wking]

On Mon, Sep 28, 2015 at 09:48:43AM -0700, Jojy George Varghese wrote:

If there has to be a way to "include" other configs to form a
configuration "union" shouldnt there be a spec so that its uniform
and unambiguous?

I think this is getting pretty off-topic for “what do we think about
this protobuf approach” ;). Maybe move this config-generation
discussion to #73 or a mailing list thread?

[vbatts]

@conqer I tried this, but when importing, it added to the golang source an import ".", so the resulting source was a cyclic dependency. Would need a more sophisticated structure. Also, would still have the XXX_* extension issue for marshalling structures.

[bufdev]

Just as an interested observer, I'd highly recommend going with proto3 syntax, the beta was only 28 days ago but it's pretty highly stable and used in production (and gRPC is really good).

Some things that might help:

go.pedge.io/google-protobuf - takes care of $(which protoc)/../include/google/protobuf files
go.pedge.io/googleapis - takes care of https://github.com/google/googleapis
go get go.pedge.io/tools/protoc-all (https://github.com/peter-edge/go-tools/tree/master/protoc-all) - I'm going to redo this code in a bit, but this is basically how I do any compilation of protos at this point, both for work and personal projects. Takes care of all import paths (I just wrote go.pedge.io/googleapis so I'll add that in soon), and grpc, grpc-gateway, and my own personal go.pedge.io/protolog.

Let me know if I can be of help.

[vbatts]

@peter-edge thanks for the pointers.

[bufdev]

I did this recently too: https://github.com/peter-edge/go-protoeasy

[vbatts]

Further updates on this protobuf. One thing that I am not entirely certain about, is the ability to have variation in the fields of a message.

For example, if there is the ability to introduce new fields in a message type, such that did not conflict with prior versions of the message type. Would decoders of the prior version of the message type decode only the fields known it its version?

Hopefully that question is clear enough.

[bufdev]

Yes, that's basically half the point of protobuf, and why a lot of people use it :) Let me see if I can find some docs to explain better.

[bufdev]

Actually https://developers.google.com/protocol-buffers/docs/overview explains it a little at the bottom in the "A bit of history" section. Most protocol buffer language implementations have the notion of unrecognized fields, that is fields that given the definition of a message it has, do not have matching tag numbers.

When you edit a protocol buffer message, you only ever add fields, and if you do want to delete fields, the general pattern is you always use an increasing field number for new fields, to avoid ever reusing old field numbers.

[bufdev]

Two other things:

1. I would highly recommend moving to proto3 instead of proto2, especially since you are using golang. Oh I just saw I already mentioned this.
2. The makefile could be reduced to protoeasy --cpp --python --go --go-import-path github.com/opencontainers/specs/proto proto, other than the C files (need to check that out) https://go.pedge.io/protoeasy

[vbatts]

I'm increasingly of the mind that going the route of protobuf is too much a hammer-looking-for-a-nail. I think this needs a step back, to solve the actual problem of implementation independent reference and schema.

[mrunalp]

I agree.

Sent from my iPhone

On Dec 16, 2015, at 8:53 AM, Vincent Batts notifications@github.com wrote:

I'm increasingly of the mind that going the route of protobuf is too much a hammer-looking-for-a-nail. I think this needs a step back, to solve the actual problem of implementation independent reference and schema.

—
Reply to this email directly or view it on GitHub.

[duglin]

I tend to agree. Going away from other, simple, choices like json, yml (or heck even xml) feels really odd to me.

[bufdev]

Just the minority opinion here:

Protobufs make dev life way nicer once you figure it out. Really the auto-generation of what amount to DTO objects is enough for a huge win - once you start using them, a lot of code that would otherwise not be easy to document or would be inconsistent is suddenly just generated. Compiling etc is very easy once you understand it.

[LK4D4]

@peter-edge I agree that code-generation is a huge plus. I tried myself and schema much more convenient than json, apart from human-readability of the ready spec itself.

[bufdev]

Protobufs have a lot of other wins too:

• Forwards/backwards compatibility (most people do not care though)
• Serialization size (most people do not care though)
• Generated enums
• Generated services
• https://github.com/grpc/grpc-go
• https://github.com/gengo/grpc-gateway (example:
  https://github.com/peter-edge/go-openflights/blob/0a97dbbd1609a1483183009e7158594a346211cf/openflights.proto#L108
  )
• Some teams are working on automatically generated Swagger doc
• Auto-generated SDKs for other languages (not sure if relevant here though)

[bufdev]

Another instructive example: here's the docker volume API as a protobuf service, generated to grpc and grpc-gateway golang code: https://github.com/peter-edge/go-dockervolume/blob/master/dockervolume.proto

[vbatts]

Closing this. The specification of the configuration file is a json schema question. Even #276 moves in this direction, by having the golang source being a reference to achieve the desired structures.
We have discussed and investigated json-schema, and perhaps may have future developments related to that. But the protobuf investigation was using protobuf primarily for code generation, and this is not a guarantee that can be entirely relied on.

See also #306