Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Runtime CLI Spec #513

Closed
wants to merge 1 commit into from
Closed

Conversation

wking
Copy link
Contributor

@wking wking commented Jul 6, 2016

In today's meeting I said I'd PR this, so here it is. I've kept the history to preserve previous review and contribution information. I've tried to summarize any discussion that happened elsewhere in the commit messages, but there may be additional background in the old repo's PRs.

Fixes #434.

@wking
Copy link
Contributor Author

wking commented Jul 6, 2016

I think this should replace #511.

@wking wking force-pushed the command-line-api branch from 58c8fa1 to cfdb790 Compare July 6, 2016 19:05
@wking
Copy link
Contributor Author

wking commented Jul 6, 2016

Travis errors are due to whitespace in other folks' commits 1 which
is no longer present in the tip. I can rebase the whitespace out if
folks need that and the authors ok the change, but I'd rather leave
that history and its commit timestamps alone.

@wking wking force-pushed the command-line-api branch 4 times, most recently from 8aa3002 to 2424618 Compare July 7, 2016 04:37
@wking
Copy link
Contributor Author

wking commented Jul 7, 2016 via email


## Commands

### version
Copy link
Member

@mikebrow mikebrow Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The runc implementation for outputting version information is runc --version or runc -v

I'm not a fan of the way it's implemented in runc, but there it is.

Suggest moving the version command out for now. Or possibly specifying that this particular command could be implemented with a global option such as -v.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Fri, Jul 08, 2016 at 09:50:59AM -0700, Mike Brown wrote:

+### version

The runc implementation for outputting version information is runc --version or runc -v

I'm not a fan of the way it's implemented in runc, but there it is.

Suggest moving the version command out for now.

Can't we just wrap runC in an argument-translator for testing? “Print
your version and exit” sounds very command-like to me, and I'd rather
not overhaul the initial two sections here to say “and when you use a
command-like global option, you don't need a COMMAND”.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For what it's worth, --version is the standard way GNU/Linux (or even UNIX commands in general) operate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Fri, Jul 08, 2016 at 09:12:46PM -0700, Aleksa Sarai wrote:

+### version

For what it's worth, --version is the standard way GNU/Linux (or
even UNIX commands in general) operate.

I think that's because most GNU/Linux/Unix commands are usually a
single command, not an umbrella with a number of subcommands.
Commands that are setup that way tend to support these sorts of things
using unhyphenated forms, although there's not a lot of consistency
:p.

  • git(1) supports both --version and ‘version’, --help and ‘help’.
  • btrfs(8) supports both --version and ‘version’, --help and ‘help’.
  • ip(1) supports -V / -Version and ‘help’.
  • npm supports ‘version’ and ‘help’.
  • pip supports ‘--version’, --help and ‘help’.

I'm not sure any of those commands are aiming for spec-level
documentation. For example, git(1) has:

git [--version] … []

even though you don't need a when you set --version.

It seems like an easy thing to wrap either way (and if I was writing a
runtime I'd support both), but with a ‘version’ command we don't need
the “but you don't have to specify a command if you use one of these
command-like global options” caveat in this document.

Copy link
Member

@mikebrow mikebrow Jul 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like you like the idea of supporting version via --version and/or version command. Let's do that. But it should mention that being runc replaceable is --version in 1.0.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Sun, Jul 10, 2016 at 06:45:27AM -0700, Mike Brown wrote:

+### version

Sounds like you like the idea of supporting version via --version
and/or version command. Let's do that.

In order of decreasing preference for this spec:

  1. Require a ‘version’ command.
  2. Require a ‘--version’ global option, and explain that it is a
    command-like option and that the interaction between command-like
    options and actual commands (e.g. ‘funC --version state …’) is
    unspecified.
  3. Require either 1 or 2.
  4. Require both 1 and 2.

3 and 4 are at the bottom of my list, because once we have either 1 or
2 spelled out in this spec, I'd rather not complicate things by
spelling out the other. And with 3, all consumers relying on this
spec will require a “try one and fall back to the other if the first
fails” wrapper to reliably determine the runtime version.

I'm in favor of the runtime supporting both 1 and 2, but this spec
is just about the minimum API we require for compliance testing.
And the fewer requirements we make, the better. Here is what
compliance would look like for various runtime implementations:

  • Runtime provides ‘version’: compatible with 1, 3, and 4.
  • Runtime provides ‘--version’: compatible with 2, 3, and 4.
  • Runtime provides both: compatible with 1, 2, 3, and 4.
  • Runtime provides neither: compatible with none.

Copy link
Member

@mikebrow mikebrow Jul 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wking I would completely agree with your preference if we were starting just from this spec. However, I don't think we should pick an option that makes runc v1 non-compliant. So that leaves us with convincing the runc guys to accept a PR to add a version command (easy to do, I'll do it, nbd... but the runc 1.0 rc is already out) or go with 2 or 3.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Sun, Jul 10, 2016 at 08:54:30AM -0700, Mike Brown wrote:

+### version

While I completely agree with your preference, I don't think we
should pick one that makes runc v1 non-compliant. So that leaves us
with convincing the runc guys to accept a PR to add a version
command (easy to do, I'll do it, nbd... but the runc 1.0 rc is
already out) or go with 2 or 3.

A third option is to supply a third-party oci-cli-runc wrapper that
translates the ‘version’ command into a ‘runC --version’ call and
otherwise just passes arguments through to runC directly. My
preferences here are:

  1. runC accepts a patch to support both --version and ‘version’.
  2. The OCI provides an oci-cli-runc wrapper. This can fall under the
    compliance-test project or the runC project or a new OCI Project,
    it doesn't matter to me. It's probably going to be about five
    lines of shell, if this is the only wrapping we have to do ;).
  3. Go with spec approach 2 (require only --version).
  4. Go with spec approach 3 (require either --version or ‘version’).

@cyphar
Copy link
Member

cyphar commented Jul 9, 2016

In my opinon, this is a very restrictive change. What if an OCI runtime wanted to instead offer a web interface to start containers that then allowed you to upload an OCI-compliant spec file? What if it was an interactive interface to manage containers? I'm not sure what benefit you get from requring that all runtimes be a mirror of runc, ccon and so on.

@wking
Copy link
Contributor Author

wking commented Jul 9, 2016

On Fri, Jul 08, 2016 at 09:16:32PM -0700, Aleksa Sarai wrote:

In my opinon, this is a very restrictive change.

The intention was to go with minimal and generic, and avoid
restrictive ;).

What if an OCI runtime wanted to instead offer a web interface to
start containers that then allowed you to upload an OCI-compliant
spec file?

They can absolutely do that. This API specification is required to
get drop-in runtime integration for runtime-spec compliance testing
and orchestration layers that want to work with any compliant
runtime 1. A command-line API seems like the simplest way to do
that.

I've tried to convey in the text here that runtimes are free to extend
this command-line API in many ways (e.g. by adding commands, options,
etc.). I haven't directly addressed the fact that they're
(obviously?) free to expose non-command-line APIs as they see fit.
The generic OCI compliance testing won't be exercising those
extensions, but that's fine with me.

What if it was an interactive interface to manage containers? I'm
not sure what benefit you get from requring that all runtimes be a
mirror of runc, ccon and so on.

You get a way to test their compliance with the spec. It seems
unlikely to me that a real-world runtime implemention would have a lot
of difficulty writing a command-line wrapper exposing this API. If
you know of one, please point it out, and we can try to find an API
that all known runtimes can easily implement. We just need some
sort of standardized handles for compliance testing, it doesn't really
matter what they are.

 Subject: Re: Specifying the runtime's command-line interface
 Date: Wed, 9 Dec 2015 13:07:56 -0800
 Message-ID: <20151209210755.GB9787@odin.tremily.us>

@mikebrow
Copy link
Member

mikebrow commented Jul 10, 2016

@cyphar this is the cli spec.. the generic runtime spec is here:
https://github.com/opencontainers/runtime-spec/blob/master/runtime.md

I think if someone wanted a web based interface that would be different than a cli interface?

@cyphar
Copy link
Member

cyphar commented Jul 10, 2016

@wking

They can absolutely do that. This API specification is required to get drop-in runtime integration for runtime-spec compliance testing and orchestration layers that want to work with any compliant runtime [1]. A command-line API seems like the simplest way to do that. [...] The generic OCI compliance testing won't be exercising those extensions, but that's fine with me.

To be clear, the point I want to make clear is that it should be up to a runtime to not implement a CLI interface at all (they would obviously have to implement their own wrappers around their management system so that we could do compliance testing). As such, I'm still on the fence about making any statements about command-line requirements (unless it's an advisory note, in which case it should be called a specification and it isn't really useful). The important thing is that the interface that an OCI implementation provides allow spec-conformant configurations to run as expected.

@mikebrow

I think if someone wanted a web based interface that would be different than a cli interface?

I understand that, but I'm questioning whether or not we should be making statements about CLI interfaces other than "recommendations". I get that it makes writing compliance testing code simpler, but surely we should allow runtimes to provide their own wrappers so they can assure compliance (without requiring that they provide a CLI interface).

@mikebrow
Copy link
Member

@cyphar don't disagree.. But at the same time there's a benefit to having specs for the various popular types of interfaces to go with the generic runtime spec. It would suck if every web based interface provider went and used a completely different rest model to implement the runtime spec. In this case I think the goal was to form a generic cli spec based on where we are with runc and other early cli adopters with the goal of making sure that cli spec complaint runtimes are replaceable at the cli. If the goal was wrong in the first place.. that's a different discussion :-)

@wking
Copy link
Contributor Author

wking commented Jul 10, 2016

On Sun, Jul 10, 2016 at 07:57:11AM -0700, Aleksa Sarai wrote:

They can absolutely do that. This API specification is required to
get drop-in runtime integration for runtime-spec compliance
testing…

To be clear, the point I want to make clear is that it should be up
to a runtime to not implement a CLI interface at all (they would
obviously have to implement their own wrappers around their
management system so that we could do compliance testing).

This seems like a fine hair to split ;). What is the difference
between “MUST provide an executable [implementing this command line
API]” (which is what I'm aiming for here) and “have to implement their
own wrappers around their management system so that we could do
compliance testing”?

What I want to avoid is patching the compliance test suite to support
per-runtime interfaces, because then it's hard to determine if a test
passing or failing is due to the runtime or the test-suite
adjustments.

@mikebrow
Copy link
Member

@wking assuming, arguento, someone has a web interface that is compatible with the runtime spec, why should they be forced into providing a client api for testing purposes? Why not allow for creation of a web interface spec and test framework?

@wking
Copy link
Contributor Author

wking commented Jul 10, 2016

On Sun, Jul 10, 2016 at 09:15:32AM -0700, Mike Brown wrote:

@wking assuming, arguento, someone has a web interface that is
compatible with the runtime spec, why should they be forced into
providing a client api for testing purposes? Why not allow for
creation of a web interface spec and test framework?

In the likely case that not all runtimes agree on the same native API,
we'll have an ecosystem like this:

{generic runtime-callers} ↔ {wrappers/shims} ↔ {runtimes} ↔ {runtime-specific runtime-callers}

Generic runtime-callers will be things like the compliance test suite
and potentially orchestration tools like containerd (although
containerd itself may decide it wants to be a runC-specific
runtime-caller). I think that breakdown is fairly non-controversial,
but if it doesn't sound right, we probably want to figure out what it
should be before dividing the components up into packages ;).

If you do accept the breakdown above, the question is where the
boundaries of responsibility are. With this proposal, I've been
shooting for a command-line interface as the single, generic API
between generic runtime-callers and the wrappers and shims. It sounds
like you're floating the idea of allowing multiple APIs between
generic runtime-callers and the runtimes (call them API 1 and API 2).
Since the compliance testers are unlikely to define two completely
separate test suites, they'll still have some generic API internally.
Say they pick API 1. To support API 2 as well, they'll need an
architecture like this:

Complaince test package:
{generic tests} ↔ {API 1} ↔ {API 1↔2 shim}

where API-1-based runtimes plug into the API 1 interface directly,
while API-2-based runtimes plug in via the API 1↔2 shim. But a
generic orchestration layer would also need the same setup
internally, which is going to lead to duplication of effort unless we
make that API 1↔2 shim a separate package. So instead of making
wrapping/shimming the responsibility of each generic runtime-caller,
I'd rather push it to the runtimes themselves:

Runtime 1 package:
{API-1-based runtime}

Runtime 2a package:
{API 1↔2 shim} ↔ {API-2-based runtime}

Or have the wrappers/shims be independent projects:

Runtime 2b package:
{API-2-based runtime}

API 2 wrapper package:
{API 1↔2 shim}

This last approach has the benefit that a few runtimes can coalesce
around API 2 and share the work of maintaining the API 1 ↔ API 2 shim.
But I don't think it's a good idea to require all generic OCI
runtime-callers to support multiple APIs (just like it's not a good
idea to require all generic OCI runtime-callers to support multiple
APIs for requesting version information, which is approach 3 in 1).

@mikebrow
Copy link
Member

@wking not sure if you caught the subtlety of cypher's point and my subsequent question. I'm drawing a distinction between a client API and a web interface where we could have two different types of interface each having their own independent specification (though with similar semantics) for implementing the same runtime spec.

I'm not saying there should be two different command line client APIs for the same runtime spec. If we don't control the APIs for the various types of interface to the container runtime.. we'll loose interoperability at said interfaces. Requiring and/or building wrappers/shims may or may not work without extra code / changes to the host runtime and encourage bifurcation.

I suppose that's why I'm against this PRs direction of creating a 1.0 spec that's not compatible with the 1.0rc runc. I don't really like the idea of having to build a shim to coax runc into OCI 1.0 compatibility.

The version issue is a great example. We have a choice of creating an API spec that is compatible with runc or one that makes runc do a PR to add the command or write a wrapper to become compliant. What is the sound technical reason / benefit for demanding version be done as a first class command verses an option parameter. Choosing the choice that makes runc incompatible with the spec should be done only if the difference is important for a known use case.

@wking
Copy link
Contributor Author

wking commented Jul 11, 2016

On Mon, Jul 11, 2016 at 06:29:22AM -0700, Mike Brown wrote:

I'm drawing a distinction between a client API and a web interface
where we could have two different types of interface each having
their own independent specification (though with similar semantics)
for implementing the same runtime spec.

And I think you should be able to wrap that web API in a command-line
client exposing this API. If you expect difficulty with that, give
some concrete examples 1.

If we don't control the APIs for the various types of interface to
the container runtime.. we'll loose interoperability at said
interfaces. Requiring and/or building wrappers/shims may or may not
work without extra code / changes to the host runtime and encourage
bifurcation.

I think introducing multiple compliance-testing APIs (e.g. a
command-line API and a web API) and allowing runtimes to pick one or
the other will certainly reproduce interop. Or at least make life
harder on generic runtime-callerer, which now need to support both
APIs. If, on the other hand, we require all compliant runtimes to
supply the same API (command line or otherwise), then generic
runtime-callers only need to support that API 2.

I suppose that's why I'm against this PRs direction of creating a
1.0 spec that's not compatible with the 1.0rc runc. I don't really
like the idea of having to build a shim to coax runc into OCI 1.0
compatibility.

I've filed ‘runc version’ in opencontainers/runc#940.

The version issue is a great example. We have a choice of creating
an API spec that is compatible with runc or one that makes runc do a
PR to add the command or write a wrapper to become compliant. What
is the sound technical reason / benefit for demanding version be
done as a first class command verses an option parameter.

The reason is that the spec wording is simpler and more
self-consistent without the “but you don't have to specify a command
if you use one of these command-like global options” caveat 3. And
simplicity and consistency make a more intuitive API. For example,
you don't have to read the global-options section to see if it's
hiding any command-like options.

 Around “It seems unlikely to me that a real-world runtime
 implemention would have a lot of difficulty writing a
 command-line wrapper exposing this API…”

@mikebrow
Copy link
Member

mikebrow commented Jul 11, 2016

And I think you should be able to wrap that web API in a command-line
client exposing this API. If you expect difficulty with that, give
some concrete examples [1].

@wking my read of your comments is that you're talking apples and I'm talking oranges.. I'm saying an OCI compatible product should not require a block of code to contrive a successful compliance test run out of it. Example: runc requires a container id be provided by the user.. your proposed cli api does not, thus your version of the spec is not compliant with the MUST language currently being used. The proposal to just add a wrapper that generates the requirement does not make the two different specs equivalent it merely shows one how one might integrate the two different specs. So is the goal compliance with a spec or integration with same?

@mikebrow
Copy link
Member

I think introducing multiple compliance-testing APIs (e.g. a
command-line API and a web API) and allowing runtimes to pick one or
the other will certainly reproduce interop. Or at least make life
harder on generic runtime-callerer, which now need to support both
APIs. If, on the other hand, we require all compliant runtimes to
supply the same API (command line or otherwise), then generic
runtime-callers only need to support that API [2].

I don't have a problem with proposing a web api include a mapping to the client api where that makes sense. That said I doubt limiting a web api to client api parameters makes a whole lot of sense given that web apis generally need to be remote and secure where cli apis do not.

@mikebrow
Copy link
Member

mikebrow commented Jul 11, 2016

I've filed ‘runc version’ in opencontainers/runc#940.

kk good

The reason is that the spec wording is simpler and more
self-consistent without the “but you don't have to specify a command
if you use one of these command-like global options” caveat [3]. And
simplicity and consistency make a more intuitive API. For example,
you don't have to read the global-options section to see if it's
hiding any command-like options.

Using -v to output version information isn't that hard a concept to understand. Yes it's sort of like using bad grammar given that it's a command not a global option. But again some people prefer -v as a convention so why not support both? It's not like version is a command that affects the primary object, a container. It's more like --help than like the other commands.

@wking
Copy link
Contributor Author

wking commented Jul 11, 2016

On Mon, Jul 11, 2016 at 10:45:42AM -0700, Mike Brown wrote:

@wking my read of your comments is that you're talking apples and
I'm talking oranges..

Maybe :p. Hopefully we'll spiral down and meet in the middle soon ;).

I'm saying an OCI compatible product should not require a block of
code to contrive a successful compliance test run out of it.

This sounds like a tautology ;).

Example: runc requires a container id be provided by the user.. your
proposed cli api does not, thus your version of the spec is not
compliant with the MUST language currently being used.

Self-consistency, one way or the other is blocking on #508.

The proposal to just add a wrapper that generates the requirement
does not make the two different specs equivalent it merely shows one
how one might integrate the two different specs. So is the goal
compliance with a spec or integration with same?

If the wrapped runtime requires an ID, then you can implement the
current API-spec with:

#!/bin/bash

GLOBAL_OPTS=()
while test 'x-' = "x${1::1}"
do
GLOBAL_OPTS+=("${1}")
shift
done
COMMAND="${1}"
shift

case "${COMMAND}" in
'create')
ARGS=()
ID=$(uuidgen)
while test -n "${1}"
do
case "${1}" in
'--id')
ID="${2}"
shift 2
continue
;;
esac
ARGS+=("${1}")
shift
done
ARGS+=("${ID}")
;;
esac
exec python -c 'import sys; print(sys.argv)' "${GLOBAL_OPTS[@]}" "${COMMAND}" "${ARGS[@]}"

where ‘python -c 'import sys; print(sys.argv)'’ is a dummy that should
be replaced with the runtime (e.g. ‘runc’). With post-#508
self-consistency between the command-line API and runtime.md, does
that address your concerns?

@wking
Copy link
Contributor Author

wking commented Jul 11, 2016

On Mon, Jul 11, 2016 at 10:52:38AM -0700, Mike Brown wrote:

That said I doubt limiting a web api to client api parameters makes
a whole lot of sense given that web apis generally need to be remote
and secure where cli apis do not.

Is that “How will the caller add their credentials when calling a
web-based backend from a command-line API?”. That's a good question,
since I'm not sure how to handle credentials in a generic way, and it
gets into issues about how the runtime system is delivered to the OCI
compliance-testing team (is it ok to test a remote runtime? Do you
need local copies of all runtime components for compliance testing?).

What do we think about an ‘--engine ENGINE’ global option that acts
like runC's current --root but also allows web-based runtimes to set:

--engine https://USER:PASS@test-runtime.example.com

@mikebrow
Copy link
Member

mikebrow commented Jul 11, 2016

does that address your concerns?

No it just points out why my statement wasn't a tautology :) Again, I believe you are mixing the idea of api adherence type compliance and pseudo api integration compliance via wrappers. IOW what is the purpose of this CLI API specification.. is it to find a spec that others can easily be complaint with no matter how different they are.. or is it a spec to specify how to be OCI compliant at the command line interface. I believe the place for documenting broad compliance issues across the different types of APIs is the broader runtime spec document not this CLI spec.

@mikebrow
Copy link
Member

mikebrow commented Jul 11, 2016

On Mon, Jul 11, 2016 at 10:52:38AM -0700, Mike Brown wrote:
That said I doubt limiting a web api to client api parameters makes
a whole lot of sense given that web apis generally need to be remote
and secure where cli apis do not.

Trevor replied:

Is that “How will the caller add their credentials when calling a
web-based backend from a command-line API?”. That's a good question,
since I'm not sure how to handle credentials in a generic way, and it
gets into issues about how the runtime system is delivered to the OCI
compliance-testing team (is it ok to test a remote runtime? Do you
need local copies of all runtime components for compliance testing?).

What do we think about an ‘--engine ENGINE’ global option that acts
like runC's current --root but also allows web-based runtimes to set:

--engine https://USER:PASS@test-runtime.example.com

Yes, that's one direction it could take, but those would be new features for the next release. Not trying to boil the ocean here just yet. It will be easier to get new features discussed once we have an API spec to work from. For example, on #508 recommend splitting it up into separate concerns, it's hard to track n-issues / features in one pull request. Hard to move forward without basic bare min baseline. See @duglin 's attempt at a baseline on 511. Doesn't solve / close every open issue but it does give us a potential starting point.

@mrunalp
Copy link
Contributor

mrunalp commented Feb 1, 2017

We can get this in and fixup up runc later. (We don't want the --console-socket option). @wking could you update this and rebase?

wking pushed a commit to wking/opencontainer-runtime-spec that referenced this pull request Feb 1, 2017
# Commands

## create

The --bundle [start-pr-bundle] and --pid-file options and ID argument
[runc-start-id] match runC's interface.

One benefit of the early-exit 'create' is that the exit code does not
conflate container process exits with "failed to setup the sandbox"
exits.  We can take advantage of that and use non-zero 'create' exits
to allow stderr writing (so the runtime can log errors while dying
without having to successfully connect to syslog or some such).
Trevor still likes the long-running 'create' API because it makes
collecting the exit code easier, see the entry under rejected-for-now
avenues at the end of this commit message.

### --pid-file

You can get the PID by calling 'state' [container-pid-from-state], and
container PIDs may not be portable [container-pid-not-portable].  But
it's a common way for interfacing with init systems like systemd
[systemd-pid], and for this first pass at the command line API folks
are ok with some Linux-centrism [linux-centric].

### Document LISTEN_FDS for passing open file descriptors

This landed in runC with [runc-listen-fds], but the bundle-author <->
runtime specs explicitly avoided talking about how this is set (since
the bundle-author didn't care about the runtime-caller <-> runtime
interface) [runtime-spec-caller-api-agnostic].  This commit steps away
from that agnosticism.

Trevor left LISTEN_PID [sd_listen_fds,listen-fds-description] out,
since he doesn't see how the runtime-caller would choose anything
other than 1 for its value.  It seems like something that a process
would have to set for itself (because guessing the PID of a child
before spawning it seems racy ;).  In any event, the runC
implementation seems to set this to 1 regardless of what systemd
passes to it [listen-fds-description].

We've borrowed Shishir's wording for the example
[listen-fds-description].

## state [state-pr]

Partially catch up with opencontainers/runtime-spec@7117ede7 (Expand
on the definition of our ops, 2015-10-13,
opencontainers#225, v0.4.0).  The state example is
adapted from runtime.md, but we defer the actual specification of the
JSON to that file.

The encoding for the output JSON (and all standard-stream activity) is
covered by the "Character encodings" section.  In cases where the
runtime ignores the SHOULD (still technically compliant), RFC 7159
makes encoding detection reasonably straightforward [rfc7159-s8.1].
The obsolete RFC 4627 has some hints as well [rfc4627-s3] (although
these were dropped in RFC 7518 [rfc7518-aA], probably as a result of
removing the constraint that "JSON text" be an object or array
[rfc7518-aA]).  The hints should still apply to the state output,
because we know it will be an object.  If that ends up being too dicey
and we want to certify runtimes that do not respect their
operating-system conventions, we can add an --encoding option later.

## kill

Partially catch up with opencontainers/runtime-spec@be594153 (Split
create and start, 2016-04-01, opencontainers#384).  The
interface is based on POSIX [posix-kill], util-linux
[util-linux-kill], and GNU coreutils [coreutils-kill].

The TERM/KILL requirement is a minimum portability requirement for
soft/hard stops.  Windows lacks POSIX signals [windows-signals], and
currently supports soft stops in Docker with whatever is behind
hcsshim.ShutdownComputeSystem [docker-hcsshim].  The docs we're
landing here explicitly allow that sort of substitution, because we
need to have soft/hard stop on those platforms but *can't* use POSIX
signals.  They borrow wording from
opencontainers/runtime-spec@35b0e9ee (config: Clarify MUST for
platform.os and .arch, 2016-05-19, opencontainers#441) to
recommend runtime authors document the alternative technology so
bundle-authors can prepare (e.g. by installing the equivalent to a
SIGTERM signal handler).

# Command style

Use imperative phrasing for command summaries, to follow the practice
recommended by Python's PEP 257 [pep257-docstring]:

  The docstring is a phrase ending in a period. It prescribes the
  function or method's effect as a command ("Do this", "Return that"),
  not as a description; e.g. don't write "Returns the pathname ...".

The commands have the following layout:

  ### {command name}

  {one-line description}

  * *Options:* ...
  ...
  * *Exit code:* ...

  {additional notes}

  #### Example

  {example}

The four-space list indents follow opencontainers/runtime-spec@7795661
(runtime.md: Fix sub-bullet indentation, 2016-06-08,
opencontainers#495).  From [markdown-syntax]:

  List items may consist of multiple paragraphs.  Each subsequent
  paragraph in a list item must be indented by either 4 spaces or one
  tab...

Trevor expects that's intended to be read with "block element" instead
of "paragraph", in which case it applies to nested lists too.

And while GitHub supports two-space indents [github-lists]:

  You can create nested lists by indenting lines by two spaces.

it seems that pandoc does not.

# Versioning

The command-line interface is largely orthogonal to the config format,
and config authors and runtime callers may be entirely different sets
of people.  Zhang Wei called for more explicit versioning for the CLI
[interface-versioning], and the approach taken here follows the
approach taken by Python's email package [python-email-version].

Wedging multiple, independently versioned entities into a single
repository can be awkward, but earlier proposals to put the CLI in its
own repository [separate-repository-proposed] were unsuccessful
because compliance testing requires both a CLI and a config
specification [separate-repository-refused].  Trevor doesn't think
that's a solid reason [separate-repository-refusal-rebutted], but
discussion along that line stalled out, so the approach taken here is
to keep both independently versioned entities in the same repository.

# Global options

This section is intended to allow runtimes to extend the command line
API with additional options and commands as they see fit without
interfering with the commands and options specified in this document.
The last line in this section makes it explicit that any later
specification (e.g. "MUST print the state JSON to its stdout") do not
apply to cases where the caller has included an unspecified option or
command (e.g. --format=protobuf).  For extensive discussion on this
point see [extensions-unspecified].

With regard to the statement "Command names MUST NOT start with
hyphens", the rationale behind this decision is to distinguish
unrecognized commands from unrecognized options
[distinguish-unrecognized-commands] because we want to allow (but not
require) runtimes to fail fast when faced with an unrecognized
command [optional-fail-fast].

# Long options

Use GNU-style long options to avoid ambiguous, one-character options
in the spec, while still allowing the runtime to support one-character
options with packing.  We don't specify one-character options in this
spec, because portable callers can use the long form, and not
specifying short forms leaves runtimes free to assign those as they
see fit.

# Character encodings

Punt to the operating system for character encodings.  Without this,
the character set for the state JSON or other command output seemed
too ambiguous.

Trevor wishes there were cleaner references for the
{language}.{encoding} locales like en_US.UTF-8 and UTF-8.  But
[wikipedia-utf-8,wikipedia-posix-locale] seems too glib, and he can't
find a more targetted UTF-8 link than just dropping folks into a
Unicode chapter (which is what [wikipedia-utf-8] does):

  The Unicode Standard, Version 6.0, §3.9 D92, §3.10 D95 (2011)

With the current v8.0 (2015-06-17), it's still §3.9 D92 and §3.10 D95.

The TR35 link is for:

  In addition, POSIX locales may also specify the character encoding,
  which requires the data to be transformed into that target encoding.

and the POSIX §6.2 link is for:

  In other locales, the presence, meaning, and representation of any
  additional characters are locale-specific.

# Standard streams

The "MUST NOT attempt to read from its stdin" means a generic caller
can safely exec the command with a closed or null stdin, and not have
to worry about the command blocking or crashing because of that.  The
stdout spec for start/delete is more lenient, because runtimes are
unlikely to change their behavior because they are unable to write to
stdout.  If this assumption proves troublesome, we may have to tighten
it up later.

Aleksa Sarai also raised concerns over the safety of potentially
giving the container process access to terminal ioctl escapes
[stdio-ioctl] and feels like the stdio file-descriptor pass-through is
surprising [stdio-surprise].

# Console socket protocol

Based on in-flight work by Aleksa in opencontainers/runc#1018, this
commit makes the following choices:

* SOCK_SEQPACKET instead of SOCK_STREAM, because this is a
  message-based protocol, so it seems more natural to use a
  message-oriented socket type.

* A string 'type' field for all messages, so we can add additional
  message types in the future without breaking backwards compatibility
  (new console-socket servers will still support old clients).  Aleksa
  favored splitting this identifier into an integer 'type' and
  'version' fields [runc-socket-type-version], but I don't see the
  point if they're both opaque integers without internal structure.
  And I expect this protocol to be stable enough that it's not worth
  involving SemVer and its structured versioning.

* Response messages, so the client can tell whether the request was
  received and processed successfully or not.  That gives the client a
  way to bail out if, for example, the server does not support the
  'terminal' message type.

* Add a sub-package specs-go/socket.  Even though there aren't many
  new types, these are fairly different from the rest of specs-go and
  that namespace was getting crowded.

# Event triggers

The "Callers MAY block..." wording is going to be hard to enforce, but
with the runC model, clients rely on the command exits to trigger
post-create and post-start activity.  The longer the runtime hangs
around after completing its action, the laggier those triggers will
be.

For an alternative event trigger approach, see the discussion of an
'event' command in the rejected-for-now avenues at the end of this
commit message.

# Lifecycle notes

These aren't documented in the current runtime-spec, and may no longer
be true.  But they were true at one point, and informed the
development of this specification.

## Process cleanup

On IRC on 2015-09-15 (with PDT timestamps):

  10:56 < crosbymichael> if the main process dies in the container,
    all other process are killed
  ...
  10:58 < julz> crosbymichael: I'm assuming what you mean is you kill
    everything in the cgroup -> everything in the container dies?
  10:58 < crosbymichael> julz: yes, that is how its implemented
  ...
  10:59 < crosbymichael> julz: we actually freeze first, send the
    KILL, then unfreeze so we don't have races

## Container IDs for namespace joiners

You can create a config that adds no isolation vs. the runtime
namespace or completely joins another set of existing namespaces.  It
seems odd to call that a new "container", but the ID is really more of
a process ID, and less of a container ID.  The "container" phrasing is
just a useful hint that there might be some isolation going on.  And
we're always creating a new "container process" with 'create'.

# Other changes

This commit also moves the file-descriptor docs from runtime-linux.md
into runtime.md and the command-line docs.  Both affect runtime
authors, but:

* The runtime.md entry is more useful for bundle authors than the old
  wording, because it gives them confidence that the runtime caller
  will have the power to set these up as they see fit (within POSIX's
  limits).  It is also API-agnostic, so bundle authors know they won't
  have to worry about which API will be used to launch the container
  before deciding whether it is safe to rely on runtime-caller
  file-descriptor control.

* The command line entry is more useful for runtime-callers than the
  old wording, because it tells you how to setup the file descriptors
  instead of just telling you that they MAY be setup.

I moved the bundle-author language from runtime-linux.md to runtime.md
because it's relying on POSIX primitives that aren't Linux-specific.

# Avenues pursued but rejected (for now)

* Early versions of this specification had 'start' taking '--config'
  and '--runtime', but this commit uses '--bundle' [start-pr-bundle].

  The single config file change [single-config-proposal] went through,
  but Trevor would also like to be able to pipe a config into the
  'funC start' command (e.g. via a /dev/fd/3 pseudo-filesystem path)
  [runc-config-via-stdin], and he has a working example that supports
  this without difficulty [ccon-config-via-stdin].  But since
  [runc-bundle-option] landed on 2015-11-16, runC has replaced their
  --config-file and --runtime-file flags with --bundle, and the
  current goal of this API is "keeping as much similarity with the
  existing runC command-line as possible", not "makes sense to Trevor"
  ;).  It looks like runC was reacting [runc-required-config-file] to
  strict wording in the spec [runtime-spec-required-config-file], so
  we might be able to revisit this if/when we lift that restriction.

* Having 'start' (now 'create') take a --state option to write state
  to a file [start-pr-state].  This is my preferred approach to
  sharing container state, since it punts a persistent state registry
  to higher-level tooling [punt-state-registry].  But runtime-spec
  currently requires the runtime to maintain such a registry
  [state-registry], and we don't need two ways to do that ;).

  On systems like Solaris, the kernel maintains a registry of
  container IDs directly, so they don't need an external registry
  [solaris-kernel-state].

* Having 'start' (now 'create') take an --id option instead of a
  required ID argument, and requiring the runtime to generate a unique
  ID if the option was not set.  When there is a long-running host
  process waiting on the container process to perform cleanup, the
  runtime-caller may not need to know the container ID.  However, runC
  has been requiring a user-specified ID since [runc-start-id], and
  this spec follows the early-exit 'create' from [runc-create-start],
  so we require one here.  We can revisit this if we regain a
  long-running 'create' process.

* Having 'create' take a '--console-socket PATH' option (required when
  process.terminal is true) with a path to a SOCK_SEQPACKET Unix
  socket for use with the console-socket protocol.  The current
  'LISTEN_FDS + 3' approach was proposed by Michael Crosby
  [console-socket-fd], but Trevor doesn't have a clear idea of the
  motivation for the change and would have preferred '--console-socket
  FD'.

* Having a long-running 'create' process.  Trevor is not a big fan of
  this early-exit 'create', which requires platform-specific magic to
  collect the container process's exit code.  The ptrace idea in this
  commit is from Mrunal [mrunal-ptrace].

  Trevor has a proposal for an 'event' operation [event] which would
  provide a convenient created trigger.  With 'event' in place, we
  don't need the 'create' process exit to serve as that trigger, and
  could have a long-running 'create' that collects the container
  process exit code using the portable waitid() family.  But the
  consensus after this week's meeting was to table that while we land
  docs for the runC API [mimic-runc].

* Having a 'version' command to make it easy for a caller to report
  which runtime they're using.  But we don't have a use-case that
  makes it strictly necessary for interop, so we're leaving it out for
  now [no-version].

* Using 'sh' syntax highlighting [syntax-highlighting] for the fenced
  code blocks.  The 'sh' keyword comes from [linguist-languages].  But
  the new fenced code blocks are shell sessions, not scripts, and we
  don't want shell-syntax highlighting in the command output.

[ccon-config-via-stdin]: https://github.com/wking/ccon/tree/v0.4.0#configuration
[console-socket-fd]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-10-19-21.00.log.html#l-30
[container-pid-from-state]: https://github.com/opencontainers/runtime-spec/pull/511/files#r70353376
  Subject: Add initial pass at a cmd line spec
[container-pid-not-portable]: opencontainers#459
  Subject: [ Runtime ] Allow for excluding pid from state
[coreutils-kill]: http://www.gnu.org/software/coreutils/manual/html_node/kill-invocation.html
[distinguish-unrecognized-commands]: https://github.com/wking/oci-command-line-api/pull/8/files#r46898167
  Subject: Clarity for commands vs global options
[docker-hcsshim]: https://github.com/docker/docker/pull/16997/files#diff-5d0b72cccc4809455d52aadc62329817R230
  moby/moby@bc503ca8 (Windows: [TP4] docker kill handling,
  2015-10-12, moby/moby#16997)
[event]: opencontainers#508
  Subject: runtime: Add an 'event' operation for subscribing to pushes
[extensions-unspecified]: http://ircbot.wl.linuxfoundation.org/eavesdrop/%23opencontainers/%23opencontainers.2016-07-27.log.html#t2016-07-27T16:37:56
[github-lists]: https://help.github.com/articles/basic-writing-and-formatting-syntax/#lists
[interface-versioning]: opencontainers#513 (comment)
[linguist-languages]: https://github.com/github/linguist/blob/master/lib/linguist/languages.yml
[linux-centric]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-07-13-17.03.log.html#l-39
[listen-fds-description]: opencontainers/runc#231 (comment)
  Subject: Systemd integration with runc, for on-demand socket
    activation
[markdown-syntax]: http://daringfireball.net/projects/markdown/syntax#list
[mimic-runc]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-07-13-17.03.log.html#l-15
[mrunal-ptrace]: http://ircbot.wl.linuxfoundation.org/eavesdrop/%23opencontainers/%23opencontainers.2016-07-13.log.html#t2016-07-13T18:58:54
[no-version]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-07-20-21.03.log.html#l-75
[optional-fail-fast]: wking/oci-command-line-api@527f3c6#commitcomment-14835617
  Subject: Use RFC 2119's keywords (MUST, MAY, ...)
[pep257-docstring]: https://www.python.org/dev/peps/pep-0257/#one-line-docstrings
[posix-kill]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/kill.html
[punt-state-registry]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2015/opencontainers.2015-12-02-18.01.log.html#l-79
[python-email-version]: https://docs.python.org/3/library/email.html#package-history
[rfc4627-s3]: https://tools.ietf.org/html/rfc4627#section-3
[rfc7158-aA]: https://tools.ietf.org/html/rfc7158#appendix-A
  RFC 7518 is currently identical to 7519.
[rfc7159-s8.1]: https://tools.ietf.org/html/rfc7159#section-8.1
[runc-bundle-option]: opencontainers/runc#373
  Subject: adding support for --bundle
[runc-config-via-stdin]: opencontainers/runc#202
  Subject: Can runc take its configuration on stdin?
[runc-listen-fds]: opencontainers/runc#231
  Subject: Systemd integration with runc, for on-demand socket
    activation
[runc-required-config-file]: opencontainers/runc#310 (comment)
  Subject: specifying a spec file on cmd line?
[runc-socket-type-version]: opencontainers/runc#1018 (comment)
  Subject: Consoles, consoles, consoles.
[runc-start-id]: opencontainers/runc#541
  opencontainers/runc@a7278cad (Require container id as arg1,
  2016-02-08, opencontainers/runc#541)
[runtime-spec-caller-api-agnostic]: opencontainers#113 (comment)
  Subject: Add fd section for linux container process
[runtime-spec-required-config-file]: https://github.com/opencontainers/runtime-spec/pull/210/files#diff-8b310563f1c6f616aa98e6aeffc4d394R14
  106ec2d (Cleanup bundle.md, 2015-10-02,
  opencontainers#210)
[sd_listen_fds]: http://www.freedesktop.org/software/systemd/man/sd_listen_fds.html
[separate-repository-proposed]: opencontainers#513 (comment)
[separate-repository-refused]: opencontainers#513 (comment)
[separate-repository-refusal-rebutted]: opencontainers#513 (comment)
[single-config-proposal]: https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/0QbyJDM9fWY
  Subject: Single, unified config file (i.e. rolling back specs#88)
  Date: Wed, 4 Nov 2015 09:53:20 -0800
  Message-ID: <20151104175320.GC24652@odin.tremily.us>
[solaris-kernel-state]: wking/oci-command-line-api#3 (comment)
  Subject: Drop exec, pause, resume, and signal
[start-pr-bundle]: wking/oci-command-line-api#11
  Subject: start: Change --config and --runtime to --bundle
[start-pr-state]: wking/oci-command-line-api#14
  Subject: start: Add a --state option
[state-pr]: wking/oci-command-line-api#16
  Subject: runtime: Add a 'state' command
[state-registry]: https://github.com/opencontainers/runtime-spec/pull/225/files#diff-b84a8d65d8ed53f4794cd2db7e8ea731R61
  7117ede (Expand on the definition of our ops, 2015-10-13,
  opencontainers#225)
[stdio-ioctl]: opencontainers#513 (comment)
[stdio-surprise]: opencontainers#513 (comment)
[syntax-highlighting]: https://help.github.com/articles/github-flavored-markdown/#syntax-highlighting
[systemd-pid]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-07-20-21.03.log.html#l-69
[util-linux-kill]: http://man7.org/linux/man-pages/man1/kill.1.html
[wikipedia-utf-8]: https://en.wikipedia.org/wiki/UTF-8
[wikipedia-posix-locale]: https://en.wikipedia.org/wiki/Locale#POSIX_platforms
[windows-singals]: https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/PlGKu7QUwLE
  Subject: Fwd: Windows support for OCI stop/signal/kill (runtime-spec#356)
  Date: Thu, 26 May 2016 11:03:29 -0700
  Message-ID: <20160526180329.GL17496@odin.tremily.us>

Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>
Hopefully-Signed-off-by: Mike Brown <brownwm@us.ibm.com>
Signed-off-by: W. Trevor King <wking@tremily.us>
Reviewed-by: Jesse Butler <jeeves.butler@gmail.com>
@wking
Copy link
Contributor Author

wking commented Feb 1, 2017

From IRC:

@mrunalp: wking, okay, just clarified. We are okay with console-socket

So I've kept it and just rebased around #640 with 8b71f78d75087c. Let me know if we need to make other changes to get this landed.

* *stdin:* The runtime MUST pass its stdin file descriptor through to the container process without manipulation or modification.
"Without manipulation or modification" means that the runtime MUST not seek on the file descriptor, or close it, or read or write to it, or [`ioctl`][ioctl.3] it, or perform any other action on it besides passing it through to the container process.

When using a container to drop privileges, note that providing a privileged terminal's file descriptor may allow the container to [execute privileged operations via `TIOCSTI`][TIOCSTI-security] or other [TTY ioctls][tty_ioctl.4].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this level of detail here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this level of detail here.

@cyphar requested a TIOCSTI mention. I don't care either way, so sort it out with him and tell me what to do.


##### Responses

All responses MUST contain a **`type`** property whose value MUST one of the following strings:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST one -> MUST be one..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Fixed with d75087cacd43c4 (also rebases on the current master).

The message's [ancillary data][socket-queue] (`msg_control*`) MUST contain at least one [`cmsghdr`][socket.h]).
The first `cmsghdr` MUST have:

* `cmsg_type` set to [`SOL_SOCKET`][socket.h],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will have to check how cross platform console-socket is.
cc: @RobDolinMS

# Commands

## create

The --bundle [start-pr-bundle] and --pid-file options and ID argument
[runc-start-id] match runC's interface.

One benefit of the early-exit 'create' is that the exit code does not
conflate container process exits with "failed to setup the sandbox"
exits.  We can take advantage of that and use non-zero 'create' exits
to allow stderr writing (so the runtime can log errors while dying
without having to successfully connect to syslog or some such).
Trevor still likes the long-running 'create' API because it makes
collecting the exit code easier, see the entry under rejected-for-now
avenues at the end of this commit message.

### --pid-file

You can get the PID by calling 'state' [container-pid-from-state], and
container PIDs may not be portable [container-pid-not-portable].  But
it's a common way for interfacing with init systems like systemd
[systemd-pid], and for this first pass at the command line API folks
are ok with some Linux-centrism [linux-centric].

### Document LISTEN_FDS for passing open file descriptors

This landed in runC with [runc-listen-fds], but the bundle-author <->
runtime specs explicitly avoided talking about how this is set (since
the bundle-author didn't care about the runtime-caller <-> runtime
interface) [runtime-spec-caller-api-agnostic].  This commit steps away
from that agnosticism.

Trevor left LISTEN_PID [sd_listen_fds,listen-fds-description] out,
since he doesn't see how the runtime-caller would choose anything
other than 1 for its value.  It seems like something that a process
would have to set for itself (because guessing the PID of a child
before spawning it seems racy ;).  In any event, the runC
implementation seems to set this to 1 regardless of what systemd
passes to it [listen-fds-description].

We've borrowed Shishir's wording for the example
[listen-fds-description].

## state [state-pr]

Partially catch up with opencontainers/runtime-spec@7117ede7 (Expand
on the definition of our ops, 2015-10-13,
opencontainers#225, v0.4.0).  The state example is
adapted from runtime.md, but we defer the actual specification of the
JSON to that file.

The encoding for the output JSON (and all standard-stream activity) is
covered by the "Character encodings" section.  In cases where the
runtime ignores the SHOULD (still technically compliant), RFC 7159
makes encoding detection reasonably straightforward [rfc7159-s8.1].
The obsolete RFC 4627 has some hints as well [rfc4627-s3] (although
these were dropped in RFC 7518 [rfc7518-aA], probably as a result of
removing the constraint that "JSON text" be an object or array
[rfc7518-aA]).  The hints should still apply to the state output,
because we know it will be an object.  If that ends up being too dicey
and we want to certify runtimes that do not respect their
operating-system conventions, we can add an --encoding option later.

## kill

Partially catch up with opencontainers/runtime-spec@be594153 (Split
create and start, 2016-04-01, opencontainers#384).  The
interface is based on POSIX [posix-kill], util-linux
[util-linux-kill], and GNU coreutils [coreutils-kill].

The TERM/KILL requirement is a minimum portability requirement for
soft/hard stops.  Windows lacks POSIX signals [windows-signals], and
currently supports soft stops in Docker with whatever is behind
hcsshim.ShutdownComputeSystem [docker-hcsshim].  The docs we're
landing here explicitly allow that sort of substitution, because we
need to have soft/hard stop on those platforms but *can't* use POSIX
signals.  They borrow wording from
opencontainers/runtime-spec@35b0e9ee (config: Clarify MUST for
platform.os and .arch, 2016-05-19, opencontainers#441) to
recommend runtime authors document the alternative technology so
bundle-authors can prepare (e.g. by installing the equivalent to a
SIGTERM signal handler).

# Command style

Use imperative phrasing for command summaries, to follow the practice
recommended by Python's PEP 257 [pep257-docstring]:

  The docstring is a phrase ending in a period. It prescribes the
  function or method's effect as a command ("Do this", "Return that"),
  not as a description; e.g. don't write "Returns the pathname ...".

The commands have the following layout:

  ### {command name}

  {one-line description}

  * *Options:* ...
  ...
  * *Exit code:* ...

  {additional notes}

  #### Example

  {example}

The four-space list indents follow opencontainers/runtime-spec@7795661
(runtime.md: Fix sub-bullet indentation, 2016-06-08,
opencontainers#495).  From [markdown-syntax]:

  List items may consist of multiple paragraphs.  Each subsequent
  paragraph in a list item must be indented by either 4 spaces or one
  tab...

Trevor expects that's intended to be read with "block element" instead
of "paragraph", in which case it applies to nested lists too.

And while GitHub supports two-space indents [github-lists]:

  You can create nested lists by indenting lines by two spaces.

it seems that pandoc does not.

# Versioning

The command-line interface is largely orthogonal to the config format,
and config authors and runtime callers may be entirely different sets
of people.  Zhang Wei called for more explicit versioning for the CLI
[interface-versioning], and the approach taken here follows the
approach taken by Python's email package [python-email-version].

Wedging multiple, independently versioned entities into a single
repository can be awkward, but earlier proposals to put the CLI in its
own repository [separate-repository-proposed] were unsuccessful
because compliance testing requires both a CLI and a config
specification [separate-repository-refused].  Trevor doesn't think
that's a solid reason [separate-repository-refusal-rebutted], but
discussion along that line stalled out, so the approach taken here is
to keep both independently versioned entities in the same repository.

# Global options

This section is intended to allow runtimes to extend the command line
API with additional options and commands as they see fit without
interfering with the commands and options specified in this document.
The last line in this section makes it explicit that any later
specification (e.g. "MUST print the state JSON to its stdout") do not
apply to cases where the caller has included an unspecified option or
command (e.g. --format=protobuf).  For extensive discussion on this
point see [extensions-unspecified].

With regard to the statement "Command names MUST NOT start with
hyphens", the rationale behind this decision is to distinguish
unrecognized commands from unrecognized options
[distinguish-unrecognized-commands] because we want to allow (but not
require) runtimes to fail fast when faced with an unrecognized
command [optional-fail-fast].

# Long options

Use GNU-style long options to avoid ambiguous, one-character options
in the spec, while still allowing the runtime to support one-character
options with packing.  We don't specify one-character options in this
spec, because portable callers can use the long form, and not
specifying short forms leaves runtimes free to assign those as they
see fit.

# Character encodings

Punt to the operating system for character encodings.  Without this,
the character set for the state JSON or other command output seemed
too ambiguous.

Trevor wishes there were cleaner references for the
{language}.{encoding} locales like en_US.UTF-8 and UTF-8.  But
[wikipedia-utf-8,wikipedia-posix-locale] seems too glib, and he can't
find a more targetted UTF-8 link than just dropping folks into a
Unicode chapter (which is what [wikipedia-utf-8] does):

  The Unicode Standard, Version 6.0, §3.9 D92, §3.10 D95 (2011)

With the current v8.0 (2015-06-17), it's still §3.9 D92 and §3.10 D95.

The TR35 link is for:

  In addition, POSIX locales may also specify the character encoding,
  which requires the data to be transformed into that target encoding.

and the POSIX §6.2 link is for:

  In other locales, the presence, meaning, and representation of any
  additional characters are locale-specific.

# Standard streams

The "MUST NOT attempt to read from its stdin" means a generic caller
can safely exec the command with a closed or null stdin, and not have
to worry about the command blocking or crashing because of that.  The
stdout spec for start/delete is more lenient, because runtimes are
unlikely to change their behavior because they are unable to write to
stdout.  If this assumption proves troublesome, we may have to tighten
it up later.

Aleksa Sarai also raised concerns over the safety of potentially
giving the container process access to terminal ioctl escapes
[stdio-ioctl] and feels like the stdio file-descriptor pass-through is
surprising [stdio-surprise].

# Console socket protocol

Based on in-flight work by Aleksa in opencontainers/runc#1018, this
commit makes the following choices:

* SOCK_SEQPACKET instead of SOCK_STREAM, because this is a
  message-based protocol, so it seems more natural to use a
  message-oriented socket type.

* A string 'type' field for all messages, so we can add additional
  message types in the future without breaking backwards compatibility
  (new console-socket servers will still support old clients).  Aleksa
  favored splitting this identifier into an integer 'type' and
  'version' fields [runc-socket-type-version], but I don't see the
  point if they're both opaque integers without internal structure.
  And I expect this protocol to be stable enough that it's not worth
  involving SemVer and its structured versioning.

* Response messages, so the client can tell whether the request was
  received and processed successfully or not.  That gives the client a
  way to bail out if, for example, the server does not support the
  'terminal' message type.

* Add a sub-package specs-go/socket.  Even though there aren't many
  new types, these are fairly different from the rest of specs-go and
  that namespace was getting crowded.

# Event triggers

The "Callers MAY block..." wording is going to be hard to enforce, but
with the runC model, clients rely on the command exits to trigger
post-create and post-start activity.  The longer the runtime hangs
around after completing its action, the laggier those triggers will
be.

For an alternative event trigger approach, see the discussion of an
'event' command in the rejected-for-now avenues at the end of this
commit message.

# Lifecycle notes

These aren't documented in the current runtime-spec, and may no longer
be true.  But they were true at one point, and informed the
development of this specification.

## Process cleanup

On IRC on 2015-09-15 (with PDT timestamps):

  10:56 < crosbymichael> if the main process dies in the container,
    all other process are killed
  ...
  10:58 < julz> crosbymichael: I'm assuming what you mean is you kill
    everything in the cgroup -> everything in the container dies?
  10:58 < crosbymichael> julz: yes, that is how its implemented
  ...
  10:59 < crosbymichael> julz: we actually freeze first, send the
    KILL, then unfreeze so we don't have races

## Container IDs for namespace joiners

You can create a config that adds no isolation vs. the runtime
namespace or completely joins another set of existing namespaces.  It
seems odd to call that a new "container", but the ID is really more of
a process ID, and less of a container ID.  The "container" phrasing is
just a useful hint that there might be some isolation going on.  And
we're always creating a new "container process" with 'create'.

# Other changes

This commit also moves the file-descriptor docs from runtime-linux.md
into runtime.md and the command-line docs.  Both affect runtime
authors, but:

* The runtime.md entry is more useful for bundle authors than the old
  wording, because it gives them confidence that the runtime caller
  will have the power to set these up as they see fit (within POSIX's
  limits).  It is also API-agnostic, so bundle authors know they won't
  have to worry about which API will be used to launch the container
  before deciding whether it is safe to rely on runtime-caller
  file-descriptor control.

* The command line entry is more useful for runtime-callers than the
  old wording, because it tells you how to setup the file descriptors
  instead of just telling you that they MAY be setup.

I moved the bundle-author language from runtime-linux.md to runtime.md
because it's relying on POSIX primitives that aren't Linux-specific.

# Avenues pursued but rejected (for now)

* Early versions of this specification had 'start' taking '--config'
  and '--runtime', but this commit uses '--bundle' [start-pr-bundle].

  The single config file change [single-config-proposal] went through,
  but Trevor would also like to be able to pipe a config into the
  'funC start' command (e.g. via a /dev/fd/3 pseudo-filesystem path)
  [runc-config-via-stdin], and he has a working example that supports
  this without difficulty [ccon-config-via-stdin].  But since
  [runc-bundle-option] landed on 2015-11-16, runC has replaced their
  --config-file and --runtime-file flags with --bundle, and the
  current goal of this API is "keeping as much similarity with the
  existing runC command-line as possible", not "makes sense to Trevor"
  ;).  It looks like runC was reacting [runc-required-config-file] to
  strict wording in the spec [runtime-spec-required-config-file], so
  we might be able to revisit this if/when we lift that restriction.

* Having 'start' (now 'create') take a --state option to write state
  to a file [start-pr-state].  This is my preferred approach to
  sharing container state, since it punts a persistent state registry
  to higher-level tooling [punt-state-registry].  But runtime-spec
  currently requires the runtime to maintain such a registry
  [state-registry], and we don't need two ways to do that ;).

  On systems like Solaris, the kernel maintains a registry of
  container IDs directly, so they don't need an external registry
  [solaris-kernel-state].

* Having 'start' (now 'create') take an --id option instead of a
  required ID argument, and requiring the runtime to generate a unique
  ID if the option was not set.  When there is a long-running host
  process waiting on the container process to perform cleanup, the
  runtime-caller may not need to know the container ID.  However, runC
  has been requiring a user-specified ID since [runc-start-id], and
  this spec follows the early-exit 'create' from [runc-create-start],
  so we require one here.  We can revisit this if we regain a
  long-running 'create' process.

* Having 'create' take a '--console-socket PATH' option (required when
  process.terminal is true) with a path to a SOCK_SEQPACKET Unix
  socket for use with the console-socket protocol.  The current
  'LISTEN_FDS + 3' approach was proposed by Michael Crosby
  [console-socket-fd], but Trevor doesn't have a clear idea of the
  motivation for the change and would have preferred '--console-socket
  FD'.

* Having a long-running 'create' process.  Trevor is not a big fan of
  this early-exit 'create', which requires platform-specific magic to
  collect the container process's exit code.  The ptrace idea in this
  commit is from Mrunal [mrunal-ptrace].

  Trevor has a proposal for an 'event' operation [event] which would
  provide a convenient created trigger.  With 'event' in place, we
  don't need the 'create' process exit to serve as that trigger, and
  could have a long-running 'create' that collects the container
  process exit code using the portable waitid() family.  But the
  consensus after this week's meeting was to table that while we land
  docs for the runC API [mimic-runc].

* Having a 'version' command to make it easy for a caller to report
  which runtime they're using.  But we don't have a use-case that
  makes it strictly necessary for interop, so we're leaving it out for
  now [no-version].

* Using 'sh' syntax highlighting [syntax-highlighting] for the fenced
  code blocks.  The 'sh' keyword comes from [linguist-languages].  But
  the new fenced code blocks are shell sessions, not scripts, and we
  don't want shell-syntax highlighting in the command output.

[ccon-config-via-stdin]: https://github.com/wking/ccon/tree/v0.4.0#configuration
[console-socket-fd]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-10-19-21.00.log.html#l-30
[container-pid-from-state]: https://github.com/opencontainers/runtime-spec/pull/511/files#r70353376
  Subject: Add initial pass at a cmd line spec
[container-pid-not-portable]: opencontainers#459
  Subject: [ Runtime ] Allow for excluding pid from state
[coreutils-kill]: http://www.gnu.org/software/coreutils/manual/html_node/kill-invocation.html
[distinguish-unrecognized-commands]: https://github.com/wking/oci-command-line-api/pull/8/files#r46898167
  Subject: Clarity for commands vs global options
[docker-hcsshim]: https://github.com/docker/docker/pull/16997/files#diff-5d0b72cccc4809455d52aadc62329817R230
  moby/moby@bc503ca8 (Windows: [TP4] docker kill handling,
  2015-10-12, moby/moby#16997)
[event]: opencontainers#508
  Subject: runtime: Add an 'event' operation for subscribing to pushes
[extensions-unspecified]: http://ircbot.wl.linuxfoundation.org/eavesdrop/%23opencontainers/%23opencontainers.2016-07-27.log.html#t2016-07-27T16:37:56
[github-lists]: https://help.github.com/articles/basic-writing-and-formatting-syntax/#lists
[interface-versioning]: opencontainers#513 (comment)
[linguist-languages]: https://github.com/github/linguist/blob/master/lib/linguist/languages.yml
[linux-centric]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-07-13-17.03.log.html#l-39
[listen-fds-description]: opencontainers/runc#231 (comment)
  Subject: Systemd integration with runc, for on-demand socket
    activation
[markdown-syntax]: http://daringfireball.net/projects/markdown/syntax#list
[mimic-runc]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-07-13-17.03.log.html#l-15
[mrunal-ptrace]: http://ircbot.wl.linuxfoundation.org/eavesdrop/%23opencontainers/%23opencontainers.2016-07-13.log.html#t2016-07-13T18:58:54
[no-version]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-07-20-21.03.log.html#l-75
[optional-fail-fast]: wking/oci-command-line-api@527f3c6#commitcomment-14835617
  Subject: Use RFC 2119's keywords (MUST, MAY, ...)
[pep257-docstring]: https://www.python.org/dev/peps/pep-0257/#one-line-docstrings
[posix-kill]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/kill.html
[punt-state-registry]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2015/opencontainers.2015-12-02-18.01.log.html#l-79
[python-email-version]: https://docs.python.org/3/library/email.html#package-history
[rfc4627-s3]: https://tools.ietf.org/html/rfc4627#section-3
[rfc7158-aA]: https://tools.ietf.org/html/rfc7158#appendix-A
  RFC 7518 is currently identical to 7519.
[rfc7159-s8.1]: https://tools.ietf.org/html/rfc7159#section-8.1
[runc-bundle-option]: opencontainers/runc#373
  Subject: adding support for --bundle
[runc-config-via-stdin]: opencontainers/runc#202
  Subject: Can runc take its configuration on stdin?
[runc-listen-fds]: opencontainers/runc#231
  Subject: Systemd integration with runc, for on-demand socket
    activation
[runc-required-config-file]: opencontainers/runc#310 (comment)
  Subject: specifying a spec file on cmd line?
[runc-socket-type-version]: opencontainers/runc#1018 (comment)
  Subject: Consoles, consoles, consoles.
[runc-start-id]: opencontainers/runc#541
  opencontainers/runc@a7278cad (Require container id as arg1,
  2016-02-08, opencontainers/runc#541)
[runtime-spec-caller-api-agnostic]: opencontainers#113 (comment)
  Subject: Add fd section for linux container process
[runtime-spec-required-config-file]: https://github.com/opencontainers/runtime-spec/pull/210/files#diff-8b310563f1c6f616aa98e6aeffc4d394R14
  106ec2d (Cleanup bundle.md, 2015-10-02,
  opencontainers#210)
[sd_listen_fds]: http://www.freedesktop.org/software/systemd/man/sd_listen_fds.html
[separate-repository-proposed]: opencontainers#513 (comment)
[separate-repository-refused]: opencontainers#513 (comment)
[separate-repository-refusal-rebutted]: opencontainers#513 (comment)
[single-config-proposal]: https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/0QbyJDM9fWY
  Subject: Single, unified config file (i.e. rolling back specs#88)
  Date: Wed, 4 Nov 2015 09:53:20 -0800
  Message-ID: <20151104175320.GC24652@odin.tremily.us>
[solaris-kernel-state]: wking/oci-command-line-api#3 (comment)
  Subject: Drop exec, pause, resume, and signal
[start-pr-bundle]: wking/oci-command-line-api#11
  Subject: start: Change --config and --runtime to --bundle
[start-pr-state]: wking/oci-command-line-api#14
  Subject: start: Add a --state option
[state-pr]: wking/oci-command-line-api#16
  Subject: runtime: Add a 'state' command
[state-registry]: https://github.com/opencontainers/runtime-spec/pull/225/files#diff-b84a8d65d8ed53f4794cd2db7e8ea731R61
  7117ede (Expand on the definition of our ops, 2015-10-13,
  opencontainers#225)
[stdio-ioctl]: opencontainers#513 (comment)
[stdio-surprise]: opencontainers#513 (comment)
[syntax-highlighting]: https://help.github.com/articles/github-flavored-markdown/#syntax-highlighting
[systemd-pid]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-07-20-21.03.log.html#l-69
[util-linux-kill]: http://man7.org/linux/man-pages/man1/kill.1.html
[wikipedia-utf-8]: https://en.wikipedia.org/wiki/UTF-8
[wikipedia-posix-locale]: https://en.wikipedia.org/wiki/Locale#POSIX_platforms
[windows-singals]: https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/PlGKu7QUwLE
  Subject: Fwd: Windows support for OCI stop/signal/kill (runtime-spec#356)
  Date: Thu, 26 May 2016 11:03:29 -0700
  Message-ID: <20160526180329.GL17496@odin.tremily.us>

Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>
Hopefully-Signed-off-by: Mike Brown <brownwm@us.ibm.com>
Signed-off-by: W. Trevor King <wking@tremily.us>
Reviewed-by: Jesse Butler <jeeves.butler@gmail.com>
package socket

// Message is the normal data for messages passed on the console socket.
type Message struct {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the maintainer consensus being to move this PR over to runtime-tools, we'll need to figure out where these Go types should live. It would be nice if Go clients of the socket protocol (e.g. runC, which would rather not keep local copies) could import them without importing all of runtime-tools.

wking added a commit to wking/oci-command-line-api that referenced this pull request Feb 9, 2017
The command-line interface is largely orthogonal to the config format,
and config authors and runtime callers may be entirely different sets
of people.  Zhang Wei called for more explicit versioning for the CLI
[1].

Wedging multiple, independently versioned entities into a single
repository can be awkward, but earlier proposals to put the CLI in its
own repository [2] were unsuccessful because compliance testing
requires both a CLI and a config specification [3].  I don't think
that's a solid reason [4], but discussion along that line stalled out,
before being revived in today's meeting.  Unfortunately, now this CLI
spec is destined for runtime-tools [5], so we may still have multiple,
independently versioned entities in a single repository.  Wherever
this lands up, it's useful for this CLI spec to be clear about it's
own versioning.

[1]: opencontainers/runtime-spec#513 (comment)
[2]: opencontainers/runtime-spec#513 (comment)
[3]: opencontainers/runtime-spec#513 (comment)
[4]: opencontainers/runtime-spec#513 (comment)
[5]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2017/opencontainers.2017-02-08-22.00.log.html#l-125

Signed-off-by: W. Trevor King <wking@tremily.us>
wking added a commit to wking/oci-command-line-api that referenced this pull request Feb 9, 2017
As requested by Aleksa [1].

[1]: opencontainers/runtime-spec#513 (comment)

Signed-off-by: W. Trevor King <wking@tremily.us>
wking added a commit to wking/oci-command-line-api that referenced this pull request Feb 9, 2017
The command line API covered here spent the bulk of a year as a
runtime-spec PR [1], before the maintainers descided today that it
would be better off in runtime-tools [2] (or at least, not in
runtime-spec).  While it was in-flight for runtime-spec, I'd squashed
it down to one unweildy commit, following requests by Doug [3] and
Vincent [4].  Now that the eventual location of the API spec is less
clear (and there are calls for further review [5]), I've split the
semantic changes back out into individual commits to make the
motivation more obvious.  It also makes it easier for folks to PR
their recommendations directly, since review in the runtime-spec PR
was getting unweildy at 250-odd comments.

[1]: opencontainers/runtime-spec#513
[2]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2017/opencontainers.2017-02-08-22.00.log.html#l-125
[3]: opencontainers/runtime-spec#513 (comment)
[4]: opencontainers/runtime-spec#513 (comment)
[5]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2017/opencontainers.2017-02-08-22.00.log.html#l-113

* runtime-spec-513:
  runtime: Replace '$LISTEN_FDS + 3' with '--console-socket FD'
  runtime: Replace '--console-socket PATH' with '$LISTEN_FDS + 3'
  runtime: Mention TIOCSTI privilege escalation
  runtime: Add --console-socket for terminal handling
  runtime: Add --pid-file to 'create'
  runtime: Split 'create' and 'start' (and add 'delete')
  runtime: "application" -> "container"
  runtime: Move start's --id to a positional parameter
  runtime: Link to the spec's 'bundle' docs
  runtime: Add a 'kill' command
  runtime: Four-space indents for nested lists
  runtime: Drop the 'version' command
  runtime: Address unspecified commands and options
  runtime: Fix "MUST not" -> "MUST NOT"
  runtime: Drop 'sh' highlighting from fenced code blocks
  runtime: Semantically version this specification
  *: "OCI Runtime Command Line Interface"
  README: Update links and text
wking added a commit to wking/oci-command-line-api that referenced this pull request Feb 9, 2017
The command-line interface is largely orthogonal to the config format,
and config authors and runtime callers may be entirely different sets
of people.  Zhang Wei called for more explicit versioning for the CLI
[1].

Wedging multiple, independently versioned entities into a single
repository can be awkward, but earlier proposals to put the CLI in its
own repository [2] were unsuccessful because compliance testing
requires both a CLI and a config specification [3].  I don't think
that's a solid reason [4], but discussion along that line stalled out,
before being revived in today's meeting.  Unfortunately, now this CLI
spec is destined for runtime-tools [5], so we may still have multiple,
independently versioned entities in a single repository.  Wherever
this lands up, it's useful for this CLI spec to be clear about it's
own versioning.

[1]: opencontainers/runtime-spec#513 (comment)
[2]: opencontainers/runtime-spec#513 (comment)
[3]: opencontainers/runtime-spec#513 (comment)
[4]: opencontainers/runtime-spec#513 (comment)
[5]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2017/opencontainers.2017-02-08-22.00.log.html#l-125

Signed-off-by: W. Trevor King <wking@tremily.us>
wking added a commit to wking/oci-command-line-api that referenced this pull request Feb 9, 2017
As requested by Aleksa [1].

[1]: opencontainers/runtime-spec#513 (comment)

Signed-off-by: W. Trevor King <wking@tremily.us>
wking added a commit to wking/oci-command-line-api that referenced this pull request Feb 9, 2017
The command line API covered here spent the bulk of a year as a
runtime-spec PR [1], before the maintainers descided today that it
would be better off in runtime-tools [2] (or at least, not in
runtime-spec).  While it was in-flight for runtime-spec, I'd squashed
it down to one unweildy commit, following requests by Doug [3] and
Vincent [4].  Now that the eventual location of the API spec is less
clear (and there are calls for further review [5]), I've split the
semantic changes back out into individual commits to make the
motivation more obvious.  It also makes it easier for folks to PR
their recommendations directly, since review in the runtime-spec PR
was getting unweildy at 250-odd comments.

[1]: opencontainers/runtime-spec#513
[2]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2017/opencontainers.2017-02-08-22.00.log.html#l-125
[3]: opencontainers/runtime-spec#513 (comment)
[4]: opencontainers/runtime-spec#513 (comment)
[5]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2017/opencontainers.2017-02-08-22.00.log.html#l-113

* runtime-spec-513:
  runtime: Replace '$LISTEN_FDS + 3' with '--console-socket FD'
  runtime: Replace '--console-socket PATH' with '$LISTEN_FDS + 3'
  runtime: Mention TIOCSTI privilege escalation
  runtime: Add --console-socket for terminal handling
  runtime: Add --pid-file to 'create'
  runtime: Split 'create' and 'start' (and add 'delete')
  runtime: "application" -> "container"
  runtime: Move start's --id to a positional parameter
  runtime: Link to the spec's 'bundle' docs
  runtime: Add a 'kill' command
  runtime: Four-space indents for nested lists
  runtime: Drop the 'version' command
  runtime: Address unspecified commands and options
  runtime: Fix "MUST not" -> "MUST NOT"
  runtime: Drop 'sh' highlighting from fenced code blocks
  runtime: Semantically version this specification
  *: "OCI Runtime Command Line Interface"
  README: Update links and text
@wking
Copy link
Contributor Author

wking commented Feb 9, 2017 via email

@wking wking closed this Feb 9, 2017
wking added a commit to wking/oci-command-line-api that referenced this pull request Feb 10, 2017
The command line API covered here spent the bulk of a year as a
runtime-spec PR [1], before the maintainers descided today that it
would be better off in runtime-tools [2] (or at least, not in
runtime-spec).  While it was in-flight for runtime-spec, I'd squashed
it down to one unweildy commit, following requests by Doug [3] and
Vincent [4].  Now that the eventual location of the API spec is less
clear (and there are calls for further review [5]), I've split the
semantic changes back out into individual commits to make the
motivation more obvious.  It also makes it easier for folks to PR
their recommendations directly, since review in the runtime-spec PR
was getting unweildy at 250-odd comments.

[1]: opencontainers/runtime-spec#513
[2]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2017/opencontainers.2017-02-08-22.00.log.html#l-125
[3]: opencontainers/runtime-spec#513 (comment)
[4]: opencontainers/runtime-spec#513 (comment)
[5]: http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2017/opencontainers.2017-02-08-22.00.log.html#l-113

* runtime-spec-513:
  runtime: Replace '$LISTEN_FDS + 3' with '--console-socket FD'
  runtime: Replace '--console-socket PATH' with '$LISTEN_FDS + 3'
  runtime: Mention TIOCSTI privilege escalation
  runtime: Add --console-socket for terminal handling
  runtime: Add --pid-file to 'create'
  runtime: Split 'create' and 'start' (and add 'delete')
  runtime: "application" -> "container"
  runtime: Move start's --id to a positional parameter
  runtime: Link to the spec's 'bundle' docs
  runtime: Add a 'kill' command
  runtime: Four-space indents for nested lists
  runtime: Drop the 'version' command
  runtime: Address unspecified commands and options
  runtime: Fix "MUST not" -> "MUST NOT"
  runtime: Drop 'sh' highlighting from fenced code blocks
  runtime: Semantically version this specification
  *: "OCI Runtime Command Line Interface"
  README: Update links and text

Signed-off-by: W. Trevor King <wking@tremily.us>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CLI defined in runtime.md should mirror that provided by the reference implementation (runc).