Add create-runtime, create-container, start-container hooks #1008

RenaudWasTaken · 2019-04-24T22:59:17Z

Hello!

Since we discussed this during the last OCI weekly meeting, I wanted to continue the conversation on a more concrete change.
Related:

I've used the same names as LXC for the hooks. I'm happy to change them if we feel there are more appropriate names.
My reasoning while drafting this was to answer some of these questions:

Definition: How can we define the mount hook in a way that would clearly define the ability to mount "stuff" inside the container
Ordering: Is there an ordering that is required?
Here it would be helpful to define the ordering such that the pre-start hook detects that it doesn't need to be executed in a setting where we specify both pre-start hook and mount hook for backwards compat reasons.

Questions still open:

Should we define "outside" and "inside" (in relation to the VM debate)

What do you think about this first draft?

Signed-off-by: Renaud Gaubert rgaubert@nvidia.com

RenaudWasTaken · 2019-04-24T22:59:49Z

cc @vbatts @crosbymichael @cyphar @alban

vbatts · 2019-04-30T17:38:49Z

cc @cyphar

config.md

alban · 2019-04-30T20:54:35Z

config.md

+The mount hooks MUST be called as part of the create operation at a time that would allow the hook to execute operations both inside and outside the container.
+For example, on linux this would happen before the `pivot_root` operation is executed but after the mount namespace was created and setup.
+
+For runtimes that implement the deprecated pre-start hooks as mount hooks, mount hooks MUST be called after the prestart hooks.


I'm not sure we should specify the order. Since the definition of the prestart hook is not changed, this order makes it impossible to implement a container runtime with support for the two hooks following the spec. And for compatibility, hooks installed on both prestart and mount could both check if the mount task is already done, so the order should not matter.

And for compatibility, hooks installed on both prestart and mount could both check if the mount task is already done, so the order should not matter.

The goal here was that you can rely on ordering instead of checking for state you might have persisted in the mount or prestart hook.

this order makes it impossible to implement a container runtime with support for the two hooks following the spec

Do you mind detailing a bit more what you mean here :) ?

Do you mind detailing a bit more what you mean here :) ?

Sorry, I take it back.

So now, the spec mandates steps to happen in this order:

the container namespaces are created

Prestart hooks

Mount hooks

pivot_root

Start hooks

The user-specified process is executed

Poststart hooks

I thought the spec was saying that the Prestart hooks must happen after the namespaces are fully configured (including the pivot_root), which would make things impossible. But the wording is more vague:

[The pre-start hooks] are called after the container namespaces are created

So I think it's ok :)

RenaudWasTaken · 2019-05-01T07:17:18Z

Thanks for your review @alban, will get to them tomorrow!

cyphar · 2019-05-01T09:46:04Z

config.md

+### <a name="configHooksMount" />Mount Hooks
+
+The mount hooks MUST be called as part of the create operation at a time that would allow the hook to execute operations both inside and outside the container.
+For example, on linux this would happen before the `pivot_root` operation is executed but after the mount namespace was created and setup.


linux -> Linux.

RenaudWasTaken · 2019-05-01T17:49:13Z

Comments addressed :)

RenaudWasTaken · 2019-05-06T23:47:26Z

Ping @cyphar @vbatts @alban :)

h-vetinari · 2019-06-01T20:33:59Z

Any update on this?

RenaudWasTaken · 2019-06-22T00:43:27Z

@cyphar can you look into this :) ?
We would be happy to have your signoff!

RenaudWasTaken · 2019-06-26T21:39:32Z

Ping @cyphar

RenaudWasTaken · 2019-07-25T22:30:30Z

Ping @cyphar @vbatts

h-vetinari · 2019-08-14T08:09:56Z

Can anyone provide an update/review here? Pings based on recent merges
@tianon @crosbymichael @hqhq @vbatts @caniszczyk

RenaudWasTaken · 2019-08-26T04:37:37Z

Ping @cyphar @vbatts

RenaudWasTaken · 2019-08-28T22:10:11Z

@cyphar @vbatts can you guys take a stab at reviewing this :) ?

Thanks!

kad · 2019-09-05T08:25:03Z

config.md

@@ -384,6 +384,10 @@ For POSIX platforms, the configuration structure supports `hooks` for configurin
 * **`env`** (array of strings, OPTIONAL) with the same semantics as [IEEE Std 1003.1-2008's `environ`][ieee-1003.1-2008-xbd-c8.1].
 * **`timeout`** (int, OPTIONAL) is the number of seconds before aborting the hook.
 If set, `timeout` MUST be greater than zero.
+ * **`mount`** (array of objects, OPTIONAL) is an array of [mount hooks](#mount-hooks).


"mount" is a really bad naming selection. It is applicable to single vendor at the moment (NVidia) and from naming denotes other usages. Currently prestart hook is used by others not only to do mounts, but also to do various devices initialization. If we want to be explicit in defining execution timings of the hooks, let's define that, but please don't imply in the names single use case of the hook.

bart0sh · 2019-09-05T09:01:10Z

config.md

@@ -375,7 +375,7 @@ For Windows based systems the user structure has the following fields:
 For POSIX platforms, the configuration structure supports `hooks` for configuring custom actions related to the [lifecycle](runtime.md#lifecycle) of the container.

 * **`hooks`** (object, OPTIONAL) MAY contain any of the following properties:
- * **`prestart`** (array of objects, OPTIONAL) is an array of [pre-start hooks](#prestart).
+ * **`prestart`** (array of objects, OPTIONAL, **DEPRECATED**) is an array of [pre-start hooks](#prestart).


I'd propose to not deprecate prestart hooks. They may be used for other purposes than mount and start hooks. For example our FPGA device plugin verifies and programs FPGA devices in a prestart hook.

RenaudWasTaken · 2019-09-06T21:44:35Z

I'd propose to not deprecate prestart hooks. They may be used for other purposes than mount and start hooks. For example our FPGA device plugin verifies and programs FPGA devices in a prestart hook.
"mount" is a really bad naming selection. It is applicable to single vendor at the moment (NVidia) and from naming denotes other usages. Currently prestart hook is used by others not only to do mounts, but also to do various devices initialization. If we want to be explicit in defining execution timings of the hooks, let's define that, but please don't imply in the names single use case of the hook.

Let's take a step back :)

Pre-start hooks are not getting deprecated. In the spec, pre-start hooks are defined as follows:

The pre-start hooks MUST be called after the start operation is called but before the user-specified program command is executed.

After discussing this in opencontainers/runc#1811 we realized that this meant that this meant that pre-start hooks needed to be called after the pivot_root operation.
However, runc implemented pre-start hooks such that they were called before the pivot_root operation.

After discussion, we decided to clarify the meaning of pre-start hooks and allow to have hooks both "before" and "after" pivot_root.

Hence, we came up with the two new hooks:

mount hooks, which would be called at the exact same time as the current pre-start hook as currently implemented in runc
start hooks which would be called at the exact same time as the one defined in the spec.

So pre-start hooks aren't being deprecated "per-se", they are being renamed. The names were chosen based on LXC's current hooks but if you feel strongly about another name, I'll happily change the PR.

kad · 2019-09-09T15:42:50Z

@RenaudWasTaken the problem is that "mount" implies a lot of meaning and does more confusion than clarification in comparison with current prestart name.

I would rather do the names that will explicitly refer to environment where it is executed (before/after pivot_root, to be explicit). Something like: keep current prestart name as it is, with adding clarification to the spec that it is executed in the host namespace and something like containerprestart to explicitly indicate that it will be executed in the container namespace.
Or something like you mentioned inside / outside combined with operations pre/post start/stop. Names should be descriptive enough for sysadmin who will be configuring those to understand without going and reading deeply OCI spec.

RenaudWasTaken · 2019-10-01T21:58:48Z

Something like: keep current prestart name as it is, with adding clarification to the spec that it is executed in the host namespace

That's unfortunately not an option, clarifying it either way is a breaking change.

containerprestart to explicitly indicate that it will be executed in the container namespace.

I have no strong feelings on the naming, containerprestart seems fine. The naming used here is the same as the one chosen by LXC with the intention to cause less confusion.

If we think there is more value in using a different name that is clearer than I'm happy to change this PR.

cyphar · 2019-10-02T00:21:13Z

config.md

@@ -399,6 +403,20 @@ The [state](runtime.md#state) of the container MUST be passed to hooks over stdi
 The pre-start hooks MUST be called after the [`start`](runtime.md#start) operation is called but [before the user-specified program command is executed](runtime.md#lifecycle).
 On Linux, for example, they are called after the container namespaces are created, so they provide an opportunity to customize the container (e.g. the network namespace could be specified in this hook).

+Note: Prestart hooks were deprecated in favor of mount and start hooks, which have a clearer definition.


I would explicitly mention that runc incorrectly implemented them, to ensure users don't depend on them.

Are we going to use -host and -container or -inside and -outside suffixes to the hook names? It's still not clear to me if proposed hooks will be executed in the host or container namespace. "mount" looks quite vague to me in this regard. It requires some guesswork to understand its purpose. Would it be better to have prestart-host and prestart-container instead?

@cyphar @RenaudWasTaken Can you clarify the statement about "runc incorrectly implemented them" ? From what I see in https://github.com/opencontainers/runtime-spec/blob/master/config.md#posix-platform-hooks it says "Hooks MUST be executed in the runtime namespace.", and prestart is executed accordingly to this specification.

The pre-start hooks MUST be called after the start operation is called but before the user-specified program command is executed.

This points to having the hook after the pivot_root is called :)

https://github.com/opencontainers/runtime-spec/blob/master/config.md#prestart

@bart0sh, is the naming something that can be resolved by better detailling the hooks in the spec?

Today they are worded as follows:
mount hooks

The mount hooks MUST be called as part of the create operation at a time that would allow the hook to execute operations both inside and outside the container.
For example, on Linux this would happen before the pivot_root operation is executed but after the mount namespace was created and setup.

start hooks

The start hooks MUST be called before the user-specified process is executed as part of the start operation.
This hook can be used to execute some operations in the container, for example running the ldconfig binary on linux before the container process is spawned.

Do you think we could reword the definition in a different way that would reduce the confusion?

The pre-start hooks MUST be called after the start operation is called but before the user-specified program command is executed.

This points to having the hook after the pivot_root is called :)
https://github.com/opencontainers/runtime-spec/blob/master/config.md#prestart

Well, not really. It specifies the timing. But where hook itself is called is specified by line that I quoted from spec above: "Hooks MUST be executed in the runtime namespace", and runtime namespace defined as https://github.com/opencontainers/runtime-spec/blob/master/glossary.md#runtime-namespace

"After pivot_root" will be already not a runtime namespace.

Prestart hooks were deprecated in favor of mount and start hooks

mount and start hooks are executed in container namespace. Prestart hooks executed in runtime namespace. They can't be deprecated in favor of something that runs in a different namespace.

cyphar · 2019-10-02T00:24:26Z

@kad runc will continue to support prestart indefinitely as-is (internally mapping them to mount hooks). This is precisely the reason why we need to have new hook names, to stop breaking existing users. But prestart is wrong as-is and so we need to deprecate it (to stop users from using it given the mismatch between runc and runtime-spec).

@RenaudWasTaken I think copying the LXC hooks is the only reasonable choice. I don't want to get caught up in bike-shedding over what hook names we should use.

From my side, this LGTM -- as long as the NVIDIA folks are happy with it (sorry for the long time to review, I've been swamped with openat2 and related work).

RenaudWasTaken · 2019-10-02T00:42:06Z

I've updated the PR :)

cyphar · 2019-10-02T00:43:04Z

/cc @opencontainers/runtime-spec-maintainers

h-vetinari · 2019-10-02T10:06:49Z

@bart0sh: Would it be better to have prestart-host and prestart-container instead?

I like this idea. It keeps the original name "prestart", and clarifies how it has been split up.

@RenaudWasTaken: I've updated the PR :)

Did you forget to push? ;-)

@cyphar: sorry for the long time to review, I've been swamped with openat2 and related work

Glad to see this moving forward again in any case! I'm guessing you're thinking about solving this issue already for runc-1.0.0rc9, which I'm also guessing should go out quickly due to the new CVE? But IMO that shouldn't rush the decision of the naming here, which the community will be stuck with for a long time.

If my surmise about the situation is correct, how about releasing rc9 first and then solving the hooks under less time pressure?

RenaudWasTaken · 2019-10-02T17:11:02Z

Did you forget to push? ;-)

Thanks! Updated

@bart0sh: Would it be better to have prestart-host and prestart-container instead?
I like this idea. It keeps the original name "prestart", and clarifies how it has been split up.

The issue that I faced with using the host word, is that it becomes pretty confusing as to what it means for a vm based runtime (is host the hypervisor or the guest?).

It seemed to me that keeping existing terminology (i.e: used in LXC) was a lot less confusing since a hook author is likely to be writing hooks for both OCI compliant runtimes (runc, ...) and non OCI compliant runtimes (LXC).

RenaudWasTaken · 2019-12-24T00:07:32Z

In the latest update, the following changes were made:

create-container has now the same structure as the other hooks (before it had a field called pathContainer rather than path)
I specified for each hook in what namespace the path value should be resolved
I specified for each hook in what namespace it should be executed

giuseppe

left some minor comments.

giuseppe · 2019-12-24T13:40:16Z

schema/defs.json

@@ -108,6 +108,12 @@
 "$ref": "#/definitions/Hook"
 }
 },
+ "ArrayOfContainerHooks": {


should this chunk be dropped now?

Addressed, thanks for noticing!

giuseppe · 2019-12-24T13:56:14Z

specs-go/config.go

@@ -125,13 +125,26 @@ type Hook struct {
 Timeout *int `json:"timeout,omitempty"`
 }

+// Hook specifies a command that is run in the container at a particular event in the lifecycle of a container


here it should start with // Hooks .... or leave an empty line after the comment.

Otherwise "make .golint" will complain about it

Addressed, thanks for noticing!

giuseppe · 2019-12-24T14:04:44Z

runtime.md

+3. The [`prestart` hooks](config.md#prestart) MUST be invoked by the runtime.
+ If any `prestart` hook fails, the runtime MUST [generate an error](#errors), stop the container, and continue the lifecycle at step 12.
+4. The [`create-runtime` hooks](config.md#create-runtime-hooks) MUST be invoked by the runtime.
+ If any `create-runtime` hook fails, the runtime MUST [generate an error](#errors), stop the container, and continue the lifecycle at step 12.


should create-runtime and create-container be specified in camel case as well as in config.md? At least the schema and the Go code seems to use the camel case version.

Signed-off-by: Renaud Gaubert <rgaubert@nvidia.com>

giuseppe

LGTM

cyphar · 2019-12-27T01:29:38Z

LGTM from my side.

crosbymichael · 2019-12-30T19:21:55Z

LGTM

h-vetinari · 2020-03-31T12:32:50Z

I'm quite sure no-one wants to receive a ping from this thread ever again, but since some parts of the new hooks were left un(der)specified, I opened a follow-up issue to track things: #1039.

In PR opencontainers#1008 3 new steps in the lifecycle were added and the referencing of the states to the corresponding lifecycle step was not adjusted. Corrected based on Tag v1.0.0-rc6.

In PR opencontainers#1008 3 new steps in the lifecycle were added and the referencing of the states to the corresponding lifecycle step was not adjusted. Corrected based on Tag v1.0.0-rc6. Signed-off-by: Pascal Theml <pascal@familie-theml.de>

Poststart hooks exist in runc since 2015 [1], and since that time until today, if a hook returned an error, runc kills the container. In 2020, commit c166268 (PR opencontainers#1008) added the following text (which became part of runtime-spec release v1.0.2): > 9. The `poststart` MUST be invoked by the runtime. If any > `poststart` hook fails, the runtime MUST log a warning, but the > remaining hooks and lifecycle continue as if the hook had succeeded. Now, this text conflicted with the pre-existing runtime (runc) behavior, and it still conflicts with the current runc behavior. At this point, we can either fix runtimes or the spec. To my mind, fixing the spec is a better approach, because: - initial implementation predates the spec wording by a few years; - the wording in the spec was never implemented (in runc); - returning an error (and stopping the container) seems like a more versatile approach, since a hook can usually choose whether to return an error or not. [1]: opencontainers/runc#392 Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

alban reviewed Apr 30, 2019

View reviewed changes

cyphar reviewed May 1, 2019

View reviewed changes

RenaudWasTaken force-pushed the hooks branch from 41a9d90 to 7edf277 Compare May 1, 2019 17:48

cyphar mentioned this pull request May 15, 2019

Tag a new release #1010

Closed

h-vetinari mentioned this pull request Jul 8, 2019

libcontainer: call Prestart, Poststart hooks from correct places opencontainers/runc#1811

Closed

mikebrow mentioned this pull request Aug 27, 2019

[WIP] adds option for setting default oci hooks containerd/cri#1248

Closed

kad suggested changes Sep 5, 2019

View reviewed changes

bart0sh reviewed Sep 5, 2019

View reviewed changes

cyphar reviewed Oct 2, 2019

View reviewed changes

RenaudWasTaken force-pushed the hooks branch from 7edf277 to 625e01a Compare October 2, 2019 16:57

RenaudWasTaken force-pushed the hooks branch 2 times, most recently from 8c0b611 to 3720456 Compare December 24, 2019 00:03

giuseppe reviewed Dec 24, 2019

View reviewed changes

RenaudWasTaken force-pushed the hooks branch from 3720456 to 0351bd8 Compare December 25, 2019 21:34

Add create-container, create-runtime and start-container hooks

c166268

Signed-off-by: Renaud Gaubert <rgaubert@nvidia.com>

RenaudWasTaken force-pushed the hooks branch from 0351bd8 to c166268 Compare December 25, 2019 21:45

giuseppe approved these changes Dec 26, 2019

View reviewed changes

cyphar mentioned this pull request Dec 27, 2019

Fix race checking for process exit and waiting for exec fifo opencontainers/runc#2185

Merged

h-vetinari approved these changes Dec 29, 2019

View reviewed changes

crosbymichael merged commit e09c7c4 into opencontainers:master Dec 30, 2019

vbatts mentioned this pull request Jan 9, 2020

Release v1.0.2 #1026

Closed

RenaudWasTaken changed the title ~~Add create-host, create-container, start-container hooks~~ Add create-runtime, create-container, start-container hooks Feb 18, 2020

h-vetinari mentioned this pull request Feb 18, 2020

Add CreateRuntime, CreateContainer and StartContainer Hooks opencontainers/runc#2229

Merged

h-vetinari mentioned this pull request Mar 31, 2020

Follow-up to under-specification of createRuntime / createContainer hooks #1039

Open

solarkennedy mentioned this pull request Dec 7, 2020

Support the new hook types of newer oci spec awslabs/oci-add-hooks#9

Closed

AkayaKuromoto mentioned this pull request Aug 10, 2021

fix the lifecycle reference in the states listing #1118

Merged

johananl mentioned this pull request Sep 19, 2022

Replace deprecated OCI prestart hook with createRuntime inspektor-gadget/inspektor-gadget#951

Closed

h-vetinari mentioned this pull request Oct 31, 2022

runtime: move prestart hook in lifecycle section #1167

Closed

h-vetinari mentioned this pull request Jan 11, 2023

Runtime and Lifecycle:Operations: why split create cmd and start cmd? #1173

Open

cyphar mentioned this pull request Jul 19, 2023

blog: Add "OCI Runtime Spec v1.1" opencontainers/opencontainers.org#133

Merged

thaJeztah mentioned this pull request Sep 29, 2023

specs-go: add missing deprecation comment for Hooks.Prestart #1232

Merged

This was referenced Jul 17, 2024

runtime: fail when a poststart hook fails #1262

Open

fix runc's poststart behaviour doesn't match the runtime-spec opencontainers/runc#4348

Draft

Add create-runtime, create-container, start-container hooks #1008

Add create-runtime, create-container, start-container hooks #1008

Conversation

RenaudWasTaken commented Apr 24, 2019

RenaudWasTaken commented Apr 24, 2019 • edited Loading

vbatts commented Apr 30, 2019

Choose a reason for hiding this comment

RenaudWasTaken May 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RenaudWasTaken commented May 1, 2019

Choose a reason for hiding this comment

RenaudWasTaken commented May 1, 2019

RenaudWasTaken commented May 6, 2019

h-vetinari commented Jun 1, 2019

RenaudWasTaken commented Jun 22, 2019

RenaudWasTaken commented Jun 26, 2019

RenaudWasTaken commented Jul 25, 2019

h-vetinari commented Aug 14, 2019

RenaudWasTaken commented Aug 26, 2019

RenaudWasTaken commented Aug 28, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RenaudWasTaken commented Sep 6, 2019

kad commented Sep 9, 2019

RenaudWasTaken commented Oct 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyphar commented Oct 2, 2019

RenaudWasTaken commented Oct 2, 2019

cyphar commented Oct 2, 2019

h-vetinari commented Oct 2, 2019 • edited Loading

RenaudWasTaken commented Oct 2, 2019

RenaudWasTaken commented Dec 24, 2019

giuseppe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

giuseppe left a comment

Choose a reason for hiding this comment

cyphar commented Dec 27, 2019

crosbymichael commented Dec 30, 2019 • edited by caniszczyk Loading

h-vetinari commented Mar 31, 2020 • edited Loading

RenaudWasTaken commented Apr 24, 2019 •

edited

Loading

RenaudWasTaken May 1, 2019 •

edited

Loading

h-vetinari commented Oct 2, 2019 •

edited

Loading

crosbymichael commented Dec 30, 2019 •

edited by caniszczyk

Loading

h-vetinari commented Mar 31, 2020 •

edited

Loading