Skip to content
This repository has been archived by the owner on Feb 24, 2020. It is now read-only.

Commit

Permalink
docs/devel: add logging and attaching design doc
Browse files Browse the repository at this point in the history
  • Loading branch information
lucab committed Dec 13, 2016
1 parent 7ab0456 commit 1218b4d
Show file tree
Hide file tree
Showing 2 changed files with 308 additions and 0 deletions.
18 changes: 18 additions & 0 deletions Documentation/devel/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,25 @@ Additionally, since appc dependencies can be found only via discovery, a depende

Given this 1:N relation between an ACI and their rendered images, the ACIStore and TreeStore are decoupled.


## Logging and attaching

Applications running inside a rkt pod can produce output on stdout/stderr, which can be redirected at runtime. Optionally, they can receive input on stdin from an external component that can be attached/detached during execution.

The internal architecture for attaching (TTY and single streams) and logging is described in full details in the [Logging and attaching design document][attach-design].

For each application, rkt support separately configuring stdin/stdout/stderr via runtime command-line flags. The following modes are available:
* interactive: application will be run under the TTY of the parent process. A single application is allowed in the pod, which is tied to the lifetime of the parent terminal and cannot be later re-attached
* TTY: selected I/O streams will be run under a newly allocated TTY, which can be later used for external attaching
* streaming: selected I/O streams will be supervised by a separate multiplexing process (running in the pod context). They can be later externally attached
* logging: selected output streams will be supervised by a separate logging process (running in the pod context). Output entries will be handled as log entries, and the application cannot be later re-attached
* null: selected I/O streams will be closed. Application will not received the file-descriptor for the corresponding stream, and it cannot be later re-attached

From a UI perspective, main consumers of the logging and attaching subsystem are the `rkt attach` subcommand and the `--stdin`, `--stdout`, `--stderr` runtime options.


[acirenderer]: https://github.com/appc/spec/tree/master/pkg/acirenderer
[attach-design]: ./log-attach-design.md
[appc-spec]: https://github.com/appc/spec
[appc-dependencies]: https://github.com/appc/spec/blob/master/spec/aci.md#image-manifest-schema
[appc-discovery]: https://github.com/appc/spec/blob/master/spec/discovery.md
Expand Down
290 changes: 290 additions & 0 deletions Documentation/devel/log-attach-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,290 @@
# Logging and attaching design

## Overview

rkt can run multiple applications in a pod, under a supervising process and alongside with a sidecar service which takes care of multiplexing its I/O toward the outside world.

Historically this has been done via systemd-journald only, meaning that all logging was handled via journald and interactive applications had to re-use a parent TTY.

Starting from systemd v232, it is possible to connect a service streams to arbitrary socket units and let custom sidecar multiplex all the I/O.

This document describes the architectural design for the current logging and attaching subsystem, which allows custom logging and attaching logic.

## Runtime modes

In order to be able to attach or apply custom logging logic to applications, an appropriate runtime mode must be specified when adding/preparing an application inside a pod.

This is done via stage0 CLI arguments (`--stdin`, `--stdout`, and `--stder`) which translate into per-application stage2 annotations.

### Interactive mode

This mode results in the application having the corresponding stream attached to the parent terminal.

For historical reasons and backward compatibility, this is a special mode activated via `--interactive` and only supports single-app pods.

Interactive mode does not support attaching and ties the runtime to the lifetime of the parent terminal.

Internally, this translates to an annotation at the app level:

```
{
"name": "coreos.com/rkt/stage2/stdin",
"value": "interactive"
},
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "interactive"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "interactive"
}
```

In this case, the corresponding service unit file gains the following properties:

```
[Service]
StandardInput=tty
StandardOutput=tty
StandardError=tty
...
```

No further sidecar dependencies are introduced in this case.

### TTY mode

This mode results in the application having the corresponding stream attached to a dedicated pseudo-terminal.

This is different from the "interactive" mode because:
* it allocates a new pseudo-terminal accounted towards pod resources
* it supports external attaching/detaching
* it supports multiple applications running inside a single pod
* it does not tie the pod lifetime to the parent terminal one

Internally, this translates to an annotation at the app level:

```
{
"name": "coreos.com/rkt/stage2/stdin",
"value": "tty"
},
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "tty"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "tty"
}
```

In this case, the corresponding service unit file gains the following properties:

```
[Service]
TTYPath=/rkt/iomux/<appname>/stage2-pts
StandardInput=tty
StandardOutput=tty
StandardError=tty
...
```

A sidecar dependency to `ttymux@.service` is introduced in this case. Application has a `Wants=` and `After=` relationship to it.

### Streaming mode

Internally, this translates to an annotation at the app level:

```
{
"name": "coreos.com/rkt/stage2/stdin",
"value": "stream"
},
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "stream"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "stream"
}
```

In this case, the corresponding service unit file gains the following properties:

```
[Service]
StandardInput=fd
Sockets=<appname>-stdin.socket
StandardOutput=fd
Sockets=<appname>-stdout.socket
StandardError=fd
Sockets=<appname>-stderr.socket
...
```

A sidecar dependency to `iomux@.service` is introduced in this case. Application has a `Wants=` and `Before=` relationship to it.

Additional per-stream socket units are generated, as follows:

```
[Unit]
Description=<stream> socket for <appname>
DefaultDependencies=no
StopWhenUnneeded=yes
RefuseManualStart=yes
RefuseManualStop=yes
BindsTo=<appname>.service
[Socket]
RemoveOnStop=yes
Service=<appname>.service
FileDescriptorName=<stream>
ListenFIFO=/rkt/iottymux/<appname>/stage2-<stream>
```

### Logging mode

This mode results in the application having the corresponding stream attached to journald.

This is the default mode for stdout/stderr, for historical reasons and backward compatibility.

Internally, this translates to an annotation at the app level:

```
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "log"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "log"
}
```

In this case, the corresponding service unit file gains the following properties:

```
[Service]
StandardOutput=journal
StandardError=journal
...
```

A sidecar dependency to `systemd-journald.service` is introduced in this case. Application has a `Wants=` and `After=` relationship to it.

Logging is not a valid mode for stdin.

### Null mode

This mode results in the application having the corresponding stream closed.

This is the default mode for stdin, for historical reasons and backward compatibility.

Internally, this translates to an annotation at the app level:

```
{
"name": "coreos.com/rkt/stage2/stdin",
"value": "null"
},
{
"name": "coreos.com/rkt/stage2/stdout",
"value": "null"
},
{
"name": "coreos.com/rkt/stage2/stderr",
"value": "null"
}
```

In this case, the corresponding service unit file gains the following properties:

```
[Service]
StandardInput=null
StandardOutput=null
StandardError=null
[...]
```

No further sidecar dependencies are introduced in this case.

## Annotations

The following per-app annotations are defined for internal use, with the corresponding set of allowed values:

* `coreos.com/rkt/stage2/stdin`
- `interactive`
- `null`
- `stream`
- `tty`
* `coreos.com/rkt/stage2/stdout`
- `interactive`
- `log`
- `null`
- `stream`
- `tty`
* `coreos.com/rkt/stage2/stderr`
- `interactive`
- `log`
- `null`
- `stream`
- `tty`


## Stage1 internals

## Attaching

### Endpoint listing

### Automatic attaching

### Custom attaching

## Logging

### Journald

This is the default output multiplexer for stdout/stderr in logging mode, for historical reasons and backward compatibility.

Restrictions:
* requires journalctl (or similar libsystemd-based helper) to decode output entries
* requires a libsystemd on the host compiled with LZ4 support
* systemd-journald does not support distinguishing between entries from stdout and stderr

### Experimental logging modes

TODO(lucab): k8s logmode

## Sidecars

### systemd-journald

### iottymux

iottymux is a multi-purpose stage1 binary. It currently serves the following purposes:
* Multiplex I/O over TTY (in TTY mode)
* Multiplex I/O from streams (in streaming mode)
* Attach to existing attachable applications (in TTY or streaming mode)

#### iomux

This component takes care of multiplexing dedicated streams and receiving clients for attaching.

It is started as an instance of the template `iomux@.service` service by a `Before=` dependency from the application.

#### ttymux

This component takes care of multiplexing TTY and receiving clients for attaching.

It is started as an instance of the templated `ttymux@.service` service by a `After=` dependency from the application.

#### iottymux-attach

This component takes care of discovering endpoints and attaching to them, both for TTY and streaming modes.

It is invoked by the "stage1" attach entrypoint and completely run in stage1 context. It is implemented as a sub-action of the main `iottymux` binary.

0 comments on commit 1218b4d

Please sign in to comment.