Skip to content

Commit 61013ac

Browse files
kcbimontekcbimonte
committed
docx: container placement and vault / credhub docs
Signed-off-by: Kevin Bimonte <kbimonte@gmail.com>
1 parent 5f81952 commit 61013ac

File tree

3 files changed

+451
-6
lines changed

3 files changed

+451
-6
lines changed

‎docs/docs/operation/container-placement.md‎

Lines changed: 124 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,28 +2,148 @@
22
title: Container Placement
33
---
44

5+
Each [step](../steps/index.md) in a build is executed inside a container. The [`web` node](../install/running-web.md)
6+
distributes containers across the worker cluster depending on the configured strategy. If no workers satisfy the
7+
configured strategy, the [step](../steps/index.md) will block until a worker becomes available.
8+
59
## The `volume-locality` strategy
610

11+
When using `volume-locality`, the [`web` node](../install/running-web.md) places [`task` step](../steps/task.md)
12+
and [`put` step](../steps/put.md) containers on workers where a majority of their inputs are already present. **This is
13+
the default strategy**.
14+
15+
The advantage of this approach is that it reduces the likelihood that large artifacts will have to be streamed from
16+
one [`worker` node](../install/running-worker.md), through the [`web` node](../install/running-web.md), and to the
17+
target `worker` node. For large artifacts, this can result in quite a bit of overhead.
18+
19+
The disadvantage of this approach is that it can sometimes result in builds "gravitating" to a particular worker and
20+
overloading it, at least until the resource caches warm across the worker pool. This disadvantage can be partially
21+
mitigated using the (currently experimental) [`limit-active-volumes` strategy](#the-limit-active-volumes-strategy) in
22+
conjunction with [Chaining Placement Strategies](#chaining-placement-strategies).
23+
24+
If your builds tend to be light on artifacts and heavy on task execution, you may want to try
25+
the [`fewest-build-containers` strategy](#the-fewest-build-containers-strategy) or the (currently
26+
experimental) [`limit-active-tasks` strategy](#the-limit-active-tasks-strategy).
27+
728
## The `fewest-build-containers` strategy
829

30+
When using the `fewest-build-containers` strategy, step containers (`get`, `put`, `task`) are placed on the worker that
31+
has the fewest build containers (i.e. containers for other steps of other builds).
32+
33+
!!! info
34+
35+
Containers used for resource checks are not counted because they are long-living containers that get re-used for
36+
multiple checks, and therefore consume very little resources on the worker.
37+
38+
To use this strategy, set the following env var on the [`web` node](../install/running-web.md):
39+
40+
```properties
41+
CONCOURSE_CONTAINER_PLACEMENT_STRATEGY=fewest-build-containers
42+
```
43+
944
## The `random` strategy
1045

46+
With the `random` strategy, the [`web` node](../install/running-web.md) places `get`, `put`, and `task` containers on
47+
any worker, ignoring any affinity.
48+
49+
As this is truly random, this will be fine until one day it's not fine.
50+
51+
To use this strategy, set the following env var on the [`web` node](../install/running-web.md):
52+
53+
```properties
54+
CONCOURSE_CONTAINER_PLACEMENT_STRATEGY=random
55+
```
56+
1157
## The `limit-active-tasks` strategy
1258

13-
!!! warning
59+
!!! warning "Experimental Feature"
1460

1561
`limit-active-tasks` is an experimental feature.
1662

63+
When selecting the `limit-active-tasks` placement strategy, each `task` executed on a worker will increase the number
64+
of "active tasks" on that worker by one. When the task completes the number is decreased by one.
65+
The [`web` node](../install/running-web.md) then places `get`, `put`, and `task` containers on the worker that currently
66+
has the _least amount of active tasks_.
67+
68+
Additionally, `max-active-tasks-per-worker` can be set to an integer of 1 or more, in which case a worker will not
69+
execute more than that amount of **tasks**. A value of 0 means that there is no limit on the maximum number of active
70+
tasks on the workers. If no worker can be selected because all of them already have `max-active-tasks-per-worker` active
71+
tasks, then the task will wait for a free worker, periodically polling the pool. The
72+
metric `concourse_steps_waiting{type="task"}` is emitted to monitor these events. Note that the parameter does not apply
73+
to `get` and `put` steps which will always be scheduled on the worker with the fewest active tasks.
74+
75+
```properties
76+
CONCOURSE_CONTAINER_PLACEMENT_STRATEGY=limit-active-tasks
77+
```
78+
79+
and, optionally
80+
81+
```properties
82+
CONCOURSE_MAX_ACTIVE_TASKS_PER_WORKER=1
83+
```
84+
1785
## The `limit-active-containers` strategy
1886

19-
!!! warning
87+
!!! warning "Experimental Feature"
2088

2189
`limit-active-containers` is an experimental feature.
2290

91+
The `limit-active-containers` placement strategy rejects workers that already have too many containers. It makes no
92+
effort to find the worker with the fewest number of containers present, and is therefore most useful when combined with
93+
other placement strategies by [Chaining Placement Strategies](#chaining-placement-strategies).
94+
95+
`max-active-containers-per-worker` can be set to an integer of 1 or more, in which case a worker will not execute more
96+
than that amount of **containers**. If unset (or set to a value of 0), the `limit-active-containers` strategy has no
97+
effect - if this is your only placement strategy, workers will be chosen at random.
98+
99+
```properties
100+
CONCOURSE_CONTAINER_PLACEMENT_STRATEGY=limit-active-containers
101+
CONCOURSE_MAX_ACTIVE_CONTAINERS_PER_WORKER=200
102+
```
103+
23104
## The `limit-active-volumes` strategy
24105

25-
!!! warning
106+
!!! warning "Experimental Feature"
26107

27108
`limit-active-volumes` is an experimental feature.
28109

29-
## Chaining Placement Strategies
110+
The `limit-active-volumes` placement strategy rejects workers that already have too many volumes. It makes no effort to
111+
find the worker with the fewest number of volumes present, and is therefore most useful when combined with other
112+
placement strategies by [Chaining Placement Strategies](#chaining-placement-strategies).
113+
114+
`max-active-volumes-per-worker` can be set to be an integer of 1 or more, in which case a worker will not execute more
115+
than that amount of **volumes**. If unset (or set to a value of 0), the `limit-active-volumes` strategy has no effect -
116+
if this is your only placement strategy, workers will be chosen at random.
117+
118+
```properties
119+
CONCOURSE_CONTAINER_PLACEMENT_STRATEGY=limit-active-volumes
120+
CONCOURSE_MAX_ACTIVE_VOLUMES_PER_WORKER=200
121+
```
122+
123+
## Chaining Placement Strategies
124+
125+
Container placement strategies can be chained together to apply multiple strategies in sequence. The first strategy in
126+
the chain receives the entire set of workers, filtering the set down in some way, and passing that new set of workers to
127+
the next strategy in the chain. If the last strategy in the chain returns multiple workers, one will be chosen at
128+
random.
129+
130+
For instance, consider the following configuration:
131+
132+
```properties
133+
CONCOURSE_CONTAINER_PLACEMENT_STRATEGY=limit-active-containers,limit-active-volumes,volume-locality,fewest-build-containers
134+
CONCOURSE_MAX_ACTIVE_CONTAINERS_PER_WORKER=200
135+
CONCOURSE_MAX_ACTIVE_VOLUMES_PER_WORKER=100
136+
```
137+
138+
This defines a chain of 4 placement strategies, plus the implicit `random` strategy. Let's look at what each strategy
139+
accomplishes:
140+
141+
1. [`limit-active-containers` strategy](#the-limit-active-containers-strategy) removes all workers that already have
142+
more than 200 containers
143+
2. [`limit-active-volumes` strategy](#the-limit-active-volumes-strategy) removes all remaining workers that already have
144+
more than 100 volumes
145+
3. [`volume-locality` strategy](#the-volume-locality-strategy) keeps only the remaining worker(s) that have the most
146+
inputs locally. This can keep more than one worker in the case of a tie
147+
4. [`fewest-build-containers` strategy](#the-fewest-build-containers-strategy) will attempt to break ties by selecting
148+
the worker with fewer build containers. If all the remaining workers have the exact same number of containers, one
149+
will be selected at random

‎docs/docs/operation/creds/credhub.md‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,34 @@ title: The CredHub credential manager
44

55
## Configuration
66

7-
## Credential Lookup Rules
7+
The ATC is statically configured with a CredHub server URL with TLS and client config.
8+
9+
For example, to point the ATC at an internal CredHub server with TLS signed by a local CA, using client id and secret,
10+
you may configure:
11+
12+
```shell
13+
concourse web ... \
14+
--credhub-url https://10.2.0.3:9000 \
15+
--credhub-ca-cert /etc/my-ca.cert \
16+
--credhub-client-id =db02de05-fa39-4855-059b-67221c5c2f63 \
17+
--credhub-client-secret 6a174c20-f6de-a53c-74d2-6018fcceff64
18+
```
19+
20+
## Credential Lookup Rules
21+
22+
When resolving a parameter such as `((foo_param))`, it will look in the following paths, in order:
23+
24+
* `/concourse/TEAM_NAME/PIPELINE_NAME/foo_param`
25+
* `/concourse/TEAM_NAME/foo_param`
26+
27+
The leading `/concourse` can be changed by specifying `--credhub-path-prefix`.
28+
29+
CredHub credentials actually have different types, which may contain multiple values. For example, the `user` type
30+
specifies both `username` and `password.` You can specify the field to grab via `.` syntax,
31+
e.g. `((foo_param.username))`.
32+
33+
If the action is being run in the context of a pipeline (e.g. a `check` or a step in a build of a job), the ATC will
34+
first look in the pipeline path. If it's not found there, it will look in the team path. This allows credentials to be
35+
scoped widely if they're common across many pipelines.
36+
37+
If an action is being run in a one-off build, the ATC will only look in the team path.

0 commit comments

Comments
 (0)