-
Notifications
You must be signed in to change notification settings - Fork 78
/
proxy.md
99 lines (68 loc) · 4.13 KB
/
proxy.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# Proxies
The installer should work behind a proxy.
## Goals
- Proxy installs complete successfully.
- Docker can pull images from remote and local registries.
## Non Goals
- Validate kotsadm add-on works with a proxy.
- Support docker versions below 19.03.
## Background
The kURL spec has a field for an http proxy address, but the feature has not been implemented.
## High-Level Design
Use the proxy when downloading packages from S3.
Configure docker to pull from remote registries with the proxy and the local registry without the proxy.
Add `HTTP_PROXY` and `NO_PROXY` environment variables to the kotsadm add-on.
## Detailed Design
### Spec
Proxy configuration uses three fields under the kurl section of the installer spec:
```yaml
apiVersion: cluster.kurl.sh/v1beta1
kind: Installer
metadata:
name: proxy
spec:
kurl:
proxyAddress: http://10.128.0.3:3128
additionalNoProxyAddresses:
- 10.128.0.4
- 10.128.0.5
- 10.128.0.6
- registry.internal
noProxy: false
```
If `noProxy` is set to `true` then the other proxy fields in the spec are ignored and the installer does not attempt to do any proxy configuration.
This field already exists on the kurl spec.
The `proxyAddress` field is the URL of a proxy.
This field already exists on the kurl spec.
The installer will validate the proxy URL at runtime by making a proxied request to `https://api.replicated.com/market/v1/echo/ip` and will bail if the request fails.
The `additionalNoProxyAddresses` field is new.
This field is ignored if `proxyAddress` is unset.
It accepts a list of IPs and hostnames.
Cluster administrators should add all node IPs to this field.
IP addresses may be in CIDR notation.
The default set of no proxy addresses is the private IP of the current machine, the pod CIDR, and the service CIDR.
Any other addresses specified in this field will be added to the default set to construct the NO_PROXY environment variable.
The join script must accept the flag `additional-no-proxy-addresses` so that the service and pod CIDRs can be included in Docker's proxy configuration on the remote node.
The default set of no-proxy addresses includes the private address of the node, the load balancer address (if set), localhost, the pod CIDR, the service CIDR, and the namespaces of any enabled add-ons.
The `.svc` and `.local` search domains are also included in the no-proxy list to support all forms of resolvable in-cluster domains, such as `kubernetes.default.svc` and `kubernetes.default.svc.cluster.local`.
### Docker add-on
If docker is enabled the installer will create the file /etc/systemd/system/docker.service.d/http-proxy.conf.
The environment variables `HTTP_PROXY` and `NO_PROXY` will be set in this file.
On subsequent runs the installer will check if any changes are required in this file and restart docker only if needed.
If `docker.preserveConfig` is set to true in the spec then this file will never be created or modified and docker will not be restarted.
The join.sh and upgrade.sh scripts will apply the same configuration to remote workers or masters.
After making a change to proxy configuration in the spec, cluster adminstrators can re-run the upgrade.sh script on remote nodes to reconfigure docker.
The online docker install will use the docker airgap packages stored on S3.
Currently the online docker install script is fetched from `get.replicated.com` and then makes requests to fetch updates from yum or apt repositories.
These requests to repositories have errors or fail when behind a proxy.
### Kotsadm add-on
If a proxy is configured then the installer will add the environment variables `HTTP_PROXY` and `NO_PROXY` to the kotsadm and kotsadm-api deployment.
The kotsadm-api deployment also requires the object store cluster IP to download logs from velero backups since the Node client does not support CIDR notation.
## Alternatives Considered
### Prompt for a proxy if none is specified.
This would make automation harder.
### Support older versions of Docker.
Docker 18.09 does not support CIDR notation in the NO_PROXY env var.
It's possible to add support for 18.09 by adding the registry cluster IP to docker's configuration.
## Security Considerations
None