Add Host process examples #161
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,2 @@ | ||
| .idea | ||
| .venv |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # HostProcess examples | ||
|
|
||
| This contains examples of services running as [HostProcess](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/) | ||
| containers. HostProcess is an alpha feature in Kubernetes 1.22. | ||
|
|
||
| The eventual goal is to move many of these examples to the corresponding repositories. | ||
|
|
||
| ## Requirements | ||
|
|
||
| The CNI examples currently require the containerd change in https://github.com/containerd/containerd/pull/5131 and a nightly build of hcsschim | ||
|
|
||
| For convience there is a nightly job in the repository that builds the required components: https://github.com/kubernetes-sigs/sig-windows-tools/releases/tag/windows-containerd-nightly | ||
|
|
||
| > Docker does not support HostProcess containers. These images will not work with Docker. | ||
|
|
||
| ## Future Improvements | ||
|
|
||
| These scripts are based off the initial Docker implementation in https://github.com/kubernetes-sigs/sig-windows-tools/tree/master/kubeadm. | ||
| Idealy these should use init containers (possible written in golang) to install and configure the binaries as is done with Linux. Then the | ||
| main container can run the required components for a given CNI. | ||
|
|
||
| kube-proxy has slightly different configurations (sourcevip as example) across cni's so they are split into separate folder for each. Kubeadm should create and configure kube-proxy for windows appropriately during node initialization. | ||
|
|
||
| ## Building images | ||
|
|
||
| To build all the images update version and images refences in `build.sh` then run it. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| #!/bin/bash | ||
|
|
||
| repository=${repository:-"sigwindowstools"} | ||
| flannelVersion=${flannelVersion:-"v0.14.0"} | ||
| proxyVersion=${proxyVersion:-"v1.22.2"} | ||
| calicoVersion=${calicoVersion:-"v3.20.0"} | ||
|
|
||
| SCRIPTROOT=$(dirname "${BASH_SOURCE[0]}") | ||
| pushd $SCRIPTROOT/flannel | ||
| ./build.sh -r $repository --flannelVersion $flannelVersion | ||
| popd | ||
| pushd $SCRIPTROOT/calico | ||
| ./build.sh -r $repository --calicoVersion $calicoVersion | ||
| popd | ||
|
|
||
| declare -a proxyVersions=("v1.22.2") | ||
|
|
||
| # Read the array values with space | ||
| for proxyVersion in "${proxyVersions[@]}"; do | ||
| pushd $SCRIPTROOT/flannel | ||
| ./build.sh -r $repository --proxyVersion $proxyVersion | ||
| popd | ||
| pushd $SCRIPTROOT/calico | ||
| ./build.sh -r $repository --proxyVersion $proxyVersion | ||
| popd | ||
| done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| #!/bin/bash | ||
|
|
||
| # https://devhints.io/bash | ||
| while [[ "$1" =~ ^- && ! "$1" == "--" ]]; do case $1 in | ||
| -n | --calicoVersion ) | ||
| shift; calicoVersion="$1" | ||
| ;; | ||
| -p | --proxyVersion ) | ||
| shift; proxyVersion="$1" | ||
| ;; | ||
| -r | --repository ) | ||
| shift; repository="$1" | ||
| ;; | ||
| -a | --all ) | ||
| shift; all="1" | ||
| ;; | ||
| esac; shift; done | ||
| if [[ "$1" == '--' ]]; then shift; fi | ||
|
|
||
| repository=${repository:-"sigwindowstools"} | ||
| calicoVersion=${calicoVersion:-"v3.20.0"} | ||
|
|
||
| docker buildx create --name img-builder --use --platform windows/amd64 | ||
| trap 'docker buildx rm img-builder' EXIT | ||
|
|
||
| if [[ -n "$calicoVersion" || "$all" == "1" ]] ; then | ||
| trap 'docker buildx rm img-builder' EXIT | ||
| pushd install | ||
| docker buildx build --platform windows/amd64 --output=type=registry --pull --build-arg=CALICO_VERSION=$calicoVersion -f Dockerfile.install -t $repository/calico-install:$calicoVersion-hostprocess . | ||
| popd | ||
| fi | ||
|
|
||
| if [[ -n "$calicoVersion" || "$all" == "1" ]] ; then | ||
| pushd node | ||
| docker buildx build --platform windows/amd64 --output=type=registry --pull --build-arg=CALICO_VERSION=$calicoVersion -f Dockerfile.node -t $repository/calico-node:$calicoVersion-hostprocess . | ||
| popd | ||
| fi | ||
|
|
||
| if [[ -n "$proxyVersion" || "$all" == "1" ]] ; then | ||
| proxyVersion=${proxyVersion:-"v1.22.2"} | ||
| pushd kube-proxy | ||
| docker buildx build --platform windows/amd64 --output=type=registry --pull --build-arg=k8sVersion=$proxyVersion -f Dockerfile -t $repository/kube-proxy:$proxyVersion-calico-hostprocess . | ||
| popd | ||
| fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,238 @@ | ||
| # strictAffinity required for windows | ||
| apiVersion: crd.projectcalico.org/v1 | ||
| kind: IPAMConfig | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| autoAllocateBlocks: true | ||
| strictAffinity: true | ||
| --- | ||
| kind: ConfigMap | ||
| apiVersion: v1 | ||
| metadata: | ||
| name: calico-config-windows | ||
| namespace: kube-system | ||
| labels: | ||
| tier: node | ||
| app: calico | ||
| data: | ||
| veth_mtu: "1350" | ||
| cni_network_config: | | ||
| { | ||
| "name": "Calico", | ||
| "cniVersion": "0.3.1", | ||
| "plugins": [ | ||
| { | ||
| "windows_use_single_network": true, | ||
| "type": "calico", | ||
| "mode": "vxlan", | ||
| "nodename": "__KUBERNETES_NODE_NAME__", | ||
| "nodename_file_optional": true, | ||
| "log_file_path": "c:/cni.log", | ||
| "log_level": "debug", | ||
|
|
||
| "vxlan_mac_prefix": "0E-2A", | ||
| "vxlan_vni": 4096, | ||
| "mtu": __CNI_MTU__, | ||
| "policy": { | ||
| "type": "k8s" | ||
| }, | ||
|
|
||
| "log_level": "info", | ||
|
|
||
| "capabilities": {"dns": true}, | ||
| "DNS": { | ||
| "Search": [ | ||
| "svc.cluster.local" | ||
| ] | ||
| }, | ||
|
|
||
| "datastore_type": "kubernetes", | ||
|
|
||
| "kubernetes": { | ||
| "kubeconfig": "__KUBECONFIG_FILEPATH__" | ||
| }, | ||
|
|
||
| "ipam": { | ||
| "type": "calico-ipam", | ||
| "subnet": "usePodCidr" | ||
| }, | ||
|
|
||
| "policies": [ | ||
| { | ||
| "Name": "EndpointPolicy", | ||
| "Value": { | ||
| "Type": "OutBoundNAT", | ||
| "ExceptionList": [ | ||
| "__K8S_SERVICE_CIDR__" | ||
| ] | ||
| } | ||
| }, | ||
| { | ||
| "Name": "EndpointPolicy", | ||
| "Value": { | ||
| "Type": "SDNROUTE", | ||
| "DestinationPrefix": "__K8S_SERVICE_CIDR__", | ||
| "NeedEncap": true | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
|
|
||
| } | ||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: DaemonSet | ||
| metadata: | ||
| name: calico-node-windows | ||
| labels: | ||
| tier: node | ||
| app: calico | ||
| namespace: kube-system | ||
| spec: | ||
| selector: | ||
| matchLabels: | ||
| app: calico | ||
| template: | ||
| metadata: | ||
| labels: | ||
| tier: node | ||
| app: calico | ||
| spec: | ||
| affinity: | ||
| nodeAffinity: | ||
| requiredDuringSchedulingIgnoredDuringExecution: | ||
| nodeSelectorTerms: | ||
| - matchExpressions: | ||
| - key: kubernetes.io/os | ||
| operator: In | ||
| values: | ||
| - windows | ||
| - key: kubernetes.io/arch | ||
| operator: In | ||
| values: | ||
| - amd64 | ||
| securityContext: | ||
| windowsOptions: | ||
| hostProcess: true | ||
| runAsUserName: "NT AUTHORITY\\system" | ||
| hostNetwork: true | ||
| serviceAccountName: calico-node | ||
| tolerations: | ||
| - operator: Exists | ||
| effect: NoSchedule | ||
| # Mark the pod as a critical add-on for rescheduling. | ||
| - key: CriticalAddonsOnly | ||
| operator: Exists | ||
| - effect: NoExecute | ||
| operator: Exists | ||
| initContainers: | ||
| # This container installs the CNI binaries | ||
| # and CNI network config file on each node. | ||
| - name: install-cni | ||
| image: sigwindowstools/calico-install:v3.20.0-hostprocess | ||
| args: ["$env:CONTAINER_SANDBOX_MOUNT_POINT/calico/install.ps1"] | ||
| imagePullPolicy: Always | ||
| env: | ||
| # Name of the CNI config file to create. | ||
| - name: CNI_CONF_NAME | ||
| value: "10-calico.conflist" | ||
| # The CNI network config to install on each node. | ||
| - name: CNI_NETWORK_CONFIG | ||
| valueFrom: | ||
| configMapKeyRef: | ||
| name: calico-config-windows | ||
| key: cni_network_config | ||
| # Set the hostname based on the k8s node name. | ||
| - name: KUBERNETES_NODE_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: spec.nodeName | ||
| # CNI MTU Config variable | ||
| - name: CNI_MTU | ||
| valueFrom: | ||
| configMapKeyRef: | ||
| name: calico-config-windows | ||
| key: veth_mtu | ||
| # Prevents the container from sleeping forever. | ||
| - name: SLEEP | ||
| value: "false" | ||
| - name: K8S_SERVICE_CIDR | ||
| value: "10.96.0.0/12" | ||
| volumeMounts: | ||
| - mountPath: /host/opt/cni/bin | ||
| name: cni-bin-dir | ||
| - mountPath: /host/etc/cni/net.d | ||
| name: cni-net-dir | ||
| - name: kubeadm-config | ||
| mountPath: /etc/kubeadm-config/ | ||
| securityContext: | ||
| windowsOptions: | ||
| hostProcess: true | ||
| runAsUserName: "NT AUTHORITY\\system" | ||
| containers: | ||
| - name: calico-node-startup | ||
| image: sigwindowstools/calico-node:v3.20.0-hostprocess | ||
| args: ["$env:CONTAINER_SANDBOX_MOUNT_POINT/calico/node-service.ps1"] | ||
| workingDir: "$env:CONTAINER_SANDBOX_MOUNT_POINT/calico/" | ||
|
There was a problem hiding this comment. same question about what component evaluates/expands this envvar... are envvars normally usable inside workingDir? it looks like the kubelet only expands envvars in command/args, and workingdir is passed as-is to the runtime: There was a problem hiding this comment. These variables are set for processes created inside the container by hcsshim (for both normal and hostProcess containers). Windows will expand and environment variables found in the command line when it tries to runt he process / entry point for the container. There was a problem hiding this comment. The runtime sets and evaluates these ENV variables as part of its path processing. We can't use the approach in https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config) since it is not known until runtime since that value is set by hcsschim. Using the vars, technically shouldn't be necessary for hostprocess but we found some bugs in hcsshim around arg and Working directory path processing. With the approach that we are evaluating now this env variable won't be necessary any longer as the container will be given its own filesystem (thought it still has access to the host filesystem unlike on linux). For reference these are the fixes that are in being worked on until the new approach is validated: cc: @dcantah |
||
| imagePullPolicy: Always | ||
| volumeMounts: | ||
| - name: calico-config-windows | ||
| mountPath: /etc/kube-calico-windows/ | ||
| env: | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| apiVersion: v1 | ||
| fieldPath: metadata.name | ||
| - name: POD_NAMESPACE | ||
| valueFrom: | ||
| fieldRef: | ||
| apiVersion: v1 | ||
| fieldPath: metadata.namespace | ||
| - name: CNI_IPAM_TYPE | ||
| value: "calico-ipam" | ||
| - name: CALICO_NETWORKING_BACKEND | ||
| value: "vxlan" | ||
| - name: KUBECONFIG | ||
| value: "C:/etc/cni/net.d/calico-kubeconfig" | ||
| - name: VXLAN_VNI | ||
| value: "4096" | ||
| - name: calico-node-felix | ||
| image: sigwindowstools/calico-node:v3.20.0-hostprocess | ||
| args: ["$env:CONTAINER_SANDBOX_MOUNT_POINT/calico/felix-service.ps1"] | ||
| imagePullPolicy: Always | ||
| workingDir: "$env:CONTAINER_SANDBOX_MOUNT_POINT/calico/" | ||
| volumeMounts: | ||
| - name: calico-config-windows | ||
| mountPath: /etc/kube-calico-windows/ | ||
| env: | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| apiVersion: v1 | ||
| fieldPath: metadata.name | ||
| - name: POD_NAMESPACE | ||
| valueFrom: | ||
| fieldRef: | ||
| apiVersion: v1 | ||
| fieldPath: metadata.namespace | ||
| - name: VXLAN_VNI | ||
| value: "4096" | ||
| - name: KUBECONFIG | ||
| value: "C:/etc/cni/net.d/calico-kubeconfig" | ||
| volumes: | ||
| - name: calico-config-windows | ||
| configMap: | ||
| name: calico-config-windows | ||
| # Used to install CNI. | ||
| - name: cni-bin-dir | ||
| hostPath: | ||
| path: /opt/cni/bin | ||
| - name: cni-net-dir | ||
| hostPath: | ||
| path: /etc/cni/net.d | ||
| - name: kubeadm-config | ||
| configMap: | ||
| name: kubeadm-config | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # Note this image doesn't really mater for hostprocess | ||
| # the files in the image are copied to $env:CONTAINER_SANDBOX_MOUNT_POINT on the host | ||
| # but the file system is the Host NOT the container | ||
| ARG BASE="mcr.microsoft.com/windows/nanoserver:1809" | ||
|
|
||
| FROM --platform=linux/amd64 curlimages/curl as bins | ||
| ARG CALICO_VERSION="v3.20.0" | ||
|
|
||
| WORKDIR /cni | ||
| RUN curl -Lo cni.zip https://github.com/projectcalico/calico/releases/download/${CALICO_VERSION}/calico-windows-${CALICO_VERSION}.zip | ||
| RUN unzip cni.zip | ||
|
|
||
| FROM $BASE | ||
|
|
||
| ENV PATH="C:\Program Files\PowerShell;C:\utils;C:\Windows\system32;C:\Windows;" | ||
|
|
||
| COPY --from=bins /cni/CalicoWindows/cni/calico.exe /calico/cni/calico.exe | ||
| COPY --from=bins /cni/CalicoWindows/cni/calico-ipam.exe /calico/cni/calico-ipam.exe | ||
| COPY --from=bins /cni/CalicoWindows/calico-kube-config.template /calico/calico-kube-config.template | ||
|
|
||
| COPY calico-install.ps1 /calico/install.ps1 | ||
| ENTRYPOINT ["powershell"] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what interprets this envvar? normally, envvars would be specified in args as
$(CONTAINER_SANDBOX_MOUNT_POINT)/calico/install.ps1(https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config)