This repository has been archived by the owner on Sep 30, 2020. It is now read-only.
refactoring kube-aws / experimental IAM-based kubelet auth #1490
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is the successor of kubernetes-retired#1473. I'll keep that one for recording purposes. Resolves kubernetes-retired#1469 ## What do we get from this? - Better, documented kube-aws plugin system. Use builtin plugins or write your own with a single file called `plugin.yaml`, to customize every aspect of kube-aws clusters. - aws-iam-authenticator for kubelet auth, as a kube-aws plugin - (near future) EKS support kubernetes-retired#1434 powered by the plugin system ## Story Take this as a continuation of kubernetes-retired#509 (comment) Towards the EKS integration(kubernetes-retired#1434), I have been working on the support for aws-iam-authenticator for kubelet authentication. And I was very annoyed by that I was forced to add small snippets of bash, cloud-config and go into several diverse locations of kube-aws's code-base, plus adding the feature to somehow obtain and transfer the `aws-iam-authenticator` "binary" onto kube-aws worker and controller nodes. To summarize, things that I had to consider were: - Creating a single keypair whose cert has two purposes - CA and Server Authn, encrypting it inside the `credentials/` dir, transferring to worker and controller nodes - Somehow downloading and installing `aws-iam-authenticator` binary onto nodes - Installing the aws-iam-authenticator deployment onto the k8s cluster - Creating a configmap for aws-iam-authenticator that refers to IAM roles that are managed by kube-aws(Templating the configmap with the results of cloudformation functions? I'm tired of writing a long `{{ "Fn::Join": ["", ["long", "bash", "script", "separated and enclosed within double-quotes", "per", "line"]]}}` thing. Being pretty annoyed, recalling kubernetes-retired#751 kubernetes-retired#791 kubernetes-retired#509, I decided to enhance the secret feature of kube-aws - "plugins" - to cover the use-case. It is resulting in a major refactoring of kube-aws. It doesn't affect existing features of kube-aws at all. But it does affect public members exposed by several kube-aws go packages, and how golang packages are organized. So to whom using kube-aws a golang library, this refactoring may affect you. Please feel free to send any feedback on that. # Notables user-visible changes ## Improved plugin system Create a `plugin.yaml` at `PROJECT_ROOT/plugins/NAME/plugin.yaml` and edit according to your needs. [The aws-iam-authenticator plugin](https://github.com/mumoshu/kube-aws/blob/aws-iam-node-auth/builtin/files/plugins/aws-iam-authenticator/plugin.yaml) would be the most informative source for learning how it write it. It can be used for: ### Existing features - Installing Kubernetes manifests - Installing files selected from the `plugins/NAME` directory onto nodes - Injecting custom CloudFormation resources into kube-aws managed stacks ### New features #### Automatically turning yaml templates to cfn expressions Imagine you have a YAML template file that looks like the below in your plugin's directory: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | {{ range $i, $role := .Config.IAMRoleARNs }}, - rolearn: {{$role}} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes {{ end }} ``` Firstly this is rendered with golang's text/template: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | - rolearn: {"Fn::GetAtt": ["IAMRoleController", "Arn"]} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes ``` And this is where kube-was becomes fancy. It detects `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` that looks like a CFN stack template expression, and converts it to: ```json { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} ``` This is then injected into the `AWS::CloudFormation::Init` so that `cfn-init` can rendered the file with the `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` replaced with the actual ARN of the IAM role managed by kube-aws. ```json "Metadata" : { "AWS::CloudFormation::Init" : { "configSets: { "path-to-file": { "files": { "/path/to/file": { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} } } } } ``` #### Automatically downloading files from source URLS ```yaml spec: cluster: machine: roles: controller: files: - path: "/opt/bin/heptio-authenticator-aws" permissions: 0755 type: binary source: path: "files/aws-iam-auth/opt/bin/heptio-authenticator-aws" url: "https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.3.0/heptio-authenticator-aws_0.3.0_linux_amd64" ``` #### Defining additional keypairs used by k8s apps that the plugin supports: ```yaml spec: cluster: pki: keypairs: - name: aws-iam-authenticator dnsNames: - localhost ipAddresses: - 0.0.0.0 - 127.0.0.1 duration: 8760h usages: - ca - server ``` For example, the above plugin.yaml results in `kube-aws render credential` creatning `plugins/aws-iam-authenticator/credentials/aws-iam-authenticator{-key,}.pem` files, that are then installed on nodes for use by the aws-iam-authenticator pods. ## `kube-aws render stack` It now renders a set of files under the `plugins/aws-iam-authenticator` diretory: - `plugin.yaml` - `manifests/*.yaml` for the daemonset and the configmap - `files/*.yaml` for kubeconfig files for the webhook authentication and kubelets ## `kube-aws render stack` It now renders a set of credentials(keypairs) as defined in `plugins/NAME/plugin.yaml` files. # Notable internal changes ## Single place to store embedded files kube-aws after this work will have a single place to store files embedded into kube-aws binaries. Here's how it looks https://github.com/mumoshu/kube-aws/tree/aws-iam-node-auth/builtin/files. ``` $ tree builtin/files builtin/files ├── cluster.yaml.tmpl ├── credentials ├── etcdadm │ ├── Makefile │ ├── README.md │ ├── etcdadm │ └── test ├── kubeconfig.tmpl ├── manifests ├── plugins │ └── aws-iam-authenticator │ ├── files │ │ ├── authentication-token-webhook-config.yaml │ │ ├── controller-kubeconfig.yaml │ │ └── worker-kubeconfig.yaml │ ├── manifests │ │ ├── aws-auth-cm.yaml │ │ └── daemonset.yaml │ └── plugin.yaml ├── stack-templates │ ├── control-plane.json.tmpl │ ├── etcd.json.tmpl │ ├── network.json.tmpl │ ├── node-pool.json.tmpl │ └── root.json.tmpl └── userdata ├── cloud-config-controller ├── cloud-config-etcd └── cloud-config-worker 9 directories, 20 files ``` # Changelog since 1473 The changes between this and kubernetes-retired#1473 are as follows - just reviving accidentally removed files: ``` $ git diff aws-iam-node-auth diff --git a/.gitignore b/.gitignore index 62e8dc26..b33d23d0 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,8 @@ kube-aws coverage.txt profile.out test-result.json +pkg/model/cache +builtin/a_builtin-packr.go # gitbook docs _book diff --git a/hack/relnote b/hack/relnote new file mode 100755 index 00000000..f01e9fd6 --- /dev/null +++ b/hack/relnote @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +go get golang.org/x/oauth2 +go get golang.org/x/net/context +go get github.com/google/go-github/github + +VERSION=$(hack/version) go run hack/relnote.go diff --git a/hack/version b/hack/version new file mode 100755 index 00000000..0f4c6780 --- /dev/null +++ b/hack/version @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +COMMIT=$(git rev-parse HEAD) +TAG=$(git describe --exact-match --abbrev=0 --tags "${COMMIT}" 2> /dev/null || true) + +echo "${TAG}" diff --git a/kube-aws-bot-git-ssh-key.enc b/kube-aws-bot-git-ssh-key.enc new file mode 100644 index 00000000..fef826ba ```
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: If they are not already assigned, you can assign the PR to them by writing The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
size/XXL
mumoshu
changed the title
Codecov Report
@@ Coverage Diff @@
## master #1490 +/- ##
===========================================
- Coverage 37.86% 25.38% -12.49%
===========================================
Files 75 97 +22
Lines 4608 4987 +379
===========================================
- Hits 1745 1266 -479
- Misses 2620 3575 +955
+ Partials 243 146 -97
Continue to review full report at Codecov.
|
|
Let's go ahead 💨 |
8 tasks
|
Any objections to take what you've written in the PR and place it under docs folder? |
|
No objection! |
|
I created a new ticket #1506 which may relate to this work. |
kevtaylor
pushed a commit
to HotelsDotCom/kube-aws
that referenced
this pull request
Jan 9, 2019
…s-retired#1490) * update vendor * refactoring kube-aws / experimental IAM-based kubelet auth This is the successor of kubernetes-retired#1473. I'll keep that one for recording purposes. Resolves kubernetes-retired#1469 ## What do we get from this? - Better, documented kube-aws plugin system. Use builtin plugins or write your own with a single file called `plugin.yaml`, to customize every aspect of kube-aws clusters. - aws-iam-authenticator for kubelet auth, as a kube-aws plugin - (near future) EKS support kubernetes-retired#1434 powered by the plugin system ## Story Take this as a continuation of kubernetes-retired#509 (comment) Towards the EKS integration(kubernetes-retired#1434), I have been working on the support for aws-iam-authenticator for kubelet authentication. And I was very annoyed by that I was forced to add small snippets of bash, cloud-config and go into several diverse locations of kube-aws's code-base, plus adding the feature to somehow obtain and transfer the `aws-iam-authenticator` "binary" onto kube-aws worker and controller nodes. To summarize, things that I had to consider were: - Creating a single keypair whose cert has two purposes - CA and Server Authn, encrypting it inside the `credentials/` dir, transferring to worker and controller nodes - Somehow downloading and installing `aws-iam-authenticator` binary onto nodes - Installing the aws-iam-authenticator deployment onto the k8s cluster - Creating a configmap for aws-iam-authenticator that refers to IAM roles that are managed by kube-aws(Templating the configmap with the results of cloudformation functions? I'm tired of writing a long `{{ "Fn::Join": ["", ["long", "bash", "script", "separated and enclosed within double-quotes", "per", "line"]]}}` thing. Being pretty annoyed, recalling kubernetes-retired#751 kubernetes-retired#791 kubernetes-retired#509, I decided to enhance the secret feature of kube-aws - "plugins" - to cover the use-case. It is resulting in a major refactoring of kube-aws. It doesn't affect existing features of kube-aws at all. But it does affect public members exposed by several kube-aws go packages, and how golang packages are organized. So to whom using kube-aws a golang library, this refactoring may affect you. Please feel free to send any feedback on that. # Notables user-visible changes ## Improved plugin system Create a `plugin.yaml` at `PROJECT_ROOT/plugins/NAME/plugin.yaml` and edit according to your needs. [The aws-iam-authenticator plugin](https://github.com/mumoshu/kube-aws/blob/aws-iam-node-auth/builtin/files/plugins/aws-iam-authenticator/plugin.yaml) would be the most informative source for learning how it write it. It can be used for: ### Existing features - Installing Kubernetes manifests - Installing files selected from the `plugins/NAME` directory onto nodes - Injecting custom CloudFormation resources into kube-aws managed stacks ### New features #### Automatically turning yaml templates to cfn expressions Imagine you have a YAML template file that looks like the below in your plugin's directory: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | {{ range $i, $role := .Config.IAMRoleARNs }}, - rolearn: {{$role}} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes {{ end }} ``` Firstly this is rendered with golang's text/template: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | - rolearn: {"Fn::GetAtt": ["IAMRoleController", "Arn"]} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes ``` And this is where kube-was becomes fancy. It detects `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` that looks like a CFN stack template expression, and converts it to: ```json { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} ``` This is then injected into the `AWS::CloudFormation::Init` so that `cfn-init` can rendered the file with the `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` replaced with the actual ARN of the IAM role managed by kube-aws. ```json "Metadata" : { "AWS::CloudFormation::Init" : { "configSets: { "path-to-file": { "files": { "/path/to/file": { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} } } } } ``` #### Automatically downloading files from source URLS ```yaml spec: cluster: machine: roles: controller: files: - path: "/opt/bin/heptio-authenticator-aws" permissions: 0755 type: binary source: path: "files/aws-iam-auth/opt/bin/heptio-authenticator-aws" url: "https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.3.0/heptio-authenticator-aws_0.3.0_linux_amd64" ``` #### Defining additional keypairs used by k8s apps that the plugin supports: ```yaml spec: cluster: pki: keypairs: - name: aws-iam-authenticator dnsNames: - localhost ipAddresses: - 0.0.0.0 - 127.0.0.1 duration: 8760h usages: - ca - server ``` For example, the above plugin.yaml results in `kube-aws render credential` creatning `plugins/aws-iam-authenticator/credentials/aws-iam-authenticator{-key,}.pem` files, that are then installed on nodes for use by the aws-iam-authenticator pods. ## `kube-aws render stack` It now renders a set of files under the `plugins/aws-iam-authenticator` diretory: - `plugin.yaml` - `manifests/*.yaml` for the daemonset and the configmap - `files/*.yaml` for kubeconfig files for the webhook authentication and kubelets ## `kube-aws render stack` It now renders a set of credentials(keypairs) as defined in `plugins/NAME/plugin.yaml` files. # Notable internal changes ## Single place to store embedded files kube-aws after this work will have a single place to store files embedded into kube-aws binaries. Here's how it looks https://github.com/mumoshu/kube-aws/tree/aws-iam-node-auth/builtin/files. ``` $ tree builtin/files builtin/files ├── cluster.yaml.tmpl ├── credentials ├── etcdadm │ ├── Makefile │ ├── README.md │ ├── etcdadm │ └── test ├── kubeconfig.tmpl ├── manifests ├── plugins │ └── aws-iam-authenticator │ ├── files │ │ ├── authentication-token-webhook-config.yaml │ │ ├── controller-kubeconfig.yaml │ │ └── worker-kubeconfig.yaml │ ├── manifests │ │ ├── aws-auth-cm.yaml │ │ └── daemonset.yaml │ └── plugin.yaml ├── stack-templates │ ├── control-plane.json.tmpl │ ├── etcd.json.tmpl │ ├── network.json.tmpl │ ├── node-pool.json.tmpl │ └── root.json.tmpl └── userdata ├── cloud-config-controller ├── cloud-config-etcd └── cloud-config-worker 9 directories, 20 files ``` # Changelog since 1473 The changes between this and kubernetes-retired#1473 are as follows - just reviving accidentally removed files: ``` $ git diff aws-iam-node-auth diff --git a/.gitignore b/.gitignore index 62e8dc26..b33d23d0 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,8 @@ kube-aws coverage.txt profile.out test-result.json +pkg/model/cache +builtin/a_builtin-packr.go # gitbook docs _book diff --git a/hack/relnote b/hack/relnote new file mode 100755 index 00000000..f01e9fd6 --- /dev/null +++ b/hack/relnote @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +go get golang.org/x/oauth2 +go get golang.org/x/net/context +go get github.com/google/go-github/github + +VERSION=$(hack/version) go run hack/relnote.go diff --git a/hack/version b/hack/version new file mode 100755 index 00000000..0f4c6780 --- /dev/null +++ b/hack/version @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +COMMIT=$(git rev-parse HEAD) +TAG=$(git describe --exact-match --abbrev=0 --tags "${COMMIT}" 2> /dev/null || true) + +echo "${TAG}" diff --git a/kube-aws-bot-git-ssh-key.enc b/kube-aws-bot-git-ssh-key.enc new file mode 100644 index 00000000..fef826ba ```
Merged
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the successor of kubernetes-incubator#1473. I'll keep that one for recording purposes.
Resolves kubernetes-incubator#1469
What do we get from this?
plugin.yaml, to customize every aspect of kube-aws clusters.Story
Take this as a continuation of kubernetes-incubator#509 (comment)
Towards the EKS integration(kubernetes-incubator#1434), I have been working on the support for aws-iam-authenticator for kubelet authentication. And I was very annoyed by that I was forced to add small snippets of bash, cloud-config and go into several diverse locations of kube-aws's code-base, plus adding the feature to somehow obtain and transfer the
aws-iam-authenticator"binary" onto kube-aws worker and controller nodes.To summarize, things that I had to consider were:
credentials/dir, transferring to worker and controller nodesaws-iam-authenticatorbinary onto nodes{{ "Fn::Join": ["", ["long", "bash", "script", "separated and enclosed within double-quotes", "per", "line"]]}}thing.Being pretty annoyed, recalling kubernetes-incubator#751 kubernetes-incubator#791 kubernetes-incubator#509, I decided to enhance the secret feature of kube-aws - "plugins" - to cover the use-case. It is resulting in a major refactoring of kube-aws.
It doesn't affect existing features of kube-aws at all. But it does affect public members exposed by several kube-aws go packages, and how golang packages are organized.
So to whom using kube-aws a golang library, this refactoring may affect you. Please feel free to send any feedback on that.
Notables user-visible changes
Improved plugin system
Create a
plugin.yamlatPROJECT_ROOT/plugins/NAME/plugin.yamland edit according to your needs. The aws-iam-authenticator plugin would be the most informative source for learning how it write it.It can be used for:
Existing features
plugins/NAMEdirectory onto nodesNew features
Automatically turning yaml templates to cfn expressions
Imagine you have a YAML template file that looks like the below in your plugin's directory:
Firstly this is rendered with golang's text/template:
And this is where kube-was becomes fancy. It detects
{"Fn::GetAtt": ["IAMRoleController", "Arn"]}that looks like a CFN stack template expression, and converts it to:{ "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]}This is then injected into the
AWS::CloudFormation::Initso thatcfn-initcan rendered the file with the{"Fn::GetAtt": ["IAMRoleController", "Arn"]}replaced with the actual ARN of the IAM role managed by kube-aws.Automatically downloading files from source URLS
Defining additional keypairs used by k8s apps that the plugin supports:
For example, the above plugin.yaml results in
kube-aws render credentialcreatningplugins/aws-iam-authenticator/credentials/aws-iam-authenticator{-key,}.pemfiles, that are then installed on nodes for use by the aws-iam-authenticator pods.kube-aws render stackIt now renders a set of files under the
plugins/aws-iam-authenticatordiretory:plugin.yamlmanifests/*.yamlfor the daemonset and the configmapfiles/*.yamlfor kubeconfig files for the webhook authentication and kubeletskube-aws render stackIt now renders a set of credentials(keypairs) as defined in
plugins/NAME/plugin.yamlfiles.Notable internal changes
Single place to store embedded files
kube-aws after this work will have a single place to store files embedded into kube-aws binaries. Here's how it looks https://github.com/mumoshu/kube-aws/tree/aws-iam-node-auth/builtin/files.
Changelog since 1473
The changes between this and kubernetes-incubator#1473 are as follows - just reviving accidentally removed files: