This repository has been archived by the owner on Sep 30, 2020. It is now read-only.
Initial plugins proposal #751
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
the
cncf-cla: yes
Codecov Report
@@ Coverage Diff @@
## master #751 +/- ##
=======================================
Coverage 34.28% 34.28%
=======================================
Files 55 55
Lines 3077 3077
=======================================
Hits 1055 1055
Misses 1863 1863
Partials 159 159Continue to review full report at Codecov.
|
mumoshu
reviewed
Jul 10, 2017
| ## Plugin Locations | ||
|
|
||
| * Built-in plugins can either be part of the kube-aws binary or downloadable from a secure source | ||
| if/when the user enables that plugin. The second is recommended. |
There was a problem hiding this comment.
I consider that built-in plugins should be part of the kube-aws binary hence "built-in",
while calling downloadable plugins contrib plugins.
There was a problem hiding this comment.
I was also considering that:
- contrib plugins can be downloaded from
kubernetes-incubator/kube-aws/contrib/plugins/<PLUGIN_NAME>to<USER_DIRECTORY>/plugins/<PLUGIN_NAME> - The user can also install or create an adhoc plugin under
<USER_DIRECTORY>/plugins/<PLUGIN_NAME>.
There was a problem hiding this comment.
I was considering built-in to mean they can be configured out of the box a bit easier but I think it's fine if they are part of the same package, I don't much of a preference either way for initial implementation.
I like your idea about contrib plugins going to the same place as the adhoc plugins.
There was a problem hiding this comment.
@c-knowles Thanks for the confirmation 👍
I'm even considering that a built-in plugin, when it is enabled, should be extracted from the kube-aws binary and then rendered into the working tree like adhoc plugins, so that it can be customized freely by the user without rebuilding the whole kube-aws binary.
What do you think?
cc @redbaron
There was a problem hiding this comment.
I was considering built-in to mean they can be configured out of the box a bit easier
Btw I agree on this 👍
mumoshu
reviewed
Jul 10, 2017
| * Some plugins will be provided by kube-aws and some will be specific to a user | ||
| * Both will use the same mechanisms to hook into kube-aws/deploy etc | ||
| * Both types of plugins will be enabled and configured in the same way | ||
| * Built-in plugins versus contrib plugins |
There was a problem hiding this comment.
Please let me also add that contrib plugins will be hosted remotely in the kube-aws repo which can be installed locally and then work similarly to adhoc plugins.
Related: https://github.com/kubernetes-incubator/kube-aws/pull/751/files#r126319557
There was a problem hiding this comment.
Agreed, we can add that.
mumoshu
reviewed
Jul 10, 2017
| * Take the CloudFormation and file(s), upload the files to S3 and deploy the template(s) to CloudFormation. This is an opportunity for plugins to perform waiting tasks and/or actions that need to be done on deploy/install, for example post to an endpoint created during deploy. | ||
| * Input - output of `render` stage, stack name | ||
| * Output - stack details such as stack name, endpoints, certs | ||
| 1. validate |
There was a problem hiding this comment.
How about naming this validatePostConditions as it takes the output of deploy stage as its input?
There was a problem hiding this comment.
The motivation for suffixing it with "post" is that at first glance I misunderstood that validate corresponds to kube-aws validate which validates pre-conditions before running kube-aws up and kube-aws update.
There was a problem hiding this comment.
Happy to rename. I agree we should avoid name clashes in cases we expose this to the users, in this case it will be exposed directly in our operator documentation so worth picking something more descriptive. How about postDeployValidate? Or simply postDeploy is an option as well if we wish to allow plugins to perform other "post" actions.
There was a problem hiding this comment.
Thanks, I like your idea of postDeployValidate 👍
|
@c-knowles I'm still reconsidering/updating my implementation proposal of the plugin system at https://gist.github.com/mumoshu/3fa94cee7be23f5a8765c10dd6a91861#file-example-plugin-yaml. |
|
@mumoshu if you think that's a good in road to get started I'm happy to roll with that and see where we get to. |
redbaron
reviewed
Jul 13, 2017
|
|
||
| 1. initialise | ||
| * Perform any prerequisite setup required such as calls to external system. Configuration, presence of files etc can be validated. | ||
| * Input - configuration |
There was a problem hiding this comment.
how plugins communicate with core? will it be YAML? JSON? binary protocols? which communication channel is used? pipes? socket? passed file descriptors?
|
Good point!
I have been scratching my head about that but I'm now considering grpc+yaml
by using hashicorp/go-plugin as a plugin system over grpc while marshalling
complex params as yaml.
I don't like to implement a yet another plugin system of my own, at least.
I considered golang's native plugin but it does't work on macOS until go
1.9.
WDYT?
2017年7月13日(木) 20:10 Maxim Ivanov <notifications@github.com>:
… ***@***.**** commented on this pull request.
------------------------------
In proposals/plugins.md
<#751 (comment)>
:
> + * Some plugins will be provided by kube-aws and some will be specific to a user
+ * Both will use the same mechanisms to hook into kube-aws/deploy etc
+ * Both types of plugins will be enabled and configured in the same way
+* Built-in plugins versus contrib plugins
+ * kube-aws will accept contrib plugins for features considered to be applicable to many users
+ * Contrib plugins will be enabled in a similar way but specifically marked as not fully
+ battle tested; they will still be configured in the same way
+* Plugins will be allowed to act across the whole cluster or act on a specific set of node pools, this should form part of the base configuration for which plugins to run
+
+## Stages
+
+The workflow for kube-aws should consist of the following stages, each of which will be called on the plugins in turn. Each stage will complete across all plugins prior to proceeding to the next stage. All plugins will receive input based on the whole setup, not just their own. This will allow them to hook into parameters from other plugins or core logic. The stages are not intended to become kube-aws commands and should mostly be transparent to a kube-aws user.
+
+1. initialise
+ * Perform any prerequisite setup required such as calls to external system. Configuration, presence of files etc can be validated.
+ * Input - configuration
how plugins communicate with core? will it be YAML? JSON? binary
protocols? which communication channel is used? pipes? socket? passed file
descriptors?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#751 (review)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABV-SHvg93LBmqgjNiSxiFlIp-Ut8y-ks5sNfs9gaJpZM4OSBem>
.
|
|
My vote goes for the hashicorp plugin because it's available now, although I've never used it before so I don't actually mind picking another. I'd like to switch to native go plugins when it's available. Grpc seems a natural fit as well. |
|
@c-knowles Thanks for your efforts on this! #791 should provide the first impl of this. |
mumoshu
added a commit
that referenced
this pull request
Aug 22, 2017
This is an initial implementation of the plugin system #509 as proposed in #751. Not all but most of knobs mentioned in the proposal except pre/post-cluster-creation validations are implemented. Basically, it allows the user to define a set of customizations to various aspects of resources created and managed by kube-aws as a "kube-aws plugin" and reuse it. The set of customizations is defined in a separate file other than a `cluster.yaml` for reusability. More concretely, provide `<your project root>/plugins/<your-plugin-name>/plugin.yaml` like seen in test/integration/plugin_test.go to extend a kube-aws cluster from many aspects including: - additional iam policy statements per node role(worker/controller/etcd) - additional cfn stack template resources per stack(root/control-plane/node-pool) - additional systemd units/custom files per node role(worker/controller/etcd) - additional kubelet feature gates for worker kubelets - additional node labels for worker/controller kubelets and so on. The new plugin system is not used to implement core features of kube-aws yet. Therefore, I believe we don't need to worry much about breaking things via this change. At least one core feature implemented as a plugin is planned in the next version of kube-aws v0.9.9, as noted in our roadmap. Changes: * Plugin System: Add support for node labels * Plugin System: Add support for feature gates * plugin-system: Add support for k8s manifests and helm releases * plugin-system: Add support for kube-apiserver server options * plugin-system: Add support for custom files * plugin-system: Add support for custom IAM policy statements * Rename plugin/api to plugin/pluginapi to better differentiate what the api is for * Move the test helper for plugin to a seperate go file * Extract a type representing the file uploaded to a kube-aws node into a separate go file * plugin-system: Seperate logics from api * plugin-system: Separate cluster extensions by plugins from cluster and plugins * plugin-system: More separation of api and logic * plugin-system: Move apply-kube-aws-plugins script for easier merging with master * plugin-system: Rename pluginapi to pluginmodel * plugin-system: Remove unused types and files * plugin-system: Comment about `values` in plugin.yaml * Reliability improvement to cloud-config-controller * Fix occasional kube-node-label, cfn-signal errors * Fix install-kube-system and apply-kube-aws-plugins services to better scheduled in order without spamming journal
kylehodgetts
pushed a commit
to HotelsDotCom/kube-aws
that referenced
this pull request
Mar 27, 2018
…gins Initial plugins proposal
kylehodgetts
pushed a commit
to HotelsDotCom/kube-aws
that referenced
this pull request
Mar 27, 2018
This is an initial implementation of the plugin system kubernetes-retired#509 as proposed in kubernetes-retired#751. Not all but most of knobs mentioned in the proposal except pre/post-cluster-creation validations are implemented. Basically, it allows the user to define a set of customizations to various aspects of resources created and managed by kube-aws as a "kube-aws plugin" and reuse it. The set of customizations is defined in a separate file other than a `cluster.yaml` for reusability. More concretely, provide `<your project root>/plugins/<your-plugin-name>/plugin.yaml` like seen in test/integration/plugin_test.go to extend a kube-aws cluster from many aspects including: - additional iam policy statements per node role(worker/controller/etcd) - additional cfn stack template resources per stack(root/control-plane/node-pool) - additional systemd units/custom files per node role(worker/controller/etcd) - additional kubelet feature gates for worker kubelets - additional node labels for worker/controller kubelets and so on. The new plugin system is not used to implement core features of kube-aws yet. Therefore, I believe we don't need to worry much about breaking things via this change. At least one core feature implemented as a plugin is planned in the next version of kube-aws v0.9.9, as noted in our roadmap. Changes: * Plugin System: Add support for node labels * Plugin System: Add support for feature gates * plugin-system: Add support for k8s manifests and helm releases * plugin-system: Add support for kube-apiserver server options * plugin-system: Add support for custom files * plugin-system: Add support for custom IAM policy statements * Rename plugin/api to plugin/pluginapi to better differentiate what the api is for * Move the test helper for plugin to a seperate go file * Extract a type representing the file uploaded to a kube-aws node into a separate go file * plugin-system: Seperate logics from api * plugin-system: Separate cluster extensions by plugins from cluster and plugins * plugin-system: More separation of api and logic * plugin-system: Move apply-kube-aws-plugins script for easier merging with master * plugin-system: Rename pluginapi to pluginmodel * plugin-system: Remove unused types and files * plugin-system: Comment about `values` in plugin.yaml * Reliability improvement to cloud-config-controller * Fix occasional kube-node-label, cfn-signal errors * Fix install-kube-system and apply-kube-aws-plugins services to better scheduled in order without spamming journal
mumoshu
added a commit
to mumoshu/kube-aws
that referenced
this pull request
Nov 9, 2018
This is the successor of kubernetes-retired#1473. I'll keep that one for recording purposes. Resolves kubernetes-retired#1469 ## What do we get from this? - Better, documented kube-aws plugin system. Use builtin plugins or write your own with a single file called `plugin.yaml`, to customize every aspect of kube-aws clusters. - aws-iam-authenticator for kubelet auth, as a kube-aws plugin - (near future) EKS support kubernetes-retired#1434 powered by the plugin system ## Story Take this as a continuation of kubernetes-retired#509 (comment) Towards the EKS integration(kubernetes-retired#1434), I have been working on the support for aws-iam-authenticator for kubelet authentication. And I was very annoyed by that I was forced to add small snippets of bash, cloud-config and go into several diverse locations of kube-aws's code-base, plus adding the feature to somehow obtain and transfer the `aws-iam-authenticator` "binary" onto kube-aws worker and controller nodes. To summarize, things that I had to consider were: - Creating a single keypair whose cert has two purposes - CA and Server Authn, encrypting it inside the `credentials/` dir, transferring to worker and controller nodes - Somehow downloading and installing `aws-iam-authenticator` binary onto nodes - Installing the aws-iam-authenticator deployment onto the k8s cluster - Creating a configmap for aws-iam-authenticator that refers to IAM roles that are managed by kube-aws(Templating the configmap with the results of cloudformation functions? I'm tired of writing a long `{{ "Fn::Join": ["", ["long", "bash", "script", "separated and enclosed within double-quotes", "per", "line"]]}}` thing. Being pretty annoyed, recalling kubernetes-retired#751 kubernetes-retired#791 kubernetes-retired#509, I decided to enhance the secret feature of kube-aws - "plugins" - to cover the use-case. It is resulting in a major refactoring of kube-aws. It doesn't affect existing features of kube-aws at all. But it does affect public members exposed by several kube-aws go packages, and how golang packages are organized. So to whom using kube-aws a golang library, this refactoring may affect you. Please feel free to send any feedback on that. # Notables user-visible changes ## Improved plugin system Create a `plugin.yaml` at `PROJECT_ROOT/plugins/NAME/plugin.yaml` and edit according to your needs. [The aws-iam-authenticator plugin](https://github.com/mumoshu/kube-aws/blob/aws-iam-node-auth/builtin/files/plugins/aws-iam-authenticator/plugin.yaml) would be the most informative source for learning how it write it. It can be used for: ### Existing features - Installing Kubernetes manifests - Installing files selected from the `plugins/NAME` directory onto nodes - Injecting custom CloudFormation resources into kube-aws managed stacks ### New features #### Automatically turning yaml templates to cfn expressions Imagine you have a YAML template file that looks like the below in your plugin's directory: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | {{ range $i, $role := .Config.IAMRoleARNs }}, - rolearn: {{$role}} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes {{ end }} ``` Firstly this is rendered with golang's text/template: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | - rolearn: {"Fn::GetAtt": ["IAMRoleController", "Arn"]} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes ``` And this is where kube-was becomes fancy. It detects `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` that looks like a CFN stack template expression, and converts it to: ```json { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} ``` This is then injected into the `AWS::CloudFormation::Init` so that `cfn-init` can rendered the file with the `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` replaced with the actual ARN of the IAM role managed by kube-aws. ```json "Metadata" : { "AWS::CloudFormation::Init" : { "configSets: { "path-to-file": { "files": { "/path/to/file": { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} } } } } ``` #### Automatically downloading files from source URLS ```yaml spec: cluster: machine: roles: controller: files: - path: "/opt/bin/heptio-authenticator-aws" permissions: 0755 type: binary source: path: "files/aws-iam-auth/opt/bin/heptio-authenticator-aws" url: "https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.3.0/heptio-authenticator-aws_0.3.0_linux_amd64" ``` #### Defining additional keypairs used by k8s apps that the plugin supports: ```yaml spec: cluster: pki: keypairs: - name: aws-iam-authenticator dnsNames: - localhost ipAddresses: - 0.0.0.0 - 127.0.0.1 duration: 8760h usages: - ca - server ``` For example, the above plugin.yaml results in `kube-aws render credential` creatning `plugins/aws-iam-authenticator/credentials/aws-iam-authenticator{-key,}.pem` files, that are then installed on nodes for use by the aws-iam-authenticator pods. ## `kube-aws render stack` It now renders a set of files under the `plugins/aws-iam-authenticator` diretory: - `plugin.yaml` - `manifests/*.yaml` for the daemonset and the configmap - `files/*.yaml` for kubeconfig files for the webhook authentication and kubelets ## `kube-aws render stack` It now renders a set of credentials(keypairs) as defined in `plugins/NAME/plugin.yaml` files. # Notable internal changes ## Single place to store embedded files kube-aws after this work will have a single place to store files embedded into kube-aws binaries. Here's how it looks https://github.com/mumoshu/kube-aws/tree/aws-iam-node-auth/builtin/files. ``` $ tree builtin/files builtin/files ├── cluster.yaml.tmpl ├── credentials ├── etcdadm │ ├── Makefile │ ├── README.md │ ├── etcdadm │ └── test ├── kubeconfig.tmpl ├── manifests ├── plugins │ └── aws-iam-authenticator │ ├── files │ │ ├── authentication-token-webhook-config.yaml │ │ ├── controller-kubeconfig.yaml │ │ └── worker-kubeconfig.yaml │ ├── manifests │ │ ├── aws-auth-cm.yaml │ │ └── daemonset.yaml │ └── plugin.yaml ├── stack-templates │ ├── control-plane.json.tmpl │ ├── etcd.json.tmpl │ ├── network.json.tmpl │ ├── node-pool.json.tmpl │ └── root.json.tmpl └── userdata ├── cloud-config-controller ├── cloud-config-etcd └── cloud-config-worker 9 directories, 20 files ``` # Changelog since 1473 The changes between this and kubernetes-retired#1473 are as follows - just reviving accidentally removed files: ``` $ git diff aws-iam-node-auth diff --git a/.gitignore b/.gitignore index 62e8dc26..b33d23d0 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,8 @@ kube-aws coverage.txt profile.out test-result.json +pkg/model/cache +builtin/a_builtin-packr.go # gitbook docs _book diff --git a/hack/relnote b/hack/relnote new file mode 100755 index 00000000..f01e9fd6 --- /dev/null +++ b/hack/relnote @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +go get golang.org/x/oauth2 +go get golang.org/x/net/context +go get github.com/google/go-github/github + +VERSION=$(hack/version) go run hack/relnote.go diff --git a/hack/version b/hack/version new file mode 100755 index 00000000..0f4c6780 --- /dev/null +++ b/hack/version @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +COMMIT=$(git rev-parse HEAD) +TAG=$(git describe --exact-match --abbrev=0 --tags "${COMMIT}" 2> /dev/null || true) + +echo "${TAG}" diff --git a/kube-aws-bot-git-ssh-key.enc b/kube-aws-bot-git-ssh-key.enc new file mode 100644 index 00000000..fef826ba ```
mumoshu
added a commit
that referenced
this pull request
Nov 12, 2018
* update vendor * refactoring kube-aws / experimental IAM-based kubelet auth This is the successor of #1473. I'll keep that one for recording purposes. Resolves #1469 ## What do we get from this? - Better, documented kube-aws plugin system. Use builtin plugins or write your own with a single file called `plugin.yaml`, to customize every aspect of kube-aws clusters. - aws-iam-authenticator for kubelet auth, as a kube-aws plugin - (near future) EKS support #1434 powered by the plugin system ## Story Take this as a continuation of #509 (comment) Towards the EKS integration(#1434), I have been working on the support for aws-iam-authenticator for kubelet authentication. And I was very annoyed by that I was forced to add small snippets of bash, cloud-config and go into several diverse locations of kube-aws's code-base, plus adding the feature to somehow obtain and transfer the `aws-iam-authenticator` "binary" onto kube-aws worker and controller nodes. To summarize, things that I had to consider were: - Creating a single keypair whose cert has two purposes - CA and Server Authn, encrypting it inside the `credentials/` dir, transferring to worker and controller nodes - Somehow downloading and installing `aws-iam-authenticator` binary onto nodes - Installing the aws-iam-authenticator deployment onto the k8s cluster - Creating a configmap for aws-iam-authenticator that refers to IAM roles that are managed by kube-aws(Templating the configmap with the results of cloudformation functions? I'm tired of writing a long `{{ "Fn::Join": ["", ["long", "bash", "script", "separated and enclosed within double-quotes", "per", "line"]]}}` thing. Being pretty annoyed, recalling #751 #791 #509, I decided to enhance the secret feature of kube-aws - "plugins" - to cover the use-case. It is resulting in a major refactoring of kube-aws. It doesn't affect existing features of kube-aws at all. But it does affect public members exposed by several kube-aws go packages, and how golang packages are organized. So to whom using kube-aws a golang library, this refactoring may affect you. Please feel free to send any feedback on that. # Notables user-visible changes ## Improved plugin system Create a `plugin.yaml` at `PROJECT_ROOT/plugins/NAME/plugin.yaml` and edit according to your needs. [The aws-iam-authenticator plugin](https://github.com/mumoshu/kube-aws/blob/aws-iam-node-auth/builtin/files/plugins/aws-iam-authenticator/plugin.yaml) would be the most informative source for learning how it write it. It can be used for: ### Existing features - Installing Kubernetes manifests - Installing files selected from the `plugins/NAME` directory onto nodes - Injecting custom CloudFormation resources into kube-aws managed stacks ### New features #### Automatically turning yaml templates to cfn expressions Imagine you have a YAML template file that looks like the below in your plugin's directory: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | {{ range $i, $role := .Config.IAMRoleARNs }}, - rolearn: {{$role}} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes {{ end }} ``` Firstly this is rendered with golang's text/template: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | - rolearn: {"Fn::GetAtt": ["IAMRoleController", "Arn"]} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes ``` And this is where kube-was becomes fancy. It detects `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` that looks like a CFN stack template expression, and converts it to: ```json { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} ``` This is then injected into the `AWS::CloudFormation::Init` so that `cfn-init` can rendered the file with the `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` replaced with the actual ARN of the IAM role managed by kube-aws. ```json "Metadata" : { "AWS::CloudFormation::Init" : { "configSets: { "path-to-file": { "files": { "/path/to/file": { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} } } } } ``` #### Automatically downloading files from source URLS ```yaml spec: cluster: machine: roles: controller: files: - path: "/opt/bin/heptio-authenticator-aws" permissions: 0755 type: binary source: path: "files/aws-iam-auth/opt/bin/heptio-authenticator-aws" url: "https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.3.0/heptio-authenticator-aws_0.3.0_linux_amd64" ``` #### Defining additional keypairs used by k8s apps that the plugin supports: ```yaml spec: cluster: pki: keypairs: - name: aws-iam-authenticator dnsNames: - localhost ipAddresses: - 0.0.0.0 - 127.0.0.1 duration: 8760h usages: - ca - server ``` For example, the above plugin.yaml results in `kube-aws render credential` creatning `plugins/aws-iam-authenticator/credentials/aws-iam-authenticator{-key,}.pem` files, that are then installed on nodes for use by the aws-iam-authenticator pods. ## `kube-aws render stack` It now renders a set of files under the `plugins/aws-iam-authenticator` diretory: - `plugin.yaml` - `manifests/*.yaml` for the daemonset and the configmap - `files/*.yaml` for kubeconfig files for the webhook authentication and kubelets ## `kube-aws render stack` It now renders a set of credentials(keypairs) as defined in `plugins/NAME/plugin.yaml` files. # Notable internal changes ## Single place to store embedded files kube-aws after this work will have a single place to store files embedded into kube-aws binaries. Here's how it looks https://github.com/mumoshu/kube-aws/tree/aws-iam-node-auth/builtin/files. ``` $ tree builtin/files builtin/files ├── cluster.yaml.tmpl ├── credentials ├── etcdadm │ ├── Makefile │ ├── README.md │ ├── etcdadm │ └── test ├── kubeconfig.tmpl ├── manifests ├── plugins │ └── aws-iam-authenticator │ ├── files │ │ ├── authentication-token-webhook-config.yaml │ │ ├── controller-kubeconfig.yaml │ │ └── worker-kubeconfig.yaml │ ├── manifests │ │ ├── aws-auth-cm.yaml │ │ └── daemonset.yaml │ └── plugin.yaml ├── stack-templates │ ├── control-plane.json.tmpl │ ├── etcd.json.tmpl │ ├── network.json.tmpl │ ├── node-pool.json.tmpl │ └── root.json.tmpl └── userdata ├── cloud-config-controller ├── cloud-config-etcd └── cloud-config-worker 9 directories, 20 files ``` # Changelog since 1473 The changes between this and #1473 are as follows - just reviving accidentally removed files: ``` $ git diff aws-iam-node-auth diff --git a/.gitignore b/.gitignore index 62e8dc26..b33d23d0 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,8 @@ kube-aws coverage.txt profile.out test-result.json +pkg/model/cache +builtin/a_builtin-packr.go # gitbook docs _book diff --git a/hack/relnote b/hack/relnote new file mode 100755 index 00000000..f01e9fd6 --- /dev/null +++ b/hack/relnote @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +go get golang.org/x/oauth2 +go get golang.org/x/net/context +go get github.com/google/go-github/github + +VERSION=$(hack/version) go run hack/relnote.go diff --git a/hack/version b/hack/version new file mode 100755 index 00000000..0f4c6780 --- /dev/null +++ b/hack/version @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +COMMIT=$(git rev-parse HEAD) +TAG=$(git describe --exact-match --abbrev=0 --tags "${COMMIT}" 2> /dev/null || true) + +echo "${TAG}" diff --git a/kube-aws-bot-git-ssh-key.enc b/kube-aws-bot-git-ssh-key.enc new file mode 100644 index 00000000..fef826ba ```
kevtaylor
pushed a commit
to HotelsDotCom/kube-aws
that referenced
this pull request
Jan 9, 2019
…s-retired#1490) * update vendor * refactoring kube-aws / experimental IAM-based kubelet auth This is the successor of kubernetes-retired#1473. I'll keep that one for recording purposes. Resolves kubernetes-retired#1469 ## What do we get from this? - Better, documented kube-aws plugin system. Use builtin plugins or write your own with a single file called `plugin.yaml`, to customize every aspect of kube-aws clusters. - aws-iam-authenticator for kubelet auth, as a kube-aws plugin - (near future) EKS support kubernetes-retired#1434 powered by the plugin system ## Story Take this as a continuation of kubernetes-retired#509 (comment) Towards the EKS integration(kubernetes-retired#1434), I have been working on the support for aws-iam-authenticator for kubelet authentication. And I was very annoyed by that I was forced to add small snippets of bash, cloud-config and go into several diverse locations of kube-aws's code-base, plus adding the feature to somehow obtain and transfer the `aws-iam-authenticator` "binary" onto kube-aws worker and controller nodes. To summarize, things that I had to consider were: - Creating a single keypair whose cert has two purposes - CA and Server Authn, encrypting it inside the `credentials/` dir, transferring to worker and controller nodes - Somehow downloading and installing `aws-iam-authenticator` binary onto nodes - Installing the aws-iam-authenticator deployment onto the k8s cluster - Creating a configmap for aws-iam-authenticator that refers to IAM roles that are managed by kube-aws(Templating the configmap with the results of cloudformation functions? I'm tired of writing a long `{{ "Fn::Join": ["", ["long", "bash", "script", "separated and enclosed within double-quotes", "per", "line"]]}}` thing. Being pretty annoyed, recalling kubernetes-retired#751 kubernetes-retired#791 kubernetes-retired#509, I decided to enhance the secret feature of kube-aws - "plugins" - to cover the use-case. It is resulting in a major refactoring of kube-aws. It doesn't affect existing features of kube-aws at all. But it does affect public members exposed by several kube-aws go packages, and how golang packages are organized. So to whom using kube-aws a golang library, this refactoring may affect you. Please feel free to send any feedback on that. # Notables user-visible changes ## Improved plugin system Create a `plugin.yaml` at `PROJECT_ROOT/plugins/NAME/plugin.yaml` and edit according to your needs. [The aws-iam-authenticator plugin](https://github.com/mumoshu/kube-aws/blob/aws-iam-node-auth/builtin/files/plugins/aws-iam-authenticator/plugin.yaml) would be the most informative source for learning how it write it. It can be used for: ### Existing features - Installing Kubernetes manifests - Installing files selected from the `plugins/NAME` directory onto nodes - Injecting custom CloudFormation resources into kube-aws managed stacks ### New features #### Automatically turning yaml templates to cfn expressions Imagine you have a YAML template file that looks like the below in your plugin's directory: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | {{ range $i, $role := .Config.IAMRoleARNs }}, - rolearn: {{$role}} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes {{ end }} ``` Firstly this is rendered with golang's text/template: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: aws-auth namespace: kube-system data: mapRoles: | - rolearn: {"Fn::GetAtt": ["IAMRoleController", "Arn"]} username: system:node:{{`{{EC2PrivateDNSName}}`}} groups: - system:bootstrappers - system:nodes ``` And this is where kube-was becomes fancy. It detects `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` that looks like a CFN stack template expression, and converts it to: ```json { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} ``` This is then injected into the `AWS::CloudFormation::Init` so that `cfn-init` can rendered the file with the `{"Fn::GetAtt": ["IAMRoleController", "Arn"]}` replaced with the actual ARN of the IAM role managed by kube-aws. ```json "Metadata" : { "AWS::CloudFormation::Init" : { "configSets: { "path-to-file": { "files": { "/path/to/file": { "Fn::Join": ["", [ "apiVersion: v1\n", "kind: ConfigMap\n", "metadata:\n", " name: aws-auth\n", " namespace: kube-system\n", "data:\n", " mapRoles: |\n", " - rolearn: ", {"Fn::GetAtt": ["IAMRoleController", "Arn"]}, "\n", " username: system:node:{{`{{EC2PrivateDNSName}}`}}\n", " groups:\n", " - system:bootstrappers\n", " - system:nodes\n", ]]} } } } } ``` #### Automatically downloading files from source URLS ```yaml spec: cluster: machine: roles: controller: files: - path: "/opt/bin/heptio-authenticator-aws" permissions: 0755 type: binary source: path: "files/aws-iam-auth/opt/bin/heptio-authenticator-aws" url: "https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.3.0/heptio-authenticator-aws_0.3.0_linux_amd64" ``` #### Defining additional keypairs used by k8s apps that the plugin supports: ```yaml spec: cluster: pki: keypairs: - name: aws-iam-authenticator dnsNames: - localhost ipAddresses: - 0.0.0.0 - 127.0.0.1 duration: 8760h usages: - ca - server ``` For example, the above plugin.yaml results in `kube-aws render credential` creatning `plugins/aws-iam-authenticator/credentials/aws-iam-authenticator{-key,}.pem` files, that are then installed on nodes for use by the aws-iam-authenticator pods. ## `kube-aws render stack` It now renders a set of files under the `plugins/aws-iam-authenticator` diretory: - `plugin.yaml` - `manifests/*.yaml` for the daemonset and the configmap - `files/*.yaml` for kubeconfig files for the webhook authentication and kubelets ## `kube-aws render stack` It now renders a set of credentials(keypairs) as defined in `plugins/NAME/plugin.yaml` files. # Notable internal changes ## Single place to store embedded files kube-aws after this work will have a single place to store files embedded into kube-aws binaries. Here's how it looks https://github.com/mumoshu/kube-aws/tree/aws-iam-node-auth/builtin/files. ``` $ tree builtin/files builtin/files ├── cluster.yaml.tmpl ├── credentials ├── etcdadm │ ├── Makefile │ ├── README.md │ ├── etcdadm │ └── test ├── kubeconfig.tmpl ├── manifests ├── plugins │ └── aws-iam-authenticator │ ├── files │ │ ├── authentication-token-webhook-config.yaml │ │ ├── controller-kubeconfig.yaml │ │ └── worker-kubeconfig.yaml │ ├── manifests │ │ ├── aws-auth-cm.yaml │ │ └── daemonset.yaml │ └── plugin.yaml ├── stack-templates │ ├── control-plane.json.tmpl │ ├── etcd.json.tmpl │ ├── network.json.tmpl │ ├── node-pool.json.tmpl │ └── root.json.tmpl └── userdata ├── cloud-config-controller ├── cloud-config-etcd └── cloud-config-worker 9 directories, 20 files ``` # Changelog since 1473 The changes between this and kubernetes-retired#1473 are as follows - just reviving accidentally removed files: ``` $ git diff aws-iam-node-auth diff --git a/.gitignore b/.gitignore index 62e8dc26..b33d23d0 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,8 @@ kube-aws coverage.txt profile.out test-result.json +pkg/model/cache +builtin/a_builtin-packr.go # gitbook docs _book diff --git a/hack/relnote b/hack/relnote new file mode 100755 index 00000000..f01e9fd6 --- /dev/null +++ b/hack/relnote @@ -0,0 +1,7 @@ +#!/usr/bin/env bash + +go get golang.org/x/oauth2 +go get golang.org/x/net/context +go get github.com/google/go-github/github + +VERSION=$(hack/version) go run hack/relnote.go diff --git a/hack/version b/hack/version new file mode 100755 index 00000000..0f4c6780 --- /dev/null +++ b/hack/version @@ -0,0 +1,6 @@ +#!/usr/bin/env bash + +COMMIT=$(git rev-parse HEAD) +TAG=$(git describe --exact-match --abbrev=0 --tags "${COMMIT}" 2> /dev/null || true) + +echo "${TAG}" diff --git a/kube-aws-bot-git-ssh-key.enc b/kube-aws-bot-git-ssh-key.enc new file mode 100644 index 00000000..fef826ba ```
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For #509