Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operator STS #1414

Merged
merged 54 commits into from
Mar 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
1c35713
operator-sts
pjuarezd Jan 26, 2023
e3e53bd
Merge branch 'master' into operator-sts-api
pjuarezd Feb 3, 2023
10c39ae
Merge branch 'master' into operator-sts-api
pjuarezd Feb 8, 2023
c2409bb
fix conflicts
pjuarezd Feb 8, 2023
b0dd714
WIP operator tls example client
pjuarezd Feb 10, 2023
95e4da3
Merge branch 'master' into operator-sts-api
pjuarezd Feb 22, 2023
f70779e
bundle Policies
pjuarezd Feb 24, 2023
4a6846d
example clients WIP
pjuarezd Feb 24, 2023
9cfbfa3
example operator sts clients checkpoint
pjuarezd Feb 26, 2023
92bd682
example clients and tests checkpoint
pjuarezd Feb 27, 2023
1b0fd70
shellcheck linter
pjuarezd Feb 27, 2023
ae3a68b
one step closer
pjuarezd Feb 27, 2023
3c0e68d
Merge branch 'master' into operator-sts-api
pjuarezd Feb 27, 2023
416b458
fix typo
pjuarezd Feb 28, 2023
d48309d
Merge branch 'master' into operator-sts-api
pjuarezd Feb 28, 2023
c0f614c
madmin-go v2
pjuarezd Feb 28, 2023
7a8dca9
Openshift test - install Minio Operator (#1435)
pjuarezd Feb 28, 2023
ee3e13f
Helm chart: Add priorityClass support for operator (#1423)
varet80 Feb 28, 2023
c5f719d
Update versions and fix KES Test (#1457)
cniackz Feb 28, 2023
34ec627
Trust openshift signer (#1461)
cniackz Feb 28, 2023
73365ca
Docs on TLS for Operator trusting other CAs (#1473)
dvaldivia Feb 28, 2023
32fbbcf
applyconfiguration generated objects (#1470)
pjuarezd Feb 28, 2023
c3ea2ee
Wording changes to PR#1473 (#1474)
allanrogerr Feb 28, 2023
d866d5f
Fix tests workflow
pjuarezd Mar 1, 2023
497b0c8
Fix shellcheck
pjuarezd Mar 1, 2023
ce46cd6
fix workflow file
pjuarezd Mar 1, 2023
c0e5021
Almost done with STS tests
pjuarezd Mar 1, 2023
829168c
unit tests 🎉
pjuarezd Mar 1, 2023
0304dfa
Merge branch 'master' into operator-sts-api
pjuarezd Mar 2, 2023
93c71eb
Fix make sure console image is using the local operator build
pjuarezd Mar 3, 2023
840b118
Merge branch 'master' into operator-sts-api
pjuarezd Mar 3, 2023
daaf249
Tweaks to STS examples (#1)
dvaldivia Mar 3, 2023
dfbf977
fix tests
pjuarezd Mar 4, 2023
7b5c9bf
fix
pjuarezd Mar 4, 2023
cc59a63
lang
pjuarezd Mar 4, 2023
f48b66c
Improve STS documentation
dvaldivia Mar 4, 2023
beb77b8
no slash in make target
pjuarezd Mar 4, 2023
7f22501
miniodev registry
pjuarezd Mar 4, 2023
631ca1e
Improve docs
dvaldivia Mar 4, 2023
cdfa74b
Some spelling/grammar changes
allanrogerr Mar 4, 2023
cd35c7c
Update crd.adoc references
pjuarezd Mar 4, 2023
842042b
rename sts api to v1alpha1
pjuarezd Mar 4, 2023
73356f7
Fix unit tests
pjuarezd Mar 4, 2023
939ba2d
fix: job name breaking the unit tests
pjuarezd Mar 4, 2023
3124380
Refer CRD doc in example
pjuarezd Mar 4, 2023
8d671aa
dotnet client example
pjuarezd Mar 5, 2023
72d32e4
job names to match pattern
pjuarezd Mar 5, 2023
50c47aa
Removing unneeded files:
pjuarezd Mar 5, 2023
6a22294
remove reference to cert-manager test
pjuarezd Mar 5, 2023
62d4b26
Addressing error handling feedback
pjuarezd Mar 6, 2023
baa7d5b
Address feedback on workgroups and concurrency
pjuarezd Mar 6, 2023
117cfab
fail fast tests if `make` or `kind load docker-image` fails
pjuarezd Mar 6, 2023
a690df9
remove policybinding queue, not being used for now
pjuarezd Mar 6, 2023
a465929
Address spelling feedback
pjuarezd Mar 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion .github/workflows/kubernetes-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -240,4 +240,25 @@ jobs:
- name: Tenant KES
run: |
"${GITHUB_WORKSPACE}/testing/console-tenant+kes.sh"


test-policy-binding:
runs-on: ${{ matrix.os }}
needs:
- logsearch
- operator
- lint
- gotest
- getdeps
- govet
strategy:
matrix:
go-version: [ 1.20.x ]
os: [ ubuntu-latest ]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go-version }}
- name: Test PolicyBinding CRD and sts call on kind
run: |
"${GITHUB_WORKSPACE}/testing/test-policy-binding.sh"
2 changes: 1 addition & 1 deletion .github/workflows/shellcheck.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- name: Run ShellCheck
uses: ludeeus/action-shellcheck@master
env:
SHELLCHECK_OPTS: -e SC2046 -e SC1091 -e SC2086 -e SC2090 -e SC2089 -e SC2006
SHELLCHECK_OPTS: -e SC2046 -e SC1091 -e SC2086 -e SC2090 -e SC2089 -e SC2006 -e SC2206
- name: Run nancy vulnerability report
run: |
nancy_version=$(curl --retry 10 -Ls -o /dev/null -w "%{url_effective}" https://github.com/sonatype-nexus-community/nancy/releases/latest | sed "s/https:\/\/github.com\/sonatype-nexus-community\/nancy\/releases\/tag\///")
Expand Down
4 changes: 3 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,12 @@ regen-crd:
@go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.11.1
@${GOPATH}/bin/controller-gen crd:maxDescLen=0,generateEmbeddedObjectMeta=true paths="./..." output:crd:artifacts:config=$(KUSTOMIZE_CRDS)
@sed 's#namespace: minio-operator#namespace: {{ .Release.Namespace }}#g' resources/base/crds/minio.min.io_tenants.yaml > $(HELM_TEMPLATES)/minio.min.io_tenants.yaml
@sed 's#namespace: minio-operator#namespace: {{ .Release.Namespace }}#g' resources/base/crds/sts.min.io_policybindings.yaml > $(HELM_TEMPLATES)/sts.min.io_policybindings.yaml

regen-crd-docs:
@which crd-ref-docs 1>/dev/null || (echo "Installing crd-ref-docs" && GO111MODULE=on go install -v github.com/elastic/crd-ref-docs@latest)
@${GOPATH}/bin/crd-ref-docs --source-path=./pkg/apis/minio.min.io/v2 --config=docs/templates/config.yaml --renderer=asciidoctor --output-path=docs/crd.adoc --templates-dir=docs/templates/asciidoctor/
@${GOPATH}/bin/crd-ref-docs --source-path=./pkg/apis/minio.min.io/v2 --config=docs/templates/config.yaml --renderer=asciidoctor --output-path=docs/tenant_crd.adoc --templates-dir=docs/templates/asciidoctor/
@${GOPATH}/bin/crd-ref-docs --source-path=./pkg/apis/sts.min.io/v1alpha1 --config=docs/templates/config.yaml --renderer=asciidoctor --output-path=docs/policybinding_crd.adoc --templates-dir=docs/templates/asciidoctor/

plugin: regen-crd
@echo "Building 'kubectl-minio' binary"
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,4 +318,5 @@ Use of MinIO Operator is governed by the GNU AGPLv3 or later, found in the [LICE
- [Apply PodSecurityPolicy](https://github.com/minio/operator/blob/master/docs/pod-security-policy.md).
- [Deploy MinIO Tenant with Console](https://github.com/minio/operator/blob/master/docs/console.md).
- [Deploy MinIO Tenant with KES](https://github.com/minio/operator/blob/master/docs/kes.md).
- [Tenant API Documentation](docs/crd.adoc)
- [Tenant API Documentation](docs/tenant_crd.adoc)
- [Policy Binding API Documentation](docs/policybinding_crd.adoc)
51 changes: 51 additions & 0 deletions docs/STS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# MinIO Operator STS

**Native IAM Authentication for Kubernetes.**

> ⚠️ This feature is an alpha release and is subject to breaking changes in future releases.

MinIO Operator offers support
for [Secure Tokens](https://min.io/docs/minio/linux/developers/security-token-service.html?ref=op-gh) (a.k.a. STS) which
are a
form of temporary access credentials for your
MinIO Tenant. In essence, this allows you to control access to your MinIO tenant from your applications without having to
explicitly create credentials for each application; in fact no credentials need to be created at all.

# Authorization Flow

For an application to gain access into a MinIO Tenant, a `PolicyBinding` granting explicit access to the application's
[Service Account](https://kubernetes.io/docs/concepts/security/service-accounts/). Authorization is then validated
using the following steps:

1. AssumeRoleWithWebIdentity call
2. Verify ServiceAccount JWT
3. JWT Validation Result
4. Validate SA has PolicyBinding in the Tenant namespace
5. Get PolicyBinding
6. AssumeRole for application
7. Obtain Temporary Credentials
8. Return Temporary Credentials To App
9. Consume Object Storage

![STS Diagram](images/sts-diagram.png)

# Requirements

## Enabling STS functionality

At the moment, the STS feature ships `off` by default. To turn it on, switch `OPERATOR_STS_ENABLED` to `on` in
the `minio-operator` deployment.

## TLS

The STS functionality works only with TLS configured. We can request certificates automatically, but additionally you can
use `cert-manager` or bring your own certificates.

## SDK support

Your application must use an SDK that supports `AssumeRole` like behavior.

# Examples

We have provided example usage in the [examples/kustomization/sts-example](../examples/kustomization/sts-example)
folder.
2 changes: 1 addition & 1 deletion docs/console.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ kubectl create -f https://raw.githubusercontent.com/minio/operator/master/exampl
kubectl minio tenant create --name tenant1 --secret tenant1-secret --servers 4 --volumes 16 --capacity 16Ti --namespace tenant1-ns --console-secret console-secret
```

A complete list of values is available [here](crd.adoc##consoleconfiguration) in the API reference.
A complete list of values is available [here](tenant_crd.adoc##consoleconfiguration) in the API reference.
Binary file added docs/images/sts-diagram.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/kes.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ KES Configuration is a part of Tenant yaml file. Check the sample file [availabl
| spec.kes.kesSecret | Secret to specify KES Configuration. This is a mandatory field. |
| spec.kes.metadata | This allows a way to map metadata to the KES pods. Internally `metadata` is a struct type as [explained here](https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta). |

A complete list of values is available [here](crd.adoc#kesconfig) in the API reference.
A complete list of values is available [here](tenant_crd.adoc#kesconfig) in the API reference.
2 changes: 1 addition & 1 deletion docs/operator-fields.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,4 @@ MinIO Operator creates native Kubernetes resources within the cluster. If the Te
| spec.pools.volumesPerServer | Set the number of volume mounts per MinIO node. For example if you set `spec.pools[0].Servers = 4`, `spec.pools[1].Servers = 8` and `spec.volumesPerServer = 4`, then you'll have total 12 MinIO Pods, with 4 volume mounts on each Pod. Note that `volumesPerServer` is static per cluster and that expanding a cluster will add new nodes. |
| spec.pools.tolerations | Define a toleration for the Tenant pod to match a taint. Refer [this document](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) for details. |

A complete list of values is available [here](crd.adoc) in the API reference.
A complete list of values is available [here](tenant_crd.adoc) in the API reference.
120 changes: 120 additions & 0 deletions docs/policybinding_crd.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
// Generated documentation. Please do not edit.
:anchor_prefix: k8s-api

[id="{p}-api-reference"]
== API Reference

:minio-image: https://hub.docker.com/r/minio/minio/tags[minio/minio:RELEASE.2023-01-12T02-06-16Z]
:kes-image: https://hub.docker.com/r/minio/kes/tags[minio/kes:v0.18.0]
:prometheus-image: https://quay.io/prometheus/prometheus:latest[prometheus/prometheus:latest]
:logsearch-image: https://hub.docker.com/r/minio/operator/tags[minio/operator:v4.5.8]
:postgres-image: https://github.com/docker-library/postgres[library/postgres]


[id="{anchor_prefix}-sts-min-io-v1alpha1"]
=== sts.min.io/v1alpha1

Package v1alpha1 - The following parameters are specific to the `sts.min.io/v1alpha1` MinIO Policy Binding CRD API
PolicyBinding is an Authorization mechanism managed by the Minio Operator.
Using Kubernetes ServiceAccount JSON Web Tokens the binding allow a ServiceAccount to assume temporary IAM credentials.
For more complete documentation on this object, see the https://docs.min.io/minio/k8s/reference/minio-operator-reference.html#minio-operator-yaml-reference[MinIO Kubernetes Documentation].
PolicyBinding is added as part of the MinIO Operator v5.0.0. +



[id="{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-application"]
==== Application

Application defines the `Namespace` and `ServiceAccount` to authorize the usage of the policies listed

.Appears In:
****
- xref:{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-policybindingspec[$$PolicyBindingSpec$$]
****

[cols="25a,75a", options="header"]
|===
| Field | Description

|*`namespace`* __string__
|*Required* +

|*`serviceaccount`* __string__
|*Required* +

|===


[id="{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-policybinding"]
==== PolicyBinding

PolicyBinding is a https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/[Kubernetes object] describing a MinIO PolicyBinding.

.Appears In:
****
- xref:{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-policybindinglist[$$PolicyBindingList$$]
****

[cols="25a,75a", options="header"]
|===
| Field | Description

|*`metadata`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#objectmeta-v1-meta[$$ObjectMeta$$]__
|Refer to Kubernetes API documentation for fields of `metadata`.


|*`spec`* __xref:{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-policybindingspec[$$PolicyBindingSpec$$]__
|*Required* +
The root field for the MinIO PolicyBinding object.

|===




[id="{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-policybindingspec"]
==== PolicyBindingSpec

PolicyBindingSpec (`spec`) defines the configuration of a MinIO PolicyBinding object. +

.Appears In:
****
- xref:{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-policybinding[$$PolicyBinding$$]
****

[cols="25a,75a", options="header"]
|===
| Field | Description

|*`application`* __xref:{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-application[$$Application$$]__
|*Required* +
The Application Property identifies the namespace and service account that will be authorized

|*`policies`* __string array__
|*Required* +

|===




[id="{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-policybindingusage"]
==== PolicyBindingUsage

PolicyBindingUsage are metrics regarding the usage of the policyBinding

.Appears In:
****
- xref:{anchor_prefix}-github-com-minio-operator-pkg-apis-sts-min-io-v1alpha1-policybindingstatus[$$PolicyBindingStatus$$]
****

[cols="25a,75a", options="header"]
|===
| Field | Description

|*`authotizations`* __integer__
|

|===


2 changes: 1 addition & 1 deletion docs/sidecars.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,4 @@ The following example configures a warp container to run in the same pod as the

**Note:** the MinIO Service for the tenant won't expose the ports added in the sidecar. It's up to the user to expose these ports with their own services.

A complete list of values is available [here](crd.adoc##sidecars) in the API reference.
A complete list of values is available [here](tenant_crd.adoc##sidecars) in the API reference.
Loading