Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Manifest] Apply kustomize best practices to standalone manifest #3978

Merged
merged 6 commits into from
Jun 16, 2020

Conversation

Bobgy
Copy link
Contributor

@Bobgy Bobgy commented Jun 12, 2020

This is part of KFP and KF 1.1 integration.
Kubeflow kustomize best practices doc: https://github.com/kubeflow/manifests/blob/master/docs/KustomizeBestPractices.md

Changes:

  • kustomize var name should be unique to allow composition of kustomize packages, therefore we prefix kustomize vars with kfp- to make sure they don't conflict with kubeflow
  • using kebab case for kustomize var to distinguish it from Env var replacement --- this is in fact my personal preference, what do you think?
  • Use kubernetes native env var from configmap/secret instead of kustomize var replacement if possible
  • Make cluster scoped resources namespace customization easier by using kustomization.yaml's namespace field for customization.

@kubeflow-bot
Copy link

This change is Reviewable

value: $(BUCKET_NAME)
valueFrom:
configMapKeyRef:
name: pipeline-install-config
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw in real deployment, the configmap name is pipeline-install-config-xxx. It has a strange post-fix, does it matter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No , it won't cause any problems. kustomize adds the suffix automatically to both configmap name and references to it. So that it can support configmap rolling update (updating configmap requires all dependent pods to restart to pick up new changes -- therefore, by adding a suffix, when configmap content changes, all deployments referring to it will use a new configmap name, so they will restart those pods with new configmap.)

configMapKeyRef from a configmap will be preserved in deployment yaml in Kubernetes, that's more kubernetes native way of adding config to a deployment.
This is suggested in https://github.com/kubeflow/manifests/blob/master/docs/KustomizeBestPractices.md#command-line-substitution.

# NOTE: var name must be unique globally to allow composition of multiple kustomize
# packages. Therefore, we added prefix `kfp-cluster-scoped-` to distinguish it from
# others.
- name: kfp-cluster-scoped-namespace
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering whether we just use "kfp-" prefix as it can already solve the concern of global conflict with other components. "kfp-cluster-sopced-namespace" would let me think on whether there is another namespace

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, any other suggestions?

I wasn't using kfp- because I want to be able to compose cluster-scoped package together with kfp namespace scoped package.
Configuration for a var can only appear once between both packages.

@rmgogogo
Copy link
Contributor

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rmgogogo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Bobgy
Copy link
Contributor Author

Bobgy commented Jun 16, 2020

Getting one strange test failure

[34mintegration-test-fvmgk-2173363153:     --- PASS: TestUpgrade/TestPrepare (6.46s)�[0m
�[34mintegration-test-fvmgk-2173363153:     --- FAIL: TestUpgrade/TestVerify (0.09s)�[0m
�[34mintegration-test-fvmgk-2173363153:         upgrade_test.go:387: �[0m
�[34mintegration-test-fvmgk-2173363153:             	Error Trace:	upgrade_test.go:387�[0m
�[34mintegration-test-fvmgk-2173363153:             	            				upgrade_test.go:282�[0m
�[34mintegration-test-fvmgk-2173363153:             	            				upgrade_test.go:64�[0m
�[34mintegration-test-fvmgk-2173363153:             	Error:      	Not equal: �[0m
�[34mintegration-test-fvmgk-2173363153:             	            	expected: &run_model.APIRun{CreatedAt:strfmt.DateTime{wall:0x0, ext:63727827823, loc:(*time.Location)(nil)}, Description:"this is hello world", Error:"", FinishedAt:strfmt.DateTime{wall:0x0, ext:62135596800, loc:(*time.Location)(nil)}, ID:"53746595-bb43-4897-a602-c3de31e826f4", Metrics:[]*run_model.APIRunMetric(nil), Name:"hello world", PipelineSpec:(*run_model.APIPipelineSpec)(0xc00062d2c0), ResourceReferences:[]*run_model.APIResourceReference{(*run_model.APIResourceReference)(0xc0001ef980), (*run_model.APIResourceReference)(0xc0001ef9b0)}, ScheduledAt:strfmt.DateTime{wall:0x0, ext:62135596800, loc:(*time.Location)(nil)}, ServiceAccount:"pipeline-runner", Status:"", StorageState:""}�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	actual  : &run_model.APIRun{CreatedAt:strfmt.DateTime{wall:0x0, ext:63727827823, loc:(*time.Location)(nil)}, Description:"this is hello world", Error:"", FinishedAt:strfmt.DateTime{wall:0x0, ext:62135596800, loc:(*time.Location)(nil)}, ID:"53746595-bb43-4897-a602-c3de31e826f4", Metrics:[]*run_model.APIRunMetric(nil), Name:"hello world", PipelineSpec:(*run_model.APIPipelineSpec)(0xc00062c7e0), ResourceReferences:[]*run_model.APIResourceReference{(*run_model.APIResourceReference)(0xc0001eecf0), (*run_model.APIResourceReference)(0xc0001eecc0)}, ScheduledAt:strfmt.DateTime{wall:0x0, ext:62135596800, loc:(*time.Location)(nil)}, ServiceAccount:"pipeline-runner", Status:"", StorageState:""}�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	Diff:�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	--- Expected�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	+++ Actual�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	@@ -18,3 +18,3 @@�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	    Key: (*run_model.APIResourceKey)({�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	-    ID: (string) (len=36) "b4ecbebf-bd3b-4d1d-88fd-99459695b412",�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	+    ID: (string) (len=36) "f0464a41-e673-461d-ba73-71e928b9949b",�[0m
�[34mintegration-test-fvmgk-2173363153:             	            	     Type: (run_model.APIResourceType) (len=10) "EXPERIMENT"�[0m
�[34mintegration-test-fvmgk-2173363153:             	Test:       	TestUpgrade/TestVerify�[0m

/retest

@Bobgy
Copy link
Contributor Author

Bobgy commented Jun 16, 2020

/retest

@Bobgy
Copy link
Contributor Author

Bobgy commented Jun 16, 2020

/assign @jlewi
I've been updating KFP's manifest to remove vars as much as possible.
Maybe you can also review this.

I plan to use this as upstream for kf 1.1 very soon

@k8s-ci-robot k8s-ci-robot merged commit c0124cb into kubeflow:master Jun 16, 2020
@Bobgy Bobgy deleted the kustomize_cleanup branch June 16, 2020 02:58
Bobgy added a commit that referenced this pull request Jun 16, 2020
* Use configMapKeyRef for env vars

* Allow easy customization of cluster-scoped resources namespace

* clean up

* Clean up

* Simplify var replacement with direct configmap value ref

* clean up params.env
RedbackThomson pushed a commit to RedbackThomson/pipelines that referenced this pull request Jun 17, 2020
…eflow#3978)

* Use configMapKeyRef for env vars

* Allow easy customization of cluster-scoped resources namespace

* clean up

* Clean up

* Simplify var replacement with direct configmap value ref

* clean up params.env
Jeffwan pushed a commit to Jeffwan/pipelines that referenced this pull request Dec 9, 2020
…eflow#3978)

* Use configMapKeyRef for env vars

* Allow easy customization of cluster-scoped resources namespace

* clean up

* Clean up

* Simplify var replacement with direct configmap value ref

* clean up params.env
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants