sync tidb configmap in controller #1291

aylei · 2019-12-05T14:51:07Z

Signed-off-by: Aylei rayingecho@gmail.com

What problem does this PR solve?

close #1192

What is changed and how does it work?

New controller logic to sync tidb configmap.

Generify the CreateOrUpdate logic and the template rendering code.

This PR introduce a new dependency controller-runtime in favor of its generic client for kubernetes.

Check List

Tests

E2E test

Does this PR introduce a user-facing change?:

The configuration of tidb-server can be managed in the `TidbCluster` resource now.

check the e2e case to get an overview of this PR: https://github.com/pingcap/tidb-operator/pull/1291/files#diff-92579a43c91e473b9716eeeee2068004

@onlymellb @cofyc @tennix @weekface @Yisaer PTAL

aylei · 2019-12-05T14:51:32Z

/run-e2e-test

aylei · 2019-12-05T14:51:35Z

/run-e2e-test

aylei · 2019-12-05T18:31:12Z

/run-e2e-test

aylei · 2019-12-06T06:43:17Z

/run-e2e-test

aylei · 2019-12-06T10:51:32Z

/run-e2e-test

cofyc · 2019-12-06T11:00:31Z

pkg/controller/configmap_control.go

+	cmC.Labels = cmD.Labels
+	for k, v := range cmD.Annotations {
+		cmC.Annotations[k] = v
+	}


why we must keep old annotations?

It is not a must. I preserve the un-overridden annotations deliberately, which allows user and third-party components could safely add their annotations (as long as the annotation key is not taken by tidb-operator) to resources owned by TidbCluster.

cofyc · 2019-12-06T11:08:35Z

pkg/manager/member/tidb_member_manager.go

 	ns := tc.GetNamespace()
 	tcName := tc.GetName()
 	instanceName := tc.GetLabels()[label.InstanceLabelKey]
 	tidbConfigMap := controller.MemberConfigMapName(tc, v1alpha1.TiDBMemberType)
+	if cm != nil {
+		tidbConfigMap = cm.Name
+	}


in which case, cm will be nil?

when .spec.tidb.config is nil

pkg/controller/configmap_control.go

Signed-off-by: Aylei <rayingecho@gmail.com> unified grep command for mac and linux (pingcap#1285) Refactor apply Signed-off-by: Aylei <rayingecho@gmail.com> pdapi: only read secret for PD client cert (on TLS enabled clusters) … (pingcap#1290) * pdapi: only read secret for PD client cert (on TLS enabled clusters) when needed * pdapi: adjust log message * pdapi: fix error in logging Remove DinD related scripts and document (pingcap#1283) * Remove DinD related scripts * Remove local-dind-tutorial doc * drop dind part from CONTRIBUTING guide. Fix configmap ownerreference Signed-off-by: Aylei <rayingecho@gmail.com>

aylei · 2019-12-07T14:49:09Z

/run-e2e-test

aylei · 2019-12-07T15:10:40Z

I'm not quite satisfied with the generalization made in this PR and find that controller-runtime has better implementation for common controller operations, refactoring to use controller-runtime now.

Marked as DNM, sorry for the reviewers @cofyc @onlymellb

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei · 2019-12-08T11:50:44Z

/run-e2e-test

aylei · 2019-12-08T12:52:40Z

pkg/controller/generic_control.go

+
+// GenericControlInterface manages generic object that managed by an arbitrary controller
+type GenericControlInterface interface {
+	CreateOrUpdate(controller, obj runtime.Object, mergeFn MergeFn) (runtime.Object, error)


Apply is ambiguous with the server-side apply of kubernetes, rename to CreateOrUpdate

aylei · 2019-12-08T12:56:35Z

pkg/controller/generic_control.go

+
+		// 5. check if the copy is actually mutated
+		if !apiequality.Semantic.DeepEqual(existing, mutated) {
+			err := c.client.Update(context.TODO(), mutated)


Optionally, we could retry on conflict like other controls do, I choose no retry here because CreateOrUpdate will never meet conflicts if there is no concurrent modification, and if there are concurrent modifications, it is better to return the error to caller instead of aggressive retrying.

aylei · 2019-12-08T13:00:16Z

cmd/controller-manager/main.go

@@ -114,6 +116,11 @@ func main() {
 	if err != nil {
 		glog.Fatalf("failed to get advanced-statefulset Clientset: %v", err)
 	}
+	// TODO: optimize the read of genericCli with the shared cache
+	genericCli, err := client.New(cfg, client.Options{Scheme: scheme.Scheme})


Using the client of controller-runtime without using the informer of controller-runtime has a drawback that all read operations will go to kube-apiserver, it is acceptable now because the only read operation is to get the latest version when perform a CreateOrUpdate.

aylei · 2019-12-08T13:02:51Z

pkg/controller/generic_control.go

+// override. Note that aggressive override usually causes unnecessary updates because the object will be mutated
+// after POST/PUT to api-server (e.g. Defaulting), an annotation based technique could be used to avoid such
+// updating: set a last-applied-config annotation and diff the annotation instead of the real spec.
+type MergeFn func(existing, desired runtime.Object) error


server-side apply maybe a better solution in the future.

ref: https://kubernetes.io/docs/reference/using-api/api-concepts/#server-side-apply

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei · 2019-12-08T13:07:24Z

/run-e2e-test

aylei · 2019-12-09T14:11:20Z

/run-e2e-test

aylei · 2019-12-09T15:00:29Z

/run-e2e-test

aylei · 2019-12-09T16:05:35Z

@weekface @cofyc I've addressed the comments, PTAL again

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei · 2019-12-10T02:48:24Z

/run-e2e-test

aylei · 2019-12-10T02:49:03Z

/run-e2e-test

aylei · 2019-12-10T03:14:42Z

/run-e2e-test

cofyc

LGTM

aylei · 2019-12-10T04:53:40Z

@weekface @tennix Could you please take a look again?

aylei · 2019-12-10T09:06:06Z

/run-e2e-test

tennix

Rest LGTM

tennix · 2019-12-10T14:33:41Z

pkg/controller/configmap_control.go

-	UpdateConfigMap(*v1alpha1.TidbCluster, *corev1.ConfigMap) (*corev1.ConfigMap, error)
-	DeleteConfigMap(*v1alpha1.TidbCluster, *corev1.ConfigMap) error
+	// CreateConfigMap create the given ConfigMap
+	CreateConfigMap(runtime.Object, *corev1.ConfigMap) (*corev1.ConfigMap, error)


Should add a comment for what the runtime.Object is.

tennix · 2019-12-10T14:52:52Z

pkg/manager/member/tidb_member_manager.go

+		if config.Security == nil {
+			config.Security = &v1alpha1.Security{}
+		}
+		config.Security.ClusterSSLCA = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"


It's better to define these 3 paths as constant.

Good catch!

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei · 2019-12-10T15:43:11Z

/run-e2e-test

tennix

LGTM

aylei marked this pull request as ready for review December 6, 2019 10:48

aylei requested review from cofyc, onlymellb, weekface, Yisaer and tennix December 6, 2019 10:51

cofyc reviewed Dec 6, 2019

View reviewed changes

onlymellb reviewed Dec 7, 2019

View reviewed changes

pkg/controller/configmap_control.go Outdated Show resolved Hide resolved

aylei force-pushed the fix-1192 branch from 0706e4e to 5490437 Compare December 7, 2019 14:48

Merge branch 'master' into fix-1192

0ee2743

aylei changed the title ~~sync tidb configmap in controller~~ [DNM] sync tidb configmap in controller Dec 7, 2019

aylei added 2 commits December 8, 2019 19:41

Use generic_control to perform configmap write

9c2b970

Signed-off-by: Aylei <rayingecho@gmail.com>

Merge branch 'master' into fix-1192

a98db9f

aylei changed the title ~~[DNM] sync tidb configmap in controller~~ sync tidb configmap in controller Dec 8, 2019

aylei commented Dec 8, 2019

View reviewed changes

remove useless code

25f4f28

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei requested review from cofyc and onlymellb December 8, 2019 13:08

aylei dismissed weekface’s stale review via 0f2f5fe December 9, 2019 14:10

Merge branch 'master' into fix-1192

40b89b9

aylei requested review from cofyc and weekface December 9, 2019 16:05

Refine tidb configmap e2e test

ccd1d32

Signed-off-by: Aylei <rayingecho@gmail.com>

Merge branch 'fix-1192' of github.com:aylei/tidb-operator into fix-1192

e29e113

Merge branch 'master' into fix-1192

e400ca2

cofyc previously approved these changes Dec 10, 2019

View reviewed changes

aylei requested a review from shuijing198799 December 10, 2019 07:48

Merge branch 'master' into fix-1192

ec209a3

tennix reviewed Dec 10, 2019

View reviewed changes

tennix added the type/refactor label Dec 10, 2019

aylei added 2 commits December 10, 2019 23:41

Address review comments

c194c66

Signed-off-by: Aylei <rayingecho@gmail.com>

Merge branch 'fix-1192' of github.com:aylei/tidb-operator into fix-1192

9ee91a4

aylei dismissed cofyc’s stale review via 9ee91a4 December 10, 2019 15:42

aylei requested review from tennix and cofyc December 10, 2019 16:57

tennix approved these changes Dec 10, 2019

View reviewed changes

cofyc approved these changes Dec 11, 2019

View reviewed changes

cofyc merged commit f13b93b into pingcap:master Dec 11, 2019

aylei mentioned this pull request Jan 6, 2020

Include component start script in control loop #1194

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sync tidb configmap in controller #1291

sync tidb configmap in controller #1291

aylei commented Dec 5, 2019 •

edited

Loading

aylei commented Dec 5, 2019

aylei commented Dec 5, 2019

aylei commented Dec 5, 2019

aylei commented Dec 6, 2019

aylei commented Dec 6, 2019

cofyc Dec 6, 2019

aylei Dec 6, 2019 •

edited

Loading

cofyc Dec 6, 2019

aylei Dec 6, 2019

aylei commented Dec 7, 2019

aylei commented Dec 7, 2019

aylei commented Dec 8, 2019

aylei Dec 8, 2019

aylei Dec 8, 2019 •

edited

Loading

aylei Dec 8, 2019

aylei Dec 8, 2019 •

edited

Loading

aylei commented Dec 8, 2019

aylei commented Dec 9, 2019

aylei commented Dec 9, 2019

aylei commented Dec 9, 2019

aylei commented Dec 10, 2019

aylei commented Dec 10, 2019

aylei commented Dec 10, 2019

cofyc left a comment

aylei commented Dec 10, 2019 •

edited

Loading

aylei commented Dec 10, 2019

tennix left a comment

tennix Dec 10, 2019

tennix Dec 10, 2019

aylei Dec 10, 2019

aylei commented Dec 10, 2019

tennix left a comment

sync tidb configmap in controller #1291

sync tidb configmap in controller #1291

Conversation

aylei commented Dec 5, 2019 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Does this PR introduce a user-facing change?:

aylei commented Dec 5, 2019

aylei commented Dec 5, 2019

aylei commented Dec 5, 2019

aylei commented Dec 6, 2019

aylei commented Dec 6, 2019

Choose a reason for hiding this comment

aylei Dec 6, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aylei commented Dec 7, 2019

aylei commented Dec 7, 2019

aylei commented Dec 8, 2019

Choose a reason for hiding this comment

aylei Dec 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aylei Dec 8, 2019 • edited Loading

Choose a reason for hiding this comment

aylei commented Dec 8, 2019

aylei commented Dec 9, 2019

aylei commented Dec 9, 2019

aylei commented Dec 9, 2019

aylei commented Dec 10, 2019

aylei commented Dec 10, 2019

aylei commented Dec 10, 2019

cofyc left a comment

Choose a reason for hiding this comment

aylei commented Dec 10, 2019 • edited Loading

aylei commented Dec 10, 2019

tennix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aylei commented Dec 10, 2019

tennix left a comment

Choose a reason for hiding this comment

aylei commented Dec 5, 2019 •

edited

Loading

aylei Dec 6, 2019 •

edited

Loading

aylei Dec 8, 2019 •

edited

Loading

aylei Dec 8, 2019 •

edited

Loading

aylei commented Dec 10, 2019 •

edited

Loading