v1alpha2: Add implementation #526

gaocegege · 2018-04-11T04:12:31Z

/assign @ddysher @DjangoPeng @jlewi @mitake @ScorpioCPH

This PR is the implementation of v1alpha2 and it is mainly contributed by @ScorpioCPH and @gaocegege

This change is

coveralls · 2018-04-11T05:40:44Z

Coverage increased (+1.6%) to 49.92% when pulling 99f6a9d on gaocegege:implementation into 6e706d6 on kubeflow:master.

gaocegege · 2018-04-11T05:44:00Z

The test coverage of v1alpha2 should be 70%+, PTAL

mitake · 2018-04-11T07:10:01Z

cmd/tf-operator.v2/app/options/options.go

+)
+
+// ServerOption is the main context object for the controller manager.
+type ServerOption struct {


How about adding the flag enableGangScheduling https://github.com/kubeflow/tf-operator/blob/master/cmd/tf-operator/app/options/options.go#L29 ? Probably I'm the most suitable person for doing this, so please let me know if you want me to add the field for the gang scheduling after merging this PR :)

#490
We have an issueto keep track of the progress of syncing commits with v1alpha1. Really appreciate if you could help us implement in v1alpha2. And as kube-arbitrator maintainers said, maybe we do not need to import PDB now.

mitake · 2018-04-11T07:24:18Z

pkg/controller.v2/controller_pod_test.go

+	}
+	go run(stopCh)
+
+	var key string


I'm not fully sure but this style of communication would be warned by the race detector? How about adding a mutex or pass the key via a channel?

Yeah I think so. I will open an issue to keep track. I think it is better not to import more enhancements in this PR since it is copied from upstream/v1alpha2 branch and I think we could record the potential issues and fix it after it is merged.

WDYT?

Yes, opening an issue and tracking would be ok.

jlewi · 2018-04-11T13:44:15Z

Would you mind submitting a separate PR for the signals package just to cut down the size of this PR a bit?

gaocegege · 2018-04-11T13:49:46Z

@jlewi Sure I will do it.

gaocegege · 2018-04-11T13:50:54Z

@jlewi But the test will fail because the code can not be built.

ddysher

thanks for the PR!

I only briefly scanned the PR, will walk though the code again later :)

ddysher · 2018-04-11T08:57:24Z

version/version.go

 )

 var (
-	Version = "0.3.0+git"
+	Version = "v0.1.0-alpha"


is this still the right version?

We released a version v0.1.0 and I am not sure if we should set the version to it. Since it is behind the master.

ddysher · 2018-04-11T09:42:18Z

pkg/controller.v2/controller.go

+	// To allow injection of syncTFJob for testing.
+	syncHandler func(tfJobKey string) (bool, error)
+
+	updateStatusHandler func(tfjob *tfv1alpha2.TFJob) error


why is this saved as a handler?

Because we need to mock it for test.

can u add a comment?

ddysher · 2018-04-11T09:43:11Z

cmd/tf-operator.v2/app/options/options.go

+
+// AddFlags adds flags for a specific CMServer to the specified FlagSet.
+func (s *ServerOption) AddFlags(fs *flag.FlagSet) {
+	fs.StringVar(&s.Kubeconfig, "kubeconfig", "~/.kube/config",


per discussion in #522, should this default flag be removed?

Yeah, I will update it.

ddysher · 2018-04-11T09:51:19Z

pkg/controller.v2/controller_pod.go

+// getPodsForTFJob returns the set of pods that this tfjob should manage.
+// It also reconciles ControllerRef by adopting/orphaning.
+// Note that the returned Pods are pointers into the cache.
+func (tc *TFJobController) getPodsForTFJob(tfjob *tfv1alpha2.TFJob) ([]*v1.Pod, error) {


correct me if i'm wrong, but getPodsForTFJob doesn't seem to capture the intent of this method - claiming (i.e. patch ownership) pods are already done in the method, returning a list of pods is just a side effect of calling this method.

Yeah I think we should change the name to getAndClaimPodsForTFJob. And I think it is better to remove the logic of claiming， since there is no need to claim the pods every time. I can open an issue for it.

WDYT @ScorpioCPH

this function name follows the other controller style, like this https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/job/job_controller.go#L401.

ddysher · 2018-04-11T10:05:55Z

pkg/controller.v2/controller_service.go

+					Ports: []v1.ServicePort{
+						{
+							Name: genGeneralName(tfjobKey, rt, index),
+							Port: 2222,


why is the port hard coded here?

I will update it.

We removed the field TFPort, thus we should reuse the ports in podtemplatespec. And I will open an issue to keep track. Currently we use a hard coded port and do not support customization.

I do not intent to implement in this PR since it is copied from upstream/v1alpha2. I will file a new PR for this feature.

WDYT?

ddysher · 2018-04-11T14:07:02Z

pkg/controller.v2/controller_ref_manager.go

+limitations under the License.
+*/
+
+// Note(CPH): this file is copied form k8s.io/kubernetes/pkg/controller


how hard would it be to vendor this?

@ScorpioCPH

We have no vendor k8s.io/kubernetes/, and CPH thinks that there is no need to import such a vendor for this struct.

WDYT @ddysher

Yes, k8s.io/kubernetes is too large for us.

gaocegege · 2018-04-11T14:19:39Z

The test will fail since there is no signals package.

Signed-off-by: Ce Gao <gaoce@caicloud.io>

ScorpioCPH · 2018-04-11T04:32:05Z

cmd/tf-operator.v2/app/server.go

+const RecommendedKubeConfigPathEnv = "KUBECONFIG"
+
+func Run(opt *options.ServerOption) error {
+


Please remove blank line.

Signed-off-by: Ce Gao <gaoce@caicloud.io>

gaocegege · 2018-04-12T06:32:50Z

Rebased upstream/master and addressed some comments. PTAL

Signed-off-by: Ce Gao <gaoce@caicloud.io>

jlewi · 2018-04-13T03:28:45Z

Review status: 0 of 15 files reviewed at latest revision, 8 unresolved discussions.

cmd/tf-operator.v2/app/options/options.go, line 38 at r2 (raw file):

Previously, gaocegege (Ce Gao) wrote…

Yeah, I will update it.

Don't we get the path of the KubeCOnfig from an environment variable? So is the plan to get rid of this flag altogether?

pkg/controller.v2/controller.go, line 152 at r7 (raw file):

expectations

Can you explain this to me? Why do track expectations in the form of aggregated statistics?

Why wouldn't we keep track of expectations for each replica? e.g. have an expectation for
service-0 create
service-1 create

If we just look at aggregated statistics won't we have problems? e.g. suppose we expect 2 workers
So in the current implementation we expect 2 create requests but if we get
service - 0 create
service -0 create

Then we haven't actually created both services even though we observed 2 create events.

pkg/controller.v2/controller_pod.go, line 54 at r7 (raw file):

	// Expect to have `replicas - succeeded` pods alive.
	expected := *spec.Replicas - succeeded

Why are you only checking the number of pods? Why aren't you checking that each individual replica has a pod in the expected state?

Not only will checking each replica be more robust, but it positions us to do a better job debugging problems because we will know what state each replica is in.

pkg/controller.v2/controller_pod.go, line 61 at r7 (raw file):

		err := tc.updateTFJobConditions(tfjob, tfv1alpha2.TFJobSucceeded, tfJobSucceededReason, msg)
		if err != nil {
			log.Infof("Append tfjob condition error: %v", err)

Here and everywhere else please use a context logger like
https://github.com/kubeflow/tf-operator/blob/master/pkg/trainer/training.go#L89

And log fields for the
job namespace + name
uuid
replica (if applicable)
replica index if applicable

This will ensure these fields are stored as metadata in the json log entry which makes filtering log messages for a particular job to figure out what happened easier.

pkg/controller.v2/controller_pod.go, line 94 at r7 (raw file):

	diff := len(activePods) - int(expected)

	if diff < 0 {

Per comments above; can we check that every replica is in the expected state?

pkg/controller.v2/controller_pod.go, line 109 at r7 (raw file):

		for _, index := range diffIndexes {
			log.Infof("need to create new pod: %s-%s", rt, index)

Please use logrus to log this with rt and index as metadata fields so we can easily filter log messages by replica.

This will make it really easy to filter logs to find out what happened during a replicas life.

pkg/controller.v2/controller_pod.go, line 148 at r7 (raw file):

			// Set restart policy
			if spec.RestartPolicy != "ExitCode" {

nit; Define a named constant

pkg/controller.v2/controller_pod.go, line 203 at r7 (raw file):

		if _, ok := desiredIndexes[index]; ok {
			desiredIndexes[index] = hit
		}

Should we log a warning if index isn't found in desiredIndexes? That would seem like it indicates a bug of some sort since that shouldn't happen.

pkg/controller.v2/controller_pod.go, line 218 at r7 (raw file):

// getPodsForTFJob returns the set of pods that this tfjob should manage.
// It also reconciles ControllerRef by adopting/orphaning.
// Note that the returned Pods are pointers into the cache.

What does adopting/orphaning pods mean? I assume it means changing the owner ref on the pod to point to this TFJob controller.

Do we actually expect this to happen? Why would the labels or selector get changed on the pod?

pkg/controller.v2/controller_pod.go, line 295 at r7 (raw file):

		tfjob := tc.resolveControllerRef(pod.Namespace, controllerRef)
		if tfjob == nil {
			log.Info("This pod's tfjob does not exists")

Lets add appropriate metadata fields to the long entry; e.g. tfjob name and uuid and maybe replica type and index?

Comments from Reviewable

gaocegege · 2018-04-17T01:57:21Z

cmd/tf-operator.v2/app/options/options.go, line 38 at r2 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Why do we need two ways of specifying the path to kubeconfig? Even if users are running locally they should be able to set the environment variable KUBECONFIG.

I think its highly undesirable to have both an environment variable and a command line flag. This inevitably leads to confusion about which should take precedence.

Make sense. I could remove the flag later. Personally, I think it is better not to import more enhancements into the PR, since I think it is a sync with upstream/v1alpha2. We are using it to implement some systems and I hope we could use this commit instead of the branch v1alpha2. Thus keeping consistent matters.

Comments from Reviewable

gaocegege · 2018-04-17T01:59:08Z

pkg/controller.v2/controller_pod.go, line 54 at r7 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

I couldn't understand from #538 what the bug is. What's the plan for this PR?

Please see above, could we merge it and enhance the implementation iteratively?

Comments from Reviewable

gaocegege · 2018-04-17T02:00:29Z

pkg/controller.v2/controller_pod.go, line 94 at r7 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Do you plan to fix this in this PR?

No, I think it is a big change and I am not sure map is the best way. Thus we could merge it and enhance the implementation iteratively.

Comments from Reviewable

gaocegege · 2018-04-17T02:03:23Z

pkg/controller.v2/controller_pod.go, line 203 at r7 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Why not just fix it in this PR?

Because we need to import logger and then do it. I think we could do it after the PR merged. Or I could file a new PR based on this PR to fix it.

Comments from Reviewable

jlewi · 2018-04-18T00:48:43Z

Review status: 3 of 15 files reviewed at latest revision, 12 unresolved discussions.

cmd/tf-operator.v2/app/options/options.go, line 38 at r2 (raw file):

Previously, gaocegege (Ce Gao) wrote…

Make sense. I could remove the flag later. Personally, I think it is better not to import more enhancements into the PR, since I think it is a sync with upstream/v1alpha2. We are using it to implement some systems and I hope we could use this commit instead of the branch v1alpha2. Thus keeping consistent matters.

I don't understand your point about doing more changes in this PR. The point of this PR is to start checking code into master.

Why would we treat the work going on in the v1alpha2 branch any differently from work going on in a local branch

I think we should make all the changes in this PR such that the resulting code that's checked into master is in a good state and the source of truth.

So I think we should make this and other straightforward changes in this PR. I don't think we should check in code only to remove it in a follow on PR.

pkg/controller.v2/controller_pod.go, line 94 at r7 (raw file):

Previously, gaocegege (Ce Gao) wrote…

No, I think it is a big change and I am not sure map is the best way. Thus we could merge it and enhance the implementation iteratively.

If we're uncertain about how to implement it then why check it in?

Why not remove the code that we are uncertain about (replace it with a TODO).

Alternatively, is there a simpler, less efficient but correct implementation we could check in? I don't like the idea of checking in code that we don't think works. Could we just iterate over all the replicas and check for each one if we need to do any work?

pkg/controller.v2/controller_pod.go, line 203 at r7 (raw file):

Previously, gaocegege (Ce Gao) wrote…

Because we need to import logger and then do it. I think we could do it after the PR merged. Or I could file a new PR based on this PR to fix it.

Isn't controller_pod.go already importing log?

Comments from Reviewable

ScorpioCPH · 2018-04-18T02:10:42Z

Review status: 3 of 15 files reviewed at latest revision, 13 unresolved discussions.

pkg/controller.v2/controller_pod.go, line 94 at r7 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

If we're uncertain about how to implement it then why check it in?

Why not remove the code that we are uncertain about (replace it with a TODO).

Alternatively, is there a simpler, less efficient but correct implementation we could check in? I don't like the idea of checking in code that we don't think works. Could we just iterate over all the replicas and check for each one if we need to do any work?

Hi, getDiffPodIndexes() will check which replica should be handled.

pkg/controller.v2/controller_pod.go, line 218 at r7 (raw file):

Previously, gaocegege (Ce Gao) wrote…

Yeah, the code refers to the job controller and I think we do not expect that happen but there might be. Thus I think we should claim them and deal with them later.

We can't assume that only tf-operator will change the pods' status, so we should claiming these pods.

Comments from Reviewable

Signed-off-by: Ce Gao <gaoce@caicloud.io>

gaocegege · 2018-04-18T02:45:18Z

cmd/tf-operator.v2/app/options/options.go, line 38 at r2 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

I don't understand your point about doing more changes in this PR. The point of this PR is to start checking code into master.

Why would we treat the work going on in the v1alpha2 branch any differently from work going on in a local branch

I think we should make all the changes in this PR such that the resulting code that's checked into master is in a good state and the source of truth.

So I think we should make this and other straightforward changes in this PR. I don't think we should check in code only to remove it in a follow on PR.

Done

Comments from Reviewable

gaocegege · 2018-04-18T03:01:20Z

pkg/controller.v2/controller_pod.go, line 94 at r7 (raw file):

Previously, ScorpioCPH (Penghao Cen) wrote…

Hi, getDiffPodIndexes() will check which replica should be handled.

@ScorpioCPH I think Jeremy means that we use an integer instead of a map to represent the succeeded/failed pods.

Comments from Reviewable

jlewi · 2018-04-18T03:55:52Z

Review status: 3 of 15 files reviewed at latest revision, 12 unresolved discussions.

pkg/controller.v2/controller_pod.go, line 94 at r7 (raw file):

Previously, gaocegege (Ce Gao) wrote…

@ScorpioCPH I think Jeremy means that we use an integer instead of a map to represent the succeeded/failed pods.

I think the problem is that we check if diff < 0 to decided whether there is any work to do.

As noted in my earlier comment I think that check is insufficient under certain conditions; e.g. the case where you have multiple pods for a given.

You already get a list of all pods for this job. So I think you want to group these pods based on replica. You can then correctly check

If there is a replica missing some pods
If there are multiple pods for a replica; you can figure out what the correct state and action is.

pkg/controller.v2/controller_pod.go, line 218 at r7 (raw file):

Previously, ScorpioCPH (Penghao Cen) wrote…

We can't assume that only tf-operator will change the pods' status, so we should claiming these pods.

Thanks.

Comments from Reviewable

u2takey · 2018-04-18T04:10:41Z

@jlewi @gaocegege v1alpha2 is not just a api change? we are doing a totally refactoring??? can we still keep compatiblty with v1alpha1?

Signed-off-by: Ce Gao <gaoce@caicloud.io>

gaocegege · 2018-04-18T05:40:55Z

@u2takey We can not since API changes. We refactored the operator because of #300

Signed-off-by: Ce Gao <gaoce@caicloud.io>

gaocegege · 2018-04-18T09:23:54Z

pkg/controller.v2/controller_pod.go, line 203 at r7 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Isn't controller_pod.go already importing log?

Done.

Comments from Reviewable

gaocegege · 2018-04-18T10:00:24Z

pkg/controller.v2/controller_pod.go, line 94 at r7 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

I think the problem is that we check if diff < 0 to decided whether there is any work to do.

As noted in my earlier comment I think that check is insufficient under certain conditions; e.g. the case where you have multiple pods for a given.

You already get a list of all pods for this job. So I think you want to group these pods based on replica. You can then correctly check

If there is a replica missing some pods

If there are multiple pods for a replica; you can figure out what the correct state and action is.

I cam not sure if I understand you. Do you mean the code similar to here:

func (tc *TFJobController) syncPods(pods []*v1.Pod, replicas int, logger  *log.Entry) {
	podSlices := getPodSlices(pods, replicas， logger)
	for index, podSlice := range podSlices {
		if len(podSlice) > 1 {
			logger.Warning("We have to many pods for the worker %d", index)
			// Kill some
		}
		if len(podsSlice) == 0 {
			// Create one
		}
		pod := podSlice[0]
		// We already have one, and check if it is succeede or something else.
	}
}

func getPodSlices(pods []*v1.Pod, replicas int, logger *log.Entry) [][]*v1.Pod {
	resMap := make([][]*v1.Pod)
	for _, pod := range pods {
		if _, ok := pod.Labels[tfReplicaIndexLabel]; !ok {
			logger.Warning("The pod do not have the index label.")
		}
		index, err := strconv.Atoi(pod.Labels[tfReplicaIndexLabel])
		if err != nil {
			logger.Warning("Error when strconv.Atoi: %v", err)
		}
		if index < 0 || index >= replicas {
			logger.Warningf("The label index is not expected: %d", index)
		}

		resMap[index] = append(resMap[index], pod)
	}
}

Comments from Reviewable

jlewi · 2018-04-19T00:28:50Z

Review status: 3 of 17 files reviewed at latest revision, 11 unresolved discussions.

cmd/tf-operator.v2/app/options/options.go, line 38 at r2 (raw file):

Previously, gaocegege (Ce Gao) wrote…

Done

Thanks.

pkg/controller.v2/controller_pod.go, line 94 at r7 (raw file):

Previously, gaocegege (Ce Gao) wrote…

I cam not sure if I understand you. Do you mean the code similar to here:

func (tc *TFJobController) syncPods(pods []*v1.Pod, replicas int, logger  *log.Entry) {
	podSlices := getPodSlices(pods, replicas， logger)
	for index, podSlice := range podSlices {
		if len(podSlice) > 0 {
			logger.Warning("We have to many pods for the worker %d", index)
			// Kill some
		}
		if len(podsSlice) == 0 {
			// Create one
		}
		pod := podSlice[0]
		// We already have one, and check if it is succeede or something else.
	}
}

func getPodSlices(pods []*v1.Pod, replicas int, logger *log.Entry) [][]*v1.Pod {
	resMap := make([][]*v1.Pod)
	for _, pod := range pods {
		if _, ok := pod.Labels[tfReplicaIndexLabel]; !ok {
			logger.Warning("The pod do not have the index label.")
		}
		index, err := strconv.Atoi(pod.Labels[tfReplicaIndexLabel])
		if err != nil {
			logger.Warning("Error when strconv.Atoi: %v", err)
		}
		if index < 0 || index >= replicas {
			logger.Warningf("The label index is not expected: %d", index)
		}

		resMap[index] = append(resMap[index], pod)
	}
}

Exactly. I'd be fine merging that snippet with TODOs to flush out the implementation.

Comments from Reviewable

Signed-off-by: Ce Gao <gaoce@caicloud.io>

gaocegege · 2018-04-19T02:28:25Z

pkg/controller.v2/controller_pod.go, line 94 at r7 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Exactly. I'd be fine merging that snippet with TODOs to flush out the implementation.

I added the snippet to the code and add a TODO in reconcilePods, PTAL

Comments from Reviewable

ScorpioCPH · 2018-04-19T05:21:35Z

@gaocegege Great job! Looking forward to seeing this PR merged.

jlewi · 2018-04-20T06:37:57Z

Thanks!
/lgtm
/approve

k8s-ci-robot · 2018-04-20T06:38:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* *: Add implementation for v1alpha2 Signed-off-by: Ce Gao <gaoce@caicloud.io> * linter: Fix Signed-off-by: Ce Gao <gaoce@caicloud.io> * options: Remove default config for kubeconfig Signed-off-by: Ce Gao <gaoce@caicloud.io> * service: Add const Signed-off-by: Ce Gao <gaoce@caicloud.io> * server.go: Remove new lines Signed-off-by: Ce Gao <gaoce@caicloud.io> * *: Fix import styles Signed-off-by: Ce Gao <gaoce@caicloud.io> * controller.go: Add comment for handler Signed-off-by: Ce Gao <gaoce@caicloud.io> * controller_pod: Fix name Signed-off-by: Ce Gao <gaoce@caicloud.io> * options: Remove kubeconfig Signed-off-by: Ce Gao <gaoce@caicloud.io> * *: Update Signed-off-by: Ce Gao <gaoce@caicloud.io> * controller_pod: Add TODO Signed-off-by: Ce Gao <gaoce@caicloud.io>

k8s-ci-robot assigned ddysher, DjangoPeng, jlewi and mitake Apr 11, 2018

k8s-ci-robot added the size/XXL label Apr 11, 2018

k8s-ci-robot requested review from jimexist and mitake April 11, 2018 04:12

mitake reviewed Apr 11, 2018

View reviewed changes

gaocegege mentioned this pull request Apr 11, 2018

[v1alpha2][test] Avoid potential data race problem #530

Closed

ddysher reviewed Apr 11, 2018

View reviewed changes

gaocegege added 4 commits April 12, 2018 11:28

*: Add implementation for v1alpha2

e8b7aa3

Signed-off-by: Ce Gao <gaoce@caicloud.io>

linter: Fix

545607a

Signed-off-by: Ce Gao <gaoce@caicloud.io>

options: Remove default config for kubeconfig

7cbd1aa

Signed-off-by: Ce Gao <gaoce@caicloud.io>

service: Add const

fe91b58

Signed-off-by: Ce Gao <gaoce@caicloud.io>

This was referenced Apr 12, 2018

[v1alpha2] Suppport customized port #532

Closed

[v1alpha2] Try to not to always claim pods #533

Closed

ScorpioCPH reviewed Apr 12, 2018

View reviewed changes

gaocegege added 2 commits April 12, 2018 14:18

server.go: Remove new lines

49e1eff

Signed-off-by: Ce Gao <gaoce@caicloud.io>

*: Fix import styles

ef7ae48

Signed-off-by: Ce Gao <gaoce@caicloud.io>

ddysher mentioned this pull request Apr 12, 2018

v1alpha2: Implementation #492

Closed

controller.go: Add comment for handler

5390442

Signed-off-by: Ce Gao <gaoce@caicloud.io>

options: Remove kubeconfig

ec42617

Signed-off-by: Ce Gao <gaoce@caicloud.io>

gaocegege referenced this pull request Apr 18, 2018

developer_guide: Add instructions for v1alpha2 (#528)

ecd792c

Signed-off-by: Ce Gao <gaoce@caicloud.io>

*: Update

6338fcd

Signed-off-by: Ce Gao <gaoce@caicloud.io>

controller_pod: Add TODO

99f6a9d

Signed-off-by: Ce Gao <gaoce@caicloud.io>

k8s-ci-robot added the lgtm label Apr 20, 2018

k8s-ci-robot added the approved label Apr 20, 2018

k8s-ci-robot merged commit 9e07f6d into kubeflow:master Apr 20, 2018

gaocegege deleted the implementation branch April 20, 2018 07:13

This was referenced Apr 20, 2018

[v1alpha2] start using kubeconfig #522

Closed

[v1alpha2] Enhance the logic about sync #547

Closed

		const RecommendedKubeConfigPathEnv = "KUBECONFIG"

		func Run(opt *options.ServerOption) error {

v1alpha2: Add implementation #526

v1alpha2: Add implementation #526

Conversation

gaocegege commented Apr 11, 2018 • edited by jlewi Loading

coveralls commented Apr 11, 2018 • edited Loading

gaocegege commented Apr 11, 2018

Choose a reason for hiding this comment

gaocegege Apr 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlewi commented Apr 11, 2018

gaocegege commented Apr 11, 2018 • edited Loading

gaocegege commented Apr 11, 2018

ddysher left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaocegege Apr 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaocegege commented Apr 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaocegege commented Apr 12, 2018

jlewi commented Apr 13, 2018

gaocegege commented Apr 17, 2018

gaocegege commented Apr 17, 2018

gaocegege commented Apr 17, 2018

gaocegege commented Apr 17, 2018

jlewi commented Apr 18, 2018

ScorpioCPH commented Apr 18, 2018

gaocegege commented Apr 18, 2018

gaocegege commented Apr 18, 2018

jlewi commented Apr 18, 2018

u2takey commented Apr 18, 2018

gaocegege commented Apr 18, 2018

gaocegege commented Apr 18, 2018

gaocegege commented Apr 18, 2018 • edited Loading

jlewi commented Apr 19, 2018

gaocegege commented Apr 19, 2018

ScorpioCPH commented Apr 19, 2018

jlewi commented Apr 20, 2018

k8s-ci-robot commented Apr 20, 2018

gaocegege commented Apr 11, 2018 •

edited by jlewi

Loading

coveralls commented Apr 11, 2018 •

edited

Loading

gaocegege Apr 11, 2018 •

edited

Loading

gaocegege commented Apr 11, 2018 •

edited

Loading

ddysher left a comment •

edited

Loading

gaocegege Apr 12, 2018 •

edited

Loading

gaocegege commented Apr 18, 2018 •

edited

Loading