Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix katib-manager crash in kubeflow cluster #900

Merged
merged 1 commit into from
Oct 28, 2019
Merged

Fix katib-manager crash in kubeflow cluster #900

merged 1 commit into from
Oct 28, 2019

Conversation

hougangliu
Copy link
Member

@hougangliu hougangliu commented Oct 28, 2019

In kubeflow cluster, there is a service named "mysql" from pipeline. so in each kubeflow POD, there is will ENV for mysql service like below, which will make getDbName() return in katib mysql DB invalid (owning to env MYSQL_PORT)

# env|grep MYSQL
MYSQL_PORT_3306_TCP_ADDR=10.0.251.111
MYSQL_PORT_3306_TCP_PORT=3306
MYSQL_SERVICE_HOST=10.0.251.111
MYSQL_PORT_3306_TCP_PROTO=tcp
MYSQL_HOST=katib-db.kubeflow
MYSQL_PORT=tcp://10.0.251.111:3306
MYSQL_SERVICE_PORT=3306
MYSQL_PORT_3306_TCP=tcp://10.0.251.111:3306

This change is Reviewable

@hougangliu
Copy link
Member Author

/cc @gaocegege @johnugeorge

@johnugeorge
Copy link
Member

Great! what we suspected about env var.

Still, how does env variable get propagated into Katib pod?

@hougangliu
Copy link
Member Author

hougangliu commented Oct 28, 2019

Still, how does env variable get propagated into Katib pod?

it is Kubernetes feature, all services in the same namespace will be imported into the Pods as env

@gaocegege
Copy link
Member

Ref https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables

Copy link
Member

@gaocegege gaocegege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@johnugeorge
Copy link
Member

/approve

@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 606736d into kubeflow:master Oct 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants