Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tweak node pool usage #984

Closed
wants to merge 3 commits into from
Closed

Tweak node pool usage #984

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7508ba6
add remaining clearml resources to clearml node pool
leej3 Dec 28, 2021
9805ade
add node affinity for forward auth resource
leej3 Jan 5, 2022
c15c224
set mongodb and redis node selector
leej3 Jan 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions qhub/template/{{ cookiecutter.repo_directory }}/infrastructure/kubernetes.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,7 @@ module "forwardauth" {
namespace = var.environment
external-url = var.endpoint

node-group = local.node_groups.general
jh-client-id = local.forwardauth-keycloak-client-id
jh-client-secret = random_password.forwardauth-jhsecret.result
callback-url-path = local.forwardauth-callback-url-path
Expand Down
16 changes: 15 additions & 1 deletion ...e/{{ cookiecutter.repo_directory }}/infrastructure/modules/kubernetes/forwardauth/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,21 @@ resource "kubernetes_deployment" "forwardauth-deployment" {
}

spec {

affinity {
node_affinity {
required_during_scheduling_ignored_during_execution {
node_selector_term {
match_expressions {
key = var.node-group.key
operator = "In"
values = [
var.node-group.value
]
}
}
}
}
}
container {
# image = "thomseddon/traefik-forward-auth:2.2.0"
# Use PR #159 https://github.com/thomseddon/traefik-forward-auth/pull/159
Expand Down
10 changes: 9 additions & 1 deletion ...cookiecutter.repo_directory }}/infrastructure/modules/kubernetes/forwardauth/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,12 @@ variable "jh-client-secret" {
variable "callback-url-path" {
description = "Path of Callback URL"
type = string
}
}

variable "node-group" {
description = "Node key value pair for bound general resources"
type = object({
key = string
value = string
})
}
4 changes: 4 additions & 0 deletions ...er.repo_directory }}/infrastructure/modules/kubernetes/services/clearml/chart/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,8 @@ redis: # configuration from https://github.com/bitnami/charts/blob/master/bitnam
master:
name: "{{ .Release.Name }}-redis-master"
port: 6379
nodeSelector:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is the node being labeled "app: clearml"?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or this an assumption that the node will have this label?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is the node being labeled "app: clearml"

good point. It's a default value set in the variable.tf file that propagest to the chart's values.yaml (see here)).

Or this an assumption that the node will have this label?

It is a requirement. We manually set this label for our deployment. We are not sure if this is automated when QHub deploys clearml when using the cloud providers (as described here).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that drew our attention to missing redis/mongdb dynamic value setting. fixed now.

app: "clearml"
persistence:
enabled: true
accessModes:
Expand All @@ -240,6 +242,8 @@ mongodb: # configuration from https://github.com/bitnami/charts/blob/master/bitn
registry: docker.io
repository: bitnami/mongodb
tag: 3.6.21-debian-9-r71
nodeSelector:
app: "clearml"
architecture: standalone
auth:
enabled: false
Expand Down
16 changes: 16 additions & 0 deletions ...cookiecutter.repo_directory }}/infrastructure/modules/kubernetes/services/clearml/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,20 @@ resource "helm_release" "clearml" {
}
}

dynamic "set" {
for_each = var.node_selector
content {
name = "mongodb.nodeSelector.${set.key}"
value = set.value
}
}

dynamic "set" {
for_each = var.node_selector
content {
name = "redis.master.nodeSelector.${set.key}"
value = set.value
}
}

}