Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Jupyterhub server deployment fails #3615

Closed
vevetron opened this issue Jan 7, 2025 · 1 comment · Fixed by #3642
Closed

Bug: Jupyterhub server deployment fails #3615

vevetron opened this issue Jan 7, 2025 · 1 comment · Fixed by #3642

Comments

@vevetron
Copy link
Contributor

vevetron commented Jan 7, 2025

Describe the bug

Kubernetes deployment fails after PR - #3614 :
https://github.com/cal-itp/data-infra/actions/runs/12658177354/job/35275319227

Run poetry run invoke release -f "./kubernetes-workloads.yaml"
  poetry run invoke release -f "./kubernetes-workloads.yaml"
  shell: /usr/bin/bash -e ***0***
  env:
    GITHUB_TOKEN: ***
    CLOUDSDK_CORE_PROJECT: cal-itp-data-infra
    GKE_NAME: data-infra-apps
    GKE_REGION: us-west1
    USE_GKE_GCLOUD_AUTH_PLUGIN: true
    CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/work/data-infra/data-infra/gha-creds-6036e5683fbb6[2](https://github.com/cal-itp/data-infra/actions/runs/12658177354/job/35275319227#step:13:2)a8.json
    GOOGLE_APPLICATION_CREDENTIALS: /home/runner/work/data-infra/data-infra/gha-creds-60[3](https://github.com/cal-itp/data-infra/actions/runs/12658177354/job/35275319227#step:13:3)6e5683fbb62a8.json
    GOOGLE_GHA_CREDS_PATH: /home/runner/work/data-infra/data-infra/gha-creds-6036e5683fbb62a8.json
    CLOUDSDK_PROJECT: cal-itp-data-infra
    GCLOUD_PROJECT: cal-itp-data-infra
    GCP_PROJECT: cal-itp-data-infra
    GOOGLE_CLOUD_PROJECT: cal-itp-data-infra
    CLOUDSDK_METRICS_ENVIRONMENT: github-actions-setup-gcloud
    CLOUDSDK_METRICS_ENVIRONMENT_VERSION: 2.1.2
    KUBECONFIG: /home/runner/work/data-infra/data-infra/gha-kubeconfig-8aa0[4](https://github.com/cal-itp/data-infra/actions/runs/12658177354/job/35275319227#step:13:4)4c66b8764e9
    KUBE_CONFIG_PATH: /home/runner/work/data-infra/data-infra/gha-kubeconfig-8aa044c66b8764e9
Executing: kubectl apply -k /home/runner/work/data-infra/data-infra/kubernetes/apps/overlays/gtfs-rt-archiver-v3-prod
namespace/gtfs-rt-v3 unchanged
configmap/archiver-app-vars unchanged
configmap/archiver-channel-vars unchanged
service/redis unchanged
deployment.apps/gtfs-rt-archiver-consumer configured
deployment.apps/gtfs-rt-archiver-ticker configured
deployment.apps/redis configured
Executing: kubectl apply -k /home/runner/work/data-infra/data-infra/kubernetes/apps/manifests/airflow-jobs
namespace/airflow-jobs unchanged
priorityclass.scheduling.k8s.io/dbt-high-priority configured
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/runner/work/data-infra/data-infra/gha-kubeconfig-8aa044c66b8764e9
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/runner/work/data-infra/data-infra/gha-kubeconfig-8aa044c66b8764e9
Executing: helm dependency update /home/runner/work/data-infra/data-infra/kubernetes/apps/charts/grafana
Executing: helm upgrade grafana /home/runner/work/data-infra/data-infra/kubernetes/apps/charts/grafana --install --namespace monitoring-grafana --values=/home/runner/work/data-infra/data-infra/kubernetes/apps/values/grafana.yaml --values=/home/runner/work/data-infra/data-infra/kubernetes/apps/values/grafana-prod.yaml --reset-values 
Release "grafana" has been upgraded. Happy Helming!
NAME: grafana
LAST DEPLOYED: Tue Jan  7 20:03:21 202[5](https://github.com/cal-itp/data-infra/actions/runs/12658177354/job/35275319227#step:13:5)
NAMESPACE: monitoring-grafana
STATUS: deployed
REVISION: 70
TEST SUITE: None
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace monitoring-grafana grafana -o jsonpath="***.data.admin-password***" | base[6](https://github.com/cal-itp/data-infra/actions/runs/12658177354/job/35275319227#step:13:6)4 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.monitoring-grafana.svc.cluster.local

   If you bind grafana to 80, please update values in values.yaml and reinstall:

securityContext:
runAsUser: 0
runAsGroup: 0
fsGroup: 0

command:

  • "setcap"
  • "'cap_net_bind_service=+ep'"
  • "/usr/sbin/grafana-server &&"
  • "sh"
  • "/run.sh"
Details refer to https://grafana.com/docs/installation/configuration/#http-port.
Or grafana would always crash.

From outside the cluster, the server URL(s) are:
  http://monitoring.k8s.calitp.jarv.us
  http://monitoring.calitp.org

3. Login with the password from step 1 and the username: admin
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/runner/work/data-infra/data-infra/gha-kubeconfig-8aa044c66b8[7](https://github.com/cal-itp/data-infra/actions/runs/12658177354/job/35275319227#step:13:7)64e9
Executing: helm dependency update /home/runner/work/data-infra/data-infra/kubernetes/apps/charts/jupyterhub
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://jupyterhub.github.io/helm-chart" chart repository
Saving 1 charts
Downloading jupyterhub from repo https://jupyterhub.github.io/helm-chart
Deleting outdated charts
Downloaded secret helm values: /tmp/tmpsw6cotpc/jupyterhub_jupyterhub-sensitive-helm-values.yaml
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/runner/work/data-infra/data-infra/gha-kubeconfig-[8](https://github.com/cal-itp/data-infra/actions/runs/12658177354/job/35275319227#step:13:8)aa044c66b8764e9
Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
 * timed out waiting for the condition


Executing: helm upgrade jupyterhub /home/runner/work/data-infra/data-infra/kubernetes/apps/charts/jupyterhub --install --namespace jupyterhub --values=/tmp/tmpsw6cotpc/jupyterhub_jupyterhub-sensitive-helm-values.yaml --reset-values --timeout [10](https://github.com/cal-itp/data-infra/actions/runs/12658177354/job/35275319227#step:13:10)m0s
Error: Process completed with exit code 1.

After making changes

To Reproduce
Re-run failed deployment here: https://github.com/cal-itp/data-infra/actions/workflows/deploy-kubernetes.yml

Expected behavior
Works well like:
https://github.com/cal-itp/data-infra/actions/runs/11395015198/job/31706447087?pr=3508

Additional context
Happy Tuesday! Happy New Year!

@vevetron vevetron changed the title Bug: Bug: Jupyterhub server deployment fails Jan 7, 2025
@edasmalchi edasmalchi mentioned this issue Jan 8, 2025
6 tasks
@vevetron
Copy link
Contributor Author

vevetron commented Jan 9, 2025

I wrote the name wrong.
From lens:

Failed to apply default image tag "map[name:ghcr.io/cal-itp/data-infra/jupyter-singleuser tag:2024.5.24]": couldn't parse image name "map[name:ghcr.io/cal-itp/data-infra/jupyter-singleuser tag:2024.5.24]": invalid reference format

https://z2jh.jupyter.org/en/latest/jupyterhub/customizing/user-environment.html#multiple-profiles

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant