Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grafana container in CrashLoopBackoff on GKE #582

Closed
siggy opened this issue Mar 17, 2018 · 1 comment
Closed

Grafana container in CrashLoopBackoff on GKE #582

siggy opened this issue Mar 17, 2018 · 1 comment
Assignees
Labels
Milestone

Comments

@siggy
Copy link
Member

siggy commented Mar 17, 2018

User reported the Grafana container in CrashLoopBackoff, with Conduit 0.3.1, on GKE.

Related issues:

Confirmed fix is in grafana/grafana:master. Will wait for this change to propagate to a release and then we can update to that image:

log output:

$ stern grafana -n conduit
+ grafana-66d57d5875-2zlxc › conduit-proxy
grafana-66d57d5875-2zlxc conduit-proxy INFO conduit_proxy using controller at HostAndPort { host: DnsName("proxy-api.conduit.svc.cluster.local"), port: 8086 }
grafana-66d57d5875-2zlxc conduit-proxy INFO conduit_proxy routing on V4(127.0.0.1:4140)
grafana-66d57d5875-2zlxc conduit-proxy INFO conduit_proxy proxying on V4(0.0.0.0:4143) to None
grafana-66d57d5875-2zlxc conduit-proxy INFO conduit_proxy::transport::connect "controller-client", DNS resolved DnsName("proxy-api.conduit.svc.cluster.local") to 10.196.240.97
grafana-66d57d5875-2zlxc conduit-proxy ERR! conduit_proxy::control "controller-client", controller error: Error attempting to establish underlying session layer: operation timed out after Duration { secs: 3, nanos: 0 }
grafana-66d57d5875-2zlxc conduit-proxy INFO conduit_proxy::transport::connect "controller-client", DNS resolved DnsName("proxy-api.conduit.svc.cluster.local") to 10.196.240.97
+ grafana-66d57d5875-2zlxc › grafana
grafana-66d57d5875-2zlxc grafana chown: changing ownership of '/var/lib/grafana/dashboards/..data': Read-only file system
grafana-66d57d5875-2zlxc grafana chown: changing ownership of '/var/lib/grafana/dashboards/..2018_03_16_23_33_08.827149221/conduit-viz.json': Read-only file system
grafana-66d57d5875-2zlxc grafana chown: changing ownership of '/var/lib/grafana/dashboards/..2018_03_16_23_33_08.827149221/conduit-health.json': Read-only file system
grafana-66d57d5875-2zlxc grafana chown: changing ownership of '/var/lib/grafana/dashboards/..2018_03_16_23_33_08.827149221': Read-only file system
grafana-66d57d5875-2zlxc grafana chown: changing ownership of '/var/lib/grafana/dashboards/conduit-viz.json': Read-only file system
grafana-66d57d5875-2zlxc grafana chown: changing ownership of '/var/lib/grafana/dashboards/conduit-health.json': Read-only file system
grafana-66d57d5875-2zlxc grafana chown: changing ownership of '/var/lib/grafana/dashboards': Read-only file system
@siggy siggy added this to the 0.4.0 milestone Mar 17, 2018
@siggy siggy self-assigned this Mar 17, 2018
xiaods added a commit to xiaods/conduit that referenced this issue Mar 29, 2018
this is a known issue with grafana in k8s. `grafana/grafana:5.0.4` was just released today. update the repo from `5.0.3` to `5.0.4`
fixed issues linkerd#582
xiaods added a commit to xiaods/conduit that referenced this issue Mar 29, 2018
this is a known issue with grafana in k8s. `grafana/grafana:5.0.4` was just released today. update the repo from `5.0.3` to `5.0.4`
fixed issues linkerd#582

Signed-off-by: Deshi Xiao <xiaods@gmail.com>
xiaods added a commit to xiaods/conduit that referenced this issue Mar 29, 2018
this is a known issue with grafana in k8s. grafana/grafana:5.0.4 was just released today. update the repo from 5.0.3 to 5.0.4
fixed issues linkerd#582

Signed-off-by: Deshi Xiao <xiaods@gmail.com>
siggy pushed a commit that referenced this issue Mar 29, 2018
this is a known issue with grafana in k8s. grafana/grafana:5.0.4 was just released today. update the repo from 5.0.3 to 5.0.4
fixed issues #582

Signed-off-by: Deshi Xiao <xiaods@gmail.com>
@siggy
Copy link
Member Author

siggy commented Mar 29, 2018

Fixed in #646 by @xiaods.

@siggy siggy closed this as completed Mar 29, 2018
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

1 participant