source-controller crashes after 20ms #1526
Comments
|
Duplicate of fluxcd/source-controller#385 |
|
Ah my bad. I had the problem yesterday and didn't see any similar issues before I went to bed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I updated to v0.15 and noticed that the source-controller was not working. I have checked the logs using both
kubectl logsanddocker logsbut it shows nothing. Everything else about the cluster seems to work as normal and all my other services and pods are working.To Reproduce
Steps to reproduce the behaviour:
I am unsure. I know that yesterday I updated docker on all my nodes and I updated flux. I have tried to uninstall flux and install it again but the source-controller still won't start. I also tried downgrading to v0.14.2 with no luck. I suspect it has something to do with me updating docker, but I can't tell because the source-controller stops without giving me any logs.
Expected behavior
The source-controller should be running.
Additional context
No logs for source-controller.
kustomize-controller logs:
docker inspect:
[ { "Id": "b7ee787235af31383f091c90f350e1ee2039e943160741df8872efc4cd5285e2", "Created": "2021-06-16T10:33:51.552645701Z", "Path": "/sbin/tini", "Args": [ "--", "source-controller", "--events-addr=http://notification-controller/", "--watch-all-namespaces=true", "--log-level=info", "--log-encoding=json", "--enable-leader-election", "--storage-path=/data", "--storage-adv-addr=source-controller.flux-system.svc.cluster.local." ], "State": { "Status": "exited", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 139, "Error": "", "StartedAt": "2021-06-16T10:33:52.396778837Z", "FinishedAt": "2021-06-16T10:33:52.418779425Z" }, "Image": "sha256:dee0ecbd8fcd562f6a3e5fb8d42b3e608b638211d602eb5247f0936114b68079", "ResolvConfPath": "/var/lib/docker/containers/d1a0f096d270786a3627db397c47eb9c0de6177b9d40fb6fb42ff7faeb6caa78/resolv.conf", "HostnamePath": "/var/lib/docker/containers/d1a0f096d270786a3627db397c47eb9c0de6177b9d40fb6fb42ff7faeb6caa78/hostname", "HostsPath": "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/etc-hosts", "LogPath": "/var/lib/docker/containers/b7ee787235af31383f091c90f350e1ee2039e943160741df8872efc4cd5285e2/b7ee787235af31383f091c90f350e1ee2039e943160741df8872efc4cd5285e2-json.log", "Name": "/k8s_manager_source-controller-cdb67965b-g9h2d_flux-system_0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4_5", "RestartCount": 0, "Driver": "devicemapper", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/volumes/kubernetes.io~empty-dir/data:/data", "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/volumes/kubernetes.io~empty-dir/tmp:/tmp", "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/volumes/kubernetes.io~projected/kube-api-access-8rg5p:/var/run/secrets/kubernetes.io/serviceaccount:ro", "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/etc-hosts:/etc/hosts", "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/containers/manager/c59a9147:/dev/termination-log" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "container:d1a0f096d270786a3627db397c47eb9c0de6177b9d40fb6fb42ff7faeb6caa78", "PortBindings": null, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "CgroupnsMode": "host", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": [ "1337" ], "IpcMode": "container:d1a0f096d270786a3627db397c47eb9c0de6177b9d40fb6fb42ff7faeb6caa78", "Cgroup": "", "Links": null, "OomScoreAdj": 992, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": true, "SecurityOpt": [ "no-new-privileges", "seccomp=unconfined" ], "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 51, "Memory": 1073741824, "NanoCpus": 0, "CgroupParent": "/kubepods/burstable/pod0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 100000, "CpuQuota": 100000, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 1073741824, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/asound", "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "DeviceId": "26509", "DeviceName": "docker-0:18-18751079-090d9d0db2176ab0fd5809e057fce94582d0097f550d1c7dd294d6022e220f33", "DeviceSize": "10737418240" }, "Name": "devicemapper" }, "Mounts": [ { "Type": "bind", "Source": "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/volumes/kubernetes.io~empty-dir/data", "Destination": "/data", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/volumes/kubernetes.io~empty-dir/tmp", "Destination": "/tmp", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/volumes/kubernetes.io~projected/kube-api-access-8rg5p", "Destination": "/var/run/secrets/kubernetes.io/serviceaccount", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/etc-hosts", "Destination": "/etc/hosts", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/kubelet/pods/0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/containers/manager/c59a9147", "Destination": "/dev/termination-log", "Mode": "", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "source-controller-cdb67965b-g9h2d", "Domainname": "", "User": "controller", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "RUNTIME_NAMESPACE=flux-system", "SOURCE_CONTROLLER_SERVICE_PORT=80", "KUBERNETES_SERVICE_PORT=443", "KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443", "NOTIFICATION_CONTROLLER_SERVICE_HOST=10.43.247.224", "NOTIFICATION_CONTROLLER_PORT_80_TCP_ADDR=10.43.247.224", "WEBHOOK_RECEIVER_SERVICE_HOST=10.43.220.10", "WEBHOOK_RECEIVER_PORT_80_TCP_ADDR=10.43.220.10", "SOURCE_CONTROLLER_PORT_80_TCP_PROTO=tcp", "NOTIFICATION_CONTROLLER_SERVICE_PORT_HTTP=80", "NOTIFICATION_CONTROLLER_PORT_80_TCP_PORT=80", "WEBHOOK_RECEIVER_SERVICE_PORT=80", "WEBHOOK_RECEIVER_PORT=tcp://10.43.220.10:80", "WEBHOOK_RECEIVER_PORT_80_TCP=tcp://10.43.220.10:80", "WEBHOOK_RECEIVER_PORT_80_TCP_PORT=80", "SOURCE_CONTROLLER_SERVICE_HOST=10.43.88.66", "SOURCE_CONTROLLER_SERVICE_PORT_HTTP=80", "SOURCE_CONTROLLER_PORT_80_TCP_ADDR=10.43.88.66", "WEBHOOK_RECEIVER_SERVICE_PORT_HTTP=80", "SOURCE_CONTROLLER_PORT=tcp://10.43.88.66:80", "SOURCE_CONTROLLER_PORT_80_TCP_PORT=80", "KUBERNETES_PORT_443_TCP_PROTO=tcp", "NOTIFICATION_CONTROLLER_PORT_80_TCP_PROTO=tcp", "KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1", "NOTIFICATION_CONTROLLER_SERVICE_PORT=80", "WEBHOOK_RECEIVER_PORT_80_TCP_PROTO=tcp", "SOURCE_CONTROLLER_PORT_80_TCP=tcp://10.43.88.66:80", "KUBERNETES_PORT=tcp://10.43.0.1:443", "KUBERNETES_PORT_443_TCP_PORT=443", "NOTIFICATION_CONTROLLER_PORT=tcp://10.43.247.224:80", "NOTIFICATION_CONTROLLER_PORT_80_TCP=tcp://10.43.247.224:80", "KUBERNETES_SERVICE_HOST=10.43.0.1", "KUBERNETES_SERVICE_PORT_HTTPS=443", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "--events-addr=http://notification-controller/", "--watch-all-namespaces=true", "--log-level=info", "--log-encoding=json", "--enable-leader-election", "--storage-path=/data", "--storage-adv-addr=source-controller.flux-system.svc.cluster.local." ], "Healthcheck": { "Test": [ "NONE" ] }, "Image": "sha256:dee0ecbd8fcd562f6a3e5fb8d42b3e608b638211d602eb5247f0936114b68079", "Volumes": null, "WorkingDir": "", "Entrypoint": [ "/sbin/tini", "--", "source-controller" ], "OnBuild": null, "Labels": { "annotation.io.kubernetes.container.hash": "ea032f41", "annotation.io.kubernetes.container.ports": "[{\"name\":\"http\",\"containerPort\":9090,\"protocol\":\"TCP\"},{\"name\":\"http-prom\",\"containerPort\":8080,\"protocol\":\"TCP\"},{\"name\":\"healthz\",\"containerPort\":9440,\"protocol\":\"TCP\"}]", "annotation.io.kubernetes.container.restartCount": "5", "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log", "annotation.io.kubernetes.container.terminationMessagePolicy": "File", "annotation.io.kubernetes.pod.terminationGracePeriod": "10", "io.kubernetes.container.logpath": "/var/log/pods/flux-system_source-controller-cdb67965b-g9h2d_0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4/manager/5.log", "io.kubernetes.container.name": "manager", "io.kubernetes.docker.type": "container", "io.kubernetes.pod.name": "source-controller-cdb67965b-g9h2d", "io.kubernetes.pod.namespace": "flux-system", "io.kubernetes.pod.uid": "0e0b51e8-61a5-4519-9fb9-8bbc0fe644b4", "io.kubernetes.sandbox.id": "d1a0f096d270786a3627db397c47eb9c0de6177b9d40fb6fb42ff7faeb6caa78", "org.opencontainers.image.created": "2021-06-09T13:26:16Z", "org.opencontainers.image.description": "The GitOps Toolkit source management component", "org.opencontainers.image.revision": "d0d2c0e21591d23d68a7a0a707ae323636e6fb5e", "org.opencontainers.image.source": "https://github.com/fluxcd/source-controller", "org.opencontainers.image.title": "source-controller", "org.opencontainers.image.url": "https://github.com/fluxcd/source-controller", "org.opencontainers.image.version": "v0.14.0" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": {} } } ]Errors from notification-controller. I think it is just the kustomization-controller that fails to download from source-controller because it is not running.
The text was updated successfully, but these errors were encountered: