docker: Detect potential DiskPressure issue (docker system prune) #6820
Assignees
Labels
co/docker-driver
Issues related to kubernetes in container
co/docker-env
docker-env issues
co/runtime/docker
Issues specific to a docker runtime
kind/feature
Categorizes issue or PR as related to a new feature.
priority/important-soon
Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
ux/embedded
Embedded UX blockers
Milestone
Comments
balopat
added
co/runtime/docker
|
Additional info: Did not help. same state, In fact, restarting caused some issue with the IP (minikube docker-env command) that was working before: Maybe this is worth to open a separate bug for. |
|
|
|
If you or anyone else sees this, I would love to see the output of:
That should help us determine which taint it is. I suspect one of:
|
tstromberg
added
the
triage/needs-information
|
yupp it is disk-pressure seemingly: |
|
|
|
balopat
changed the title
|
Just hit the same thing (I think also due to disk pressure), attaching my logs: |
|
|
tstromberg
changed the title
tstromberg
added
priority/important-soon
tstromberg
changed the title
This was referenced Apr 16, 2020
|
done |
|
I too ran into this issue. Resetting the kubernetes clusters or restarting the system did not work for me. But |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I was running a docker driver / docker runtime node for a couple of days
Node details:
{ "Name": "docker3", "Status": "Running", "Config": { "Name": "docker3", "KeepContext": false, "EmbedCerts": false, "MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v1.7.3.iso", "Memory": 2000, "CPUs": 2, "DiskSize": 20000, "Driver": "docker", "HyperkitVpnKitSock": "", "HyperkitVSockPorts": [], "DockerEnv": null, "InsecureRegistry": null, "RegistryMirror": null, "HostOnlyCIDR": "192.168.99.1/24", "HypervVirtualSwitch": "", "HypervUseExternalSwitch": false, "HypervExternalAdapter": "", "KVMNetwork": "default", "KVMQemuURI": "qemu:///system", "KVMGPU": false, "KVMHidden": false, "DockerOpt": null, "DisableDriverMounts": false, "NFSShare": [], "NFSSharesRoot": "/nfsshares", "UUID": "", "NoVTXCheck": false, "DNSProxy": false, "HostDNSResolver": true, "HostOnlyNicType": "virtio", "NatNicType": "virtio", "KubernetesConfig": { "KubernetesVersion": "v1.17.3", "ClusterName": "docker3", "APIServerName": "minikubeCA", "APIServerNames": null, "APIServerIPs": null, "DNSDomain": "cluster.local", "ContainerRuntime": "docker", "CRISocket": "", "NetworkPlugin": "", "FeatureGates": "", "ServiceCIDR": "10.96.0.0/12", "ImageRepository": "", "ExtraOptions": [ { "Component": "kubeadm", "Key": "pod-network-cidr", "Value": "10.244.0.0/16" } ], "ShouldLoadCachedImages": true, "EnableDefaultCNI": false }, "Nodes": [ { "Name": "docker3", "IP": "172.17.0.3", "Port": 8443, "KubernetesVersion": "v1.17.3", "ControlPlane": true, "Worker": true } ], "Addons": { "default-storageclass": true, "storage-provisioner": true } } }And somehow I got into this state where an offending taint appeared somehow...that doesn't let me deploy anything to the cluster. The pods are
PENDINGstate and the pod events show the taint:I'm not confident that I can reproduce this and haven't put the effort to repro it yet either. But it seems like a provisioning bug somewhere that we could maybe spot by inspection?
The text was updated successfully, but these errors were encountered: