Pods unable to be created on windows nodes #109

lizhuqi · 2021-04-26T18:23:40Z

It seems that Windows OS has some issue handling concurrent actions. Concurrent actions sometimes failed to start containers on Windows.

27m Warning FailedCreatePodSandBox pod/[POD_ONE] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "[POD_ONE]": Error response from daemon: hcsshim::PrepareLayer - failed failed in Win32: The device is not ready. (0x15)
24m Warning FailedMount pod/[POD_ONE] Unable to attach or mount volumes: unmounted volumes=[workspace-volume default-token-xyz12], unattached volumes=[workspace-volume default-token-xyz12l]: timed out waiting for the condition
30m Normal Scheduled pod/[POD_TWO] Successfully assigned cje/[POD_TWO] to [NODE_ID]
29m Warning FailedCreatePodSandBox pod/[POD_TWO] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "[POD_TWO]": Error response from daemon: hcsshim::PrepareLayer - failed failed in Win32: The device is not ready. (0x15)

vrapolinario · 2021-04-26T20:38:58Z

Hi @lizhuqi an you please provide more details on the issue you're finding? Such as: Is this a managed K8s (AKS, EKS)? What version of K8s? K8s node config, etc?

marosset · 2021-04-27T16:58:38Z

Can you include a few more details such as

Pod spec
number of containers starting concurrently
Node side (reserved)
Any other repro steps.
OS version

Thanks!

lizhuqi · 2021-04-27T22:45:31Z

I noticed that cluster which has the issue is still using 10.0.17763.1577. I will ask the owner to upgrade to 10.0.17763.1757 first and then add more details.

zhiweiv · 2021-05-13T01:26:16Z

This is also reported here microsoft/hcsshim#919.
VM spec is E2_V4(2 cores/16 GB) , we use bare pod as job workloads, there are about 20 concurrent pod creation/termination always.
The pod spec

    resources:
      requests:
        cpu: 50m
        memory: 500Mi
      limits:
        cpu: 1000m
        memory: 4Gi

immuzz · 2021-06-09T21:49:14Z

Is this on AKS or some other distro or DIY K8s?

zhiweiv · 2021-06-10T01:00:40Z

For me, it is reproduced in both AKS and AKS-Engine.

ghost · 2021-07-10T16:01:07Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

ghost · 2021-08-11T16:01:04Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

zhiweiv · 2021-08-13T06:32:13Z

There is a fix microsoft/hcsshim#1091 (comment).
Per microsoft/hcsshim#919 (comment), I guess still need a long time for it to be available in AKS.

AbelHu · 2021-08-16T02:23:18Z

@zhiweiv, I think that the fix needs to be picked up by moby at first and then we can upgrade the moby version in AKS side.
@immuzz do you know who can speed up taking the new hccshim release with this fix in next moby version?

zhiweiv · 2021-08-16T13:01:36Z

Hope it can be merged into AKS Windows ContainerD ASAP too. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Chou Hu ***@***.***> Sent: Monday, August 16, 2021 10:23:29 AM To: microsoft/Windows-Containers ***@***.***> Cc: Zhiwei Liu ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/Windows-Containers] Pods unable to be created on windows nodes (#109) @zhiweiv<https://github.com/zhiweiv>, I think that the fix needs to be picked up by moby at first and then we can upgrade the moby version in AKS side. @immuzz<https://github.com/immuzz> do you know who can speed up taking the new hccshim release with this fix in next moby version? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#109 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFH3LQPHTZH6GWJSLQOVGADT5BZCDANCNFSM43TNATIA>.

AbelHu · 2021-08-17T01:43:08Z

Hope it can be merged into AKS Windows ContainerD ASAP too. Get Outlook for iOShttps://aka.ms/o0ukef
…
________________________________ From: Chou Hu @.> Sent: Monday, August 16, 2021 10:23:29 AM To: microsoft/Windows-Containers @.> Cc: Zhiwei Liu @.>; Mention @.> Subject: Re: [microsoft/Windows-Containers] Pods unable to be created on windows nodes (#109) @zhiweiv https://github.com/zhiweiv, I think that the fix needs to be picked up by moby at first and then we can upgrade the moby version in AKS side. @immuzz https://github.com/immuzz do you know who can speed up taking the new hccshim release with this fix in next moby version? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#109 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFH3LQPHTZH6GWJSLQOVGADT5BZCDANCNFSM43TNATIA.

AKS will pick up it after the containerd 1.5.x has this fix.
kubernetes/kubernetes#101908 (comment)

AbelHu · 2021-08-18T06:38:20Z

@zhiweiv FYI: this fix will be backported to 1.4.x

zhiweiv · 2021-08-31T03:34:57Z

The fix is in latest hcsshim release: https://github.com/microsoft/hcsshim/releases/tag/v0.8.21, the next step is moby update? Anyone can push this further?

immuzz · 2021-09-01T00:11:22Z

I dont know if this will be fixed in Moby. Have you tried this with containerd?

zhiweiv · 2021-09-30T02:00:24Z

@AbelHu
The Containerd updated hcsshim in https://github.com/containerd/containerd/releases/tag/v1.4.10, any plan for AKS Windows Containerd update?

AbelHu · 2021-09-30T12:44:33Z

@kevpar could you help to confirm whether the hcsshim issue is fixed in the latest contianerd package (v0.0.42) used by AKS?

zhiweiv · 2021-10-01T01:30:13Z

@AbelHu
I often saw when you guys reference Windows Containerd, you use version 0.0.x, is it a internal build of Containerd in MS?

What is the difference with regular Containerd?

AbelHu · 2021-10-01T12:49:46Z

In AKS, Windows containerd is still in public review so we use an internal package.

AbelHu · 2021-10-01T12:50:46Z

I think that @immuzz should be able to answer more details for the difference.

zhiweiv · 2021-10-28T08:06:46Z

Windows Containerd on AKS is going GA per Azure/AKS#1976 (comment), any update on this?

AbelHu · 2021-10-28T08:30:03Z

@zhiweiv it should be coming soon. Current plan is next month.

ghost · 2021-11-27T16:01:09Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

zhiweiv · 2021-12-23T08:33:20Z

AKS Windows Containerd has been updated to 1.5.8.

ghost · 2022-01-22T16:01:20Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

ghost · 2022-02-22T16:01:17Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

cwilhit · 2022-03-07T21:10:52Z

@zhiweiv with containerd now GA in AKS, can you let me know if you are still seeing this issue? Else I will mark this issue as resolved.

zhiweiv · 2022-03-08T01:16:30Z

You can close this issue, it doesn't occur since containerd updated to 1.5.8.

ghost added the triage New and needs attention label Apr 26, 2021

vrapolinario assigned immuzz Apr 26, 2021

vrapolinario added Windows on Kubernetes Windows Containers using Kubernetes and removed triage New and needs attention labels Apr 26, 2021

cwilhit closed this as completed Mar 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pods unable to be created on windows nodes #109

Pods unable to be created on windows nodes #109

lizhuqi commented Apr 26, 2021

vrapolinario commented Apr 26, 2021

marosset commented Apr 27, 2021 •

edited

Loading

lizhuqi commented Apr 27, 2021

zhiweiv commented May 13, 2021 •

edited

Loading

immuzz commented Jun 9, 2021

zhiweiv commented Jun 10, 2021

ghost commented Jul 10, 2021

ghost commented Aug 11, 2021

zhiweiv commented Aug 13, 2021 •

edited

Loading

AbelHu commented Aug 16, 2021

zhiweiv commented Aug 16, 2021 via email

AbelHu commented Aug 17, 2021

AbelHu commented Aug 18, 2021

zhiweiv commented Aug 31, 2021

immuzz commented Sep 1, 2021

zhiweiv commented Sep 30, 2021 •

edited

Loading

AbelHu commented Sep 30, 2021

zhiweiv commented Oct 1, 2021

AbelHu commented Oct 1, 2021

AbelHu commented Oct 1, 2021

zhiweiv commented Oct 28, 2021 •

edited

Loading

AbelHu commented Oct 28, 2021

ghost commented Nov 27, 2021

zhiweiv commented Dec 23, 2021 •

edited

Loading

ghost commented Jan 22, 2022

ghost commented Feb 22, 2022

cwilhit commented Mar 7, 2022

zhiweiv commented Mar 8, 2022

Pods unable to be created on windows nodes #109

Pods unable to be created on windows nodes #109

Comments

lizhuqi commented Apr 26, 2021

vrapolinario commented Apr 26, 2021

marosset commented Apr 27, 2021 • edited Loading

lizhuqi commented Apr 27, 2021

zhiweiv commented May 13, 2021 • edited Loading

immuzz commented Jun 9, 2021

zhiweiv commented Jun 10, 2021

ghost commented Jul 10, 2021

ghost commented Aug 11, 2021

zhiweiv commented Aug 13, 2021 • edited Loading

AbelHu commented Aug 16, 2021

zhiweiv commented Aug 16, 2021 via email

AbelHu commented Aug 17, 2021

AbelHu commented Aug 18, 2021

zhiweiv commented Aug 31, 2021

immuzz commented Sep 1, 2021

zhiweiv commented Sep 30, 2021 • edited Loading

AbelHu commented Sep 30, 2021

zhiweiv commented Oct 1, 2021

AbelHu commented Oct 1, 2021

AbelHu commented Oct 1, 2021

zhiweiv commented Oct 28, 2021 • edited Loading

AbelHu commented Oct 28, 2021

ghost commented Nov 27, 2021

zhiweiv commented Dec 23, 2021 • edited Loading

ghost commented Jan 22, 2022

ghost commented Feb 22, 2022

cwilhit commented Mar 7, 2022

zhiweiv commented Mar 8, 2022

marosset commented Apr 27, 2021 •

edited

Loading

zhiweiv commented May 13, 2021 •

edited

Loading

zhiweiv commented Aug 13, 2021 •

edited

Loading

zhiweiv commented Sep 30, 2021 •

edited

Loading

zhiweiv commented Oct 28, 2021 •

edited

Loading

zhiweiv commented Dec 23, 2021 •

edited

Loading