Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ctrd: try to fix fail to destroy container #2661

Merged
merged 1 commit into from
Jan 16, 2019
Merged

ctrd: try to fix fail to destroy container #2661

merged 1 commit into from
Jan 16, 2019

Conversation

Ace-Tang
Copy link
Contributor

@Ace-Tang Ace-Tang commented Jan 11, 2019

we got error in stop pouch container

the task has quit, id: 6a8dd031f5c87b4c67813908bbe7d3b3d96239e0054dd2516b411c88f7aa681f, err: <nil>, exitcode: 137, time: 2019-01-02 14:18:19.201110172 +0000 UTC"
failed to delete task, container id: 6a8dd031f5c87b4c67813908bbe7d3b3d96239e0054dd2516b411c88f7aa681f: task must be stopped before deletion: running: failed precondition
failed to delete container, container id: 6a8dd031f5c87b4c67813908bbe7d3b3d96239e0054dd2516b411c88f7aa681f: cannot delete running task 6a8dd031f5c87b4c67813908bbe7d3b3d96239e0054dd2516b411c88f7aa681f: failed precondition"

For checking containerd code, the problem probably happend maybe:
when called delete task, first get task status, taskService().Get()
-> processFromContainerd -> shim.State() -> init.Status() -> p.runtime.State()
containerd-shim not use init proc status as container status, they
should always keep same, but obviously, here is not.

Signed-off-by: Ace-Tang aceapril@126.com

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗?)

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

@codecov
Copy link

codecov bot commented Jan 11, 2019

Codecov Report

Merging #2661 into master will decrease coverage by 3.53%.
The diff coverage is 40%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2661      +/-   ##
==========================================
- Coverage   69.72%   66.18%   -3.54%     
==========================================
  Files         281      283       +2     
  Lines       18699    18904     +205     
==========================================
- Hits        13037    12512     -525     
- Misses       4213     5049     +836     
+ Partials     1449     1343     -106
Flag Coverage Δ
#criv1alpha1test 24.25% <40%> (-7.48%) ⬇️
#criv1alpha2test 35.74% <40%> (-0.35%) ⬇️
#integrationtest 41.82% <0%> (-0.32%) ⬇️
#nodee2etest 32.87% <40%> (+0.33%) ⬆️
#unittest 26.95% <0%> (-0.28%) ⬇️
Impacted Files Coverage Δ
ctrd/container.go 59.01% <40%> (+0.52%) ⬆️
cri/v1alpha1/service/cri.go 0% <0%> (-84.62%) ⬇️
cri/v1alpha1/cri.go 0% <0%> (-59.44%) ⬇️
cri/v1alpha1/cri_wrapper.go 0% <0%> (-55.31%) ⬇️
cri/v1alpha1/cri_utils.go 60.37% <0%> (-23.16%) ⬇️
cri/criservice.go 34.93% <0%> (-21.69%) ⬇️
cri/stream/portforward/httpstream.go 70.94% <0%> (-6.84%) ⬇️
cri/stream/runtime.go 67.85% <0%> (-2.39%) ⬇️
daemon/mgr/spec_linux.go 76.74% <0%> (-2%) ⬇️
cri/v1alpha2/cri.go 67.57% <0%> (-1.96%) ⬇️
... and 22 more

@Ace-Tang Ace-Tang requested a review from rudyfly January 11, 2019 06:25
@@ -396,7 +396,16 @@ func (c *Client) destroyContainer(ctx context.Context, id string, timeout int64)
clean:
if err := pack.container.Delete(ctx); err != nil {
if !errdefs.IsNotFound(err) {
return msg, errors.Wrap(err, "failed to delete container")
// try to delete task again
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should check any others to call container.Delete

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think no need, check the error message I write in the commit, for normal remove process, code under clean will not take effect, only deal with some error case, and not always have effect.

we got error in stop pouch container
```
the task has quit, id: 6a8dd031f5c87b4c67813908bbe7d3b3d96239e0054dd2516b411c88f7aa681f, err: <nil>, exitcode: 137, time: 2019-01-02 14:18:19.201110172 +0000 UTC"
failed to delete task, container id: 6a8dd031f5c87b4c67813908bbe7d3b3d96239e0054dd2516b411c88f7aa681f: task must be stopped before deletion: running: failed precondition
failed to delete container, container id: 6a8dd031f5c87b4c67813908bbe7d3b3d96239e0054dd2516b411c88f7aa681f: cannot delete running task 6a8dd031f5c87b4c67813908bbe7d3b3d96239e0054dd2516b411c88f7aa681f: failed precondition"
```

For checking containerd code, the problem probably happend maybe:
when called delete task, first get task status, taskService().Get()
-> processFromContainerd -> shim.State() -> init.Status() -> p.runtime.State()
containerd-shim not use init proc status as container status, they
should always keep same, but obviously, here is not.

Signed-off-by: Ace-Tang <aceapril@126.com>
@pouchrobot pouchrobot added size/XS and removed size/S labels Jan 15, 2019
@rudyfly
Copy link
Collaborator

rudyfly commented Jan 16, 2019

LGTM

@pouchrobot pouchrobot added the LGTM one maintainer or community participant agrees to merge the pull reuqest. label Jan 16, 2019
@rudyfly rudyfly merged commit fa8101e into AliyunContainerService:master Jan 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
containerd-related LGTM one maintainer or community participant agrees to merge the pull reuqest. size/XS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants