Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[specific ci=1-14-Docker-Kill] publish containerPoweredOff event after kill #6206

Closed
wants to merge 1 commit into from

Conversation

chengwang86
Copy link
Contributor

Fixes #5948

This PR should be able to at least alleviate the failure seen in our recent compost tests.

In the issue, when the VMPoweredOff event was to be processed, the container had already been removed from cache; therefore, we couldn't unbind the container from network. In this PR, we publish the containerPoweredOff event as soon as it is killed. This should guarantee that the container can be unbound from network most of the time. But it still does not fully resolve the race condition.

I think the race condition revealed by the compose test is quite general in our code and we need a comprehensive design for how to handle the async events, instead of changing our current logic for network rm and/or container rm as a fix.

@chengwang86 chengwang86 requested a review from hickeng September 5, 2017 15:02
@chengwang86 chengwang86 requested a review from hmahmood September 5, 2017 15:03
@chengwang86 chengwang86 changed the title publish containerPoweredOff event after kill [specific ci=1-14-Docker-Kill] publish containerPoweredOff event after kill Sep 5, 2017
return c.containerBase.kill(ctx)
err := c.containerBase.kill(ctx)
if err == nil {
publishContainerEvent(c.ExecConfig.ID, time.Now(), events.ContainerPoweredOff)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this result in the powered off event getting published twice in some cases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are three places where this ContainerPoweredOff event is used:

  1. https://github.com/vmware/vic/blob/master/lib/portlayer/network/network.go#L122
    If the container is already unbound by the previous ContainerPowerOff event, the second event would be a no-op.

  2. https://github.com/vmware/vic/blob/master/lib/portlayer/exec/container.go#L647
    This should not make any difference.

  3. https://github.com/vmware/vic/blob/master/lib/apiservers/engine/backends/eventmonitor.go#L230
    This would be a problem because docker event would show this event twice ... :(

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regardless of where the event is currently being consumed, this would potentially put any future uses of the event in jeopardy. It also is not correct to fire the event twice for the same occurrence. I would rather we not do this fix, especially since it also does not really address the problem.

@chengwang86 chengwang86 closed this Sep 5, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

docker-compose integration tests not cleaning up on failures
3 participants