Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plugins/logs: fixed reconfiguration race condition #2948

Merged

Conversation

kubaj
Copy link
Contributor

@kubaj kubaj commented Nov 25, 2020

Bug summary

Logging plugin had race condition while it was masking the log and plugin was reconfigured at the same time. This caused panic of the OPA. Fixed issue for us and also may fix #2835 .

Details

Masking part of the logging plugin had incorrectly placed mutex -- only
for writing the variable p.mask, but not for reading it. This caused
race condition in the situation when bundle update calls plugin.Reconfigure()
method between setting the p.mask variable and then evaluating it.
Reconfiguration of the plugin sets p.mask to nil and therefore calling
p.mask.Eval() results in panic.

Tests covering functionality shouldn't change, test covering the fixed issue is not delivered, because it's difficult to write a test simulating the race condition (if you know about some efficient way, feel free to guide me, I'll happily learn).

Copy link
Member

@tsandall tsandall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find! See my comment about only grabbing the mask while holding the mutex.

plugins/logs/plugin.go Outdated Show resolved Hide resolved
@kubaj kubaj force-pushed the fix-log-plugin-race-condition branch from fb16bb6 to 551bb2a Compare November 26, 2020 09:02
@kubaj
Copy link
Contributor Author

kubaj commented Nov 26, 2020

I updated the PR.

srenatus
srenatus previously approved these changes Nov 27, 2020
Copy link
Contributor

@srenatus srenatus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏 Good find, thank you!

@kubaj
Copy link
Contributor Author

kubaj commented Nov 27, 2020

@srenatus, I'm glad I could help. I squashed commits according to instructions, can you reapprove please so I can merge?

@kubaj kubaj requested a review from srenatus December 1, 2020 15:14
tsandall
tsandall previously approved these changes Dec 1, 2020
Copy link
Member

@tsandall tsandall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ashutosh-narkar
Copy link
Member

@kubaj Can you please squash the commits and update the commit message ? The sign-off line is duplicated.

@kubaj
Copy link
Contributor Author

kubaj commented Dec 2, 2020

@ashutosh-narkar I updated the commit message to remove duplicate sign-off. I hope it's all you need.

@ashutosh-narkar
Copy link
Member

@kubaj thanks for updating your commit message. I think you'll need to rebase from master and then push your single commit to your fork as currently it's showing that 11 commits need to be merged.

Masking part of the logging plugin had incorrectly placed mutex -- only
for writing the variable p.mask, but not for reading it. This caused
race condition in the situation when bundle update calls plugin.Reconfigure()
method between setting the p.mask variable and then evaluating it.
Reconfiguration of the plugin sets p.mask to nil and therefore calling
p.mask.Eval() results in panic.

Signed-off-by: Jakub Kulich <jakub.kulich@exponea.com>
@kubaj kubaj force-pushed the fix-log-plugin-race-condition branch from 3cfa8d8 to a8fd154 Compare December 3, 2020 08:10
@kubaj
Copy link
Contributor Author

kubaj commented Dec 3, 2020

@ashutosh-narkar sorry about that. I fixed that, it should be ok now.

Copy link
Contributor

@srenatus srenatus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! 🎉

@tsandall tsandall merged commit 9dbdcc9 into open-policy-agent:master Dec 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

logging panic under load test with unix domain socket
4 participants