Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ROX-27414: Prevent deduping Node Index in Central #13605

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

vikin91
Copy link
Contributor

@vikin91 vikin91 commented Dec 13, 2024

Description

Condition

If Central receives NodeInventory and NodeIndex (IndexReport) messages in parallel for the same node...

Observed behavior

...the message that arrives later will get deduped.

Why is this a problem?

This is bad for many reasons:

  • No ACK can be sent back to Compliance and it will retry forever
  • Switching from v2 to v4 node scanning without restarting Central may never work.

Moreover

This change puts all Node, NodeInventory, and IndexReport into single worker queue. This shall prevent race condition if multiple events of those three kinds arrive very close to each other. The code in the pipelines is not bullet-proof regarding tight race conditions (that was a decision when working on Node scanning v2), because making it so, would require adding more complexity to it. Those situations can be prevented by using single worker for those type of messages, as it guarantees no parallelism among the messages handled by the same worker.

User-facing documentation

  • CHANGELOG is updated OR update is not needed
  • documentation PR is created and is linked above OR is not needed

Testing and quality

  • the change is production ready: the change is GA or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests
  • added e2e tests
  • added regression tests
  • added compatibility tests
  • modified existing tests

How I validated my change

  • TODO: There are automated tests, but they do not cover this situation
  • One can verify that manually on a cluster where v2 and v4 node scanning runs in parallel - then both shall arrive to Central and Central shall drop one of them in the pipeline. If that change is not in place, there is a chance that the later message (usually v4 scan) will be deduped before being passed onto the respective pipieline.

Copy link

openshift-ci bot commented Dec 13, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@rhacs-bot
Copy link
Contributor

rhacs-bot commented Dec 13, 2024

Images are ready for the commit at fbdb6d2.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.7.x-294-gfbdb6d24b9.

@vikin91
Copy link
Contributor Author

vikin91 commented Dec 13, 2024

/test all

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants