Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose of PR?:
Implement NRI support in KubeArmor and mostly fix container confusion issues in BPF-LSM caused by reused Linux namespace IDs.
Does this PR introduce a breaking change? No
If the changes in this PR are manually verified, list down the scenarios covered: Verified functionality in my own sandbox environment.
Additional information for reviewer? :
This NRI solution isn't perfect, but it's much better than the existing containerd and CRI-O solutions at preventing false positives.
For whatever reason, Linux can reuse namespace IDs between containers so long as one container exits before the next one starts. Those namespace IDs are freed as soon as the container's root process exits, and they get created during the
runc
container initialization process. This means that if KubeArmor's enforcement ends too late for a given container, it could be incorrectly enforcing behaviors on a new container as if they were happening on the old container, including therunc init
process to set up a container! On a default "Audit" posture, this is sorta ok, it will just produce false positive alerts, but on a default "Block" posture this will block the new container from being created.This NRI solution mostly solves these namespace ID overlap issues by starting enforcement on a container after it's already started, and stopping enforcement on a container just before it stops. The only case I can come up with where this doesn't work is if a container crashes unexpectedly, in which case this NRI solution may produce false positive alerts and/or blocks, but it won't be any worse than the existing containerd and CRI-O solutions. That said, I've been running very similar NRI code in my fork of KubeArmor in production for over a month, and I haven't seen any namespace ID overlap issues and I can't cause it to happen synthetically either.
This solution does make one major tradeoff though, which is that there's a very short period of time after a container starts as well as however long it takes for the container to gracefully exit during termination where the container's actions won't be enforced. The existing containerd and CRI-O solutions don't enforce the very start of a container either, and since NRI is event-based it should both react faster on average and use less resources than the current solutions (we did see a significant decrease in CPU after switching from containerd to NRI in my environment).
Checklist:
<type>(<scope>): <subject>
Also, a quick note about using NRI:
As of writing this, the NRI API is currently in draft status, and while both containerd and CRI-O support NRI, it's not enabled by default in either runtime.
In my case, my company uses EKS with self-managed node groups (so we can tweak configuration as needed) and to enable NRI in the EKS nodes running containerd, we had to both update to the latest AMI and also tweak the user data of the EC2 instances to be the following script:
Hopefully NRI is stabilized and enabled by default in containerd and CRI-O soon, but until then I hope this extra information will help anyone else who needs to use NRI.