Support affinities for scan jobs

[maxbrunet]

Hello! 👋

I would like to allow node-level jobs to run on any node with taints, except some. Currently, a config with the exhaustive list of acceptable taints is required, and every time a new taint is added to the cluster, it also needs to be added to the scan job tolerations:

nodeCollector.excludeNodes: type=virtual-kubelet
scanJob.tolerations: '[ many many many taints ]'

Ideally, I would like to tolerate all taints, but exclude some nodes with a node anti-affinity instead:

nodeCollector.excludeNodes: type=virtual-kubelet
scanJob.affinity: '{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"virtual-kubelet.io/provider","operator":"DoesNotExist"}]}]}}}'
scanJob.tolerations: '[{"operator":"Exists"}]'

A scanJob.affinity option would be nice for this purpose.

Please let me know if there is already a way to achieve that or if I am missing something.

[chen-keinan]

@maxbrunet sound a vital config for your use-case contribution are welcome if you have time.

[maxbrunet]

In the implementation should the Linux node-affinity just become the default value of scanJob.affinity which that means it could be overwritten? I guess this default would be set from the Helm chart? Or should a strategic merge happen with scanJob.affinity? The later might not be as obvious to the user about what the effects are since it involves lists

trivy-operator/pkg/trivyoperator/config.go

Lines 519 to 520 in 29b5c83

	// LinuxNodeAffinity constructs a new Affinity resource with linux supported nodes.
	func LinuxNodeAffinity() *corev1.Affinity {

I will see if I can find the time.