[operator] Ability to throttle scanning job creation

[consideRatio]

The starboard-operator will create all the pods it needs to scan all docker containers it finds, which can be so many that it ends up flooding the kubernetes cluster with pods.

This can lead to two issues:

1. Lots of scanner pods -> unwanted k8s cluster autoscaling for a non-timesensitive job makes additional nodes be provisioned instead of just taking it a bit slow
2. Lots of scanner pods -> no cache of the known vulnerability database among the pods -> lots of GitHub API requests -> 403 rate limit issues

I'd suggest to add a feature to the starboard-operator binary, to queue the creation of scanning jobs, and throttle the creation to be at most X at the time, where a sensible default would be three at the time or so.

I think by throttling the scanning operations, we end up avoiding both the issues listed above. But, it would still be relevant to cache the vulnerability database in case someone didn't want to throttle. I would suggest this to be done by exposing a service, perhaps built into the starboard-operator pod, to get the vulnerability database. The cache mechanism could be to download the latest version unless there is a cached version less than X minutes old available already or currently being downloaded.

[danielpacak]

This is great feedback @consideRatio We're aware of such limitations in v0.6.0 and we want to resolve them with highest priority in v0.7.0. I'll create follow up issues to solve scan job throttling and Trivy database caching. Please let me know if you'd like to work on one of those.

[danielpacak]

As mentioned above, I created these two issues to follow up:

• [operator] Limit the number of scan jobs run in parallel #202
• [operator] Design and implement Trivy DB caching #213