Starboard Not creating job for vulnerability scanning

[deegee-dev]

Hi everyone, I installed Starboard Operator (v0.10.1 using helm) on Openshift OKD 4.7.5. and would appreciate any assistance.

When I create a new deployment on the monitored namespace, Starboard-operator automatically creates the scan-configauditreport job but NOT the scan-vulnerabilityreport job. There were no errors from the logs that could result in the job not created for the expected scan-vulnerabilityreport job.

I have checked the configmaps and have tried both trivy mode (ClientServer and Standalone), verified that

Configmap

vulnerabilityReports.scanner = trivy

Pod environment variable

OPERATOR_VULNERABILITY_SCANNER_ENABLED = True

I'm not sure what was not configured properly but Kube-bench and Polaris ran as expected.

Thanks !

[danielpacak]

We had a similar problem with configuration checker jobs when we explicitly set SecurityContext with UID and GID.

Notice that vulnerability scan jobs are run with the starboard-operator service account, with UID equal to 1000, and GID equal to 1000. See https://github.com/aquasecurity/starboard/blob/main/pkg/plugin/trivy/plugin.go#L335

For OpenShift Container Platform usually the admission controller assigns UID and GID for a Pod based on the range defined for the OpenShift project / K8s namespace in which it's running. So the solution might be to not set UID and GID at all in https://github.com/aquasecurity/starboard/blob/main/pkg/plugin/trivy/plugin.go#L335 Alternatively, you could define a custom SCC and add the starboard-operator service account to it.

To confirm that, could you check events in the Starboard operator's namespace? If that's the problem with Security Context Constraints you should see errors saying that UID 1000 is not in the allowed range.

kubectl get events -n <starboard operator namespace>

[deegee-dev]

Thank you for your reply, Security context constrain is likely not the cause. I have encountered the issue you have described on the configuration checker job and resolved it prior to encountering this vulnerability scan job issue with

oc adm policy add-scc-to-user privileged -z default -n starboard-operator

I've checked my event logs on Starboard operator namespace and but it is no different from the picture I've attached

1. Assignment to Starboard-operator,
2. Pod creation for configuration checker job,
3. Created Polaris container,
4. Started Polaris Container,
5. Stopping Polaris container.

There are no events regarding vulnerability scanning, not even assignment of vulnerability job.

Hope this clarifies

[deegee-dev]

On the logs I find there are "Reconciler error" due to "unexpected end of JSON input" on introducing new workloads. This is the error after adding a deployment nginx 1.16 on "default" namespace. Configuration auditing proceeded but not vulnerability scanning. These errors were not reflected on the events. I'm not sure if these would impact vulnerability scanning

{
   "level":"error",
   "ts":1620737456.8141055,
   "logger":"controller-runtime.manager.controller.replicaset",
   "msg":"Reconciler error",
   "reconciler group":"apps",
   "reconciler kind":"ReplicaSet",
   "name":"nginx-6d4cf56db6",
   "namespace":"default",
   "error":"unexpected end of JSON input",
   "stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:267\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.2/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99"
}

[danielpacak]

Hey @deegee-dev I was able to reproduce that issue. The problem is that apparently OKD creates the image pull secret with empty data and attaches it to the starboard-operator service account. Starboard does not handle it properly and fails. I created #569 to fix it and follow up.

[danielpacak]

@deegee-dev This has been fixed in v0.10.3 and verified on OKD 4.7. Let us know if it works on your end as well.

[deegee-dev]

Thanks @danielpacak
It works now, tested it with OKD version 4.7 and openshift 4.7.8
Appreciate your effort :)