fix: make audit more fault tolerant, log error instead of skipping update

[ritazh]

Signed-off-by: Rita Zhang rita.z.zhang@gmail.com

What this PR does / why we need it:
make audit more fault tolerant, log error instead of skipping status update

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #2120

Special notes for your reviewer:

[codecov-commenter]

Codecov Report

Merging #2162 (3aa1b9d) into master (f5eacc7) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #2162      +/-   ##
==========================================
- Coverage   54.41%   54.38%   -0.04%     
==========================================
  Files         111      111              
  Lines        9530     9529       -1     
==========================================
- Hits         5186     5182       -4     
- Misses       3946     3949       +3     
  Partials      398      398              

Flag	Coverage Δ
unittests	54.38% <0.00%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/audit/manager.go	0.00% <0.00%> (ø)
...onstrainttemplate/constrainttemplate_controller.go	56.59% <0.00%> (-1.20%)	⬇️
pkg/readiness/object_tracker.go	82.91% <0.00%> (-1.07%)	⬇️
pkg/readiness/list.go	91.17% <0.00%> (+11.76%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5eacc7...3aa1b9d. Read the comment docs.

[maxsmythe]

LGTM

thanks for doing this!

[maxsmythe]

I wonder if we should also be working to make it easier to verify that audit is healthy. Maybe Prometheus metrics listing # of audit errors for the last audit run? Could be separate work.

[ritazh]

I wonder if we should also be working to make it easier to verify that audit is healthy. Maybe Prometheus metrics listing # of audit errors for the last audit run? Could be separate work.

I will open another issue/PR for this.