Ensure Akri's Helm templates use the most restrictive settings suggested by Snyk report

[harrison-tin]

What this PR does / why we need it:
This PR explicitly sets more SecurityContext rather than using default in the helm templates, using Akri's Snyk report for guidance. The templates changed are:

1. agent: added more securityContext
2. controller: added more securityContext
3. anomaly-detection-app: added more securityContext and resource limits
4. video-streaming-app: added more securityContext and resource limits

Special notes for your reviewer:

If applicable:

• this PR has an associated PR with documentation in akri-docs
• this PR contains unit tests
• added code adheres to standard Rust formatting (cargo fmt)
• code builds properly (cargo build)
• code is free of common mistakes (cargo clippy)
• all Akri tests succeed (cargo test)
• inline documentation builds (cargo doc)
• all commits pass the DCO bot check by being signed off -- see the failing DCO check for instructions on how to retroactively sign commits

[kate-goldenring]

I think we need to open up the Agent as it needs privileged access. This requires running this workflow locally and inspecting kubelet logs for errors journalctl -fu kubelet

[jbpaux]

Tests are failing because tests are writing a file in the agent pod:

akri/test/run-end-to-end.py

Line 106 in 47b4258

os.system('sudo {} exec -i {} -- /bin/sh -c "echo "OFFLINE" > /tmp/debug-echo-availability.txt"'.format(kubectl_cmd, shared_test_code.agent_pod_name))

[kate-goldenring]

Good point! Maybe we add that file as a volume mount for the Pod during the test

[harrison-tin]

updated the securityContext and removed readOnlyRootFilesystem

[kate-goldenring]

I am not sure i understand how this was resolved. Looks like readOnlyRootFilesystem: true still exists which is ideal.

[harrison-tin]

I think the above highlight is outdated? Here shows the updated security context for akri agent. I removed setting readOnlyRootFilesystem to true (followed what nvidia gpu device plugin did)

[kate-goldenring]

Thanks for putting this together! We may want to do a round two at some point with the discovery handlers. If you pull in the latest from main, tests should pass. We may need to bump version depending on whether this goes in before #556