chore: Add some empty dir volume mounts for the application controller (#19474)

[andrii-korotkov-verkada]

Closes #19474

Kube cache couldn't be used on read-only root file system leading to errors as revealed with --gloglevel equal to 6.
Create an empty dir mount for /tmp and add a config-map-based param to override KUBECACHEDIR.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell]

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment

[bunnyshell]

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (master@eb10b70). Learn more about missing BASE report.
Report is 26 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #19480   +/-   ##
=========================================
  Coverage          ?   55.11%           
=========================================
  Files             ?      324           
  Lines             ?    55201           
  Branches          ?        0           
=========================================
  Hits              ?    30426           
  Misses            ?    22159           
  Partials          ?     2616           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ChristianCiach]

It may be worth mentioning that, when using the "Security hardened images" by Akuity, the path would be /home/argocd/.kube/cache instead of /home/ubuntu/.kube/cache.

In fact, this PR would introduce the first and only mention of the word "ubuntu" in all of the installation manifests, and I don't know if that's desirable.

Maybe it would be better to add KUBECACHEDIR (see kubernetes/kubernetes#109479) and set it to /tmp/kubecache or something?

Edit: Actually, it's /home/argocd/.kube/cache for the official images (quay.io/argoproj/argocd), too.

[andrii-korotkov-verkada]

@ChristianCiach, thanks for letting me know. I've removed that mount and instead added a parameter in a config map to override the KUBECACHEDIR.

[ChristianCiach]

I would probably just put ENV KUBECACHEDIR=/tmp/kubecache into the Dockerfile, because I see no need to make this configurable, but don't let me influence you too much. Let's see what the maintainers think about this.

[andrii-korotkov-verkada]

I've updated the ENV variable to just point to /tmp/kubecache

[jannfis]

LGTM

[pasha-codefresh]

Thanks @jannfis @rumstead for reviewing it. It also LGTM for me . @andrii-korotkov-verkada sorry for delay in reviewing it

[andrii-korotkov-verkada]

No worries, I'm glad that it got attention.