[agent-smith] improve performance

[kylos101]

Is your feature request related to a problem? Please describe

The # of cores used is often high and we believe this might be one of the contributing causes for disconnects in the XL clusters. Not as high as ws-daemon, but is higher than registry-facade. In other words, it is taking CPU that workspaces could be using. Why? Please profile agent-smith, and then check to see if there are opportunities to optimize.

Describe the behaviour you'd like

Max core usage is consistently less. Perhaps less than 2.5?

Describe alternatives you've considered

Less average and max memory used would be ideal, too.

Additional context

Tuning done to improve ws-daemon performance. [1][2]

[atduarte]

Updated the description to reflect what triggered us to work on this: "we believe this might be one of the contributing causes for disconnects in the XL clusters".

Agent-smith has a lot more work to do on Standard nodes as these have more workspaces running. When looking at the XL clusters at the times of the issues, I don't see high CPU usage: https://grafana.gitpod.io/d/agent-smith/gitpod-component-agent-smith?orgId=1&from=1653552000000&to=1653573600000&var-cluster=us44xl&var-node=All&var-pod=All&var-datasource=VictoriaMetrics&viewPanel=2

[kylos101]

Resolved in #10356