api,agent,server,engine-schema: scalability improvements

[shwstppr]

Description

Following changes and improvements have been added:

• Improvements in handling of PingRoutingCommand

  1. Added global config - vm.sync.power.state.transitioning, default value: true, to control syncing of power states for transitioning VMs. This can be set to false to prevent computation of transitioning state VMs.
  2. Improved VirtualMachinePowerStateSync to allow power state sync for host VMs in a batch
  3. Optimized scanning stalled VMs

• Added option to set worker threads for capacity calculation using config - capacity.calculate.workers

• Added caching for account/use role API access with expiration after write can be configured using config - dynamic.apichecker.cache.period. If set to zero then there will be no caching. Default is 0.

• Added caching for account/use role API access with expiration after write set to 60 seconds.

• Added caching for some recurring DB retrievals

  1. CapacityManager - listing service offerings - beneficial in host capacity calculation
  2. LibvirtServerDiscoverer existing host for the cluster - beneficial for host joins
  3. DownloadListener - hypervisors for zone - beneficial for host joins
  4. VirtualMachineManagerImpl - VMs in progress- beneficial for processing stalled VMs during PingRoutingCommands

• Optimized MS list retrieval for agent connect

• Optimize finding ready systemvm template for zone

• Database retrieval optimisations - fix and refactor for cases where only IDs or counts are used mainly for hosts and other infra entities. Also similar cases for VMs and other entities related to host concerning background tasks

• Changes in agent-agentmanager connection with NIO client-server classes

  1. Optimized the use of the executor service
  2. Refactore Agent class to better handle connections.
  3. Do SSL handshakes within worker threads
  4. Added global configs to control the behaviour depending on the infra. SSL handshake could be a bottleneck during agent connections. Configs - agent.ssl.handshake.min.workers and agent.ssl.handshake.max.workers can be used to control number of new connections management server handles at a time. agent.ssl.handshake.timeout can be used to set number of seconds after which SSL handshake times out at MS end.
  5. On agent side backoff and sslhandshake timeout can be controlled by agent properties. backoff.seconds and ssl.handshake.timeout properties can be used.

• Improvements in StatsCollection - minimize DB retrievals.

• Improvements in DeploymentPlanner allow for the retrieval of only desired host fields and fewer retrievals.

• Improvements in hosts connection for a storage pool. Added config - storage.pool.host.connect.workers to control the number of worker threads that can be used to connect hosts to a storage pool. Worker thread approach is followed currently only for NFS and ScaleIO pools.

• Minor improvements in resource limit calculations wrt DB retrievals

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

Attention: Patch coverage is 25.38717% with 1349 lines in your changes missing coverage. Please review.

Project coverage is 15.86%. Comparing base (32af4a2) to head (91a71a6).

Files with missing lines	Patch %	Lines
agent/src/main/java/com/cloud/agent/Agent.java	19.88%	273 Missing and 9 partials ⚠️
.../main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java	11.56%	127 Missing and 3 partials ⚠️
.../src/main/java/com/cloud/host/dao/HostDaoImpl.java	50.94%	64 Missing and 14 partials ⚠️
.../apache/cloudstack/metrics/MetricsServiceImpl.java	0.00%	73 Missing and 1 partial ⚠️
...com/cloud/vm/VirtualMachinePowerStateSyncImpl.java	0.00%	71 Missing ⚠️
...n/java/com/cloud/capacity/CapacityManagerImpl.java	20.68%	69 Missing ⚠️
...n/java/com/cloud/vm/VirtualMachineManagerImpl.java	0.00%	60 Missing ⚠️
...src/main/java/com/cloud/server/StatsCollector.java	0.00%	47 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	30.00%	39 Missing and 3 partials ⚠️
...c/main/java/com/cloud/utils/nio/NioConnection.java	53.62%	28 Missing and 4 partials ⚠️
... and 54 more

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20    #9840      +/-   ##
============================================
- Coverage     16.04%   15.86%   -0.18%     
- Complexity    12835    12890      +55     
============================================
  Files          5637     5641       +4     
  Lines        493651   494249     +598     
  Branches      59874    59890      +16     
============================================
- Hits          79205    78412     -793     
- Misses       405648   407047    +1399     
+ Partials       8798     8790       -8     

Flag	Coverage Δ
uitests	4.02% <ø> (ø)
unittests	16.69% <25.38%> (-0.20%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[GutoVeronezi]

Honestly, I don't like PRs with thousand of lines doing thousand of things. It is hard to review and test. I encourage you to separate it in several minor PRs that address each one of the changes you are proposing.

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11441

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11486

[shwstppr]

@blueorangutan test

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-11738)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 71834 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9840-t11738-kvm-ol8.zip
Smoke tests completed. 134 look OK, 6 have errors, 1 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_host_tags	Failure	10.40	test_host_tags.py
test_03_secured_to_nonsecured_vm_migration	Error	1286.75	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	Error	1286.76	test_vm_life_cycle.py
ContextSuite context=TestSecuredVmMigration>:teardown	Error	2253.98	test_vm_life_cycle.py
ContextSuite context=TestVAppsVM>:setup	Error	2409.13	test_vm_life_cycle.py
ContextSuite context=TestVMLifeCycle>:setup	Error	2499.28	test_vm_life_cycle.py
ContextSuite context=TestVMSchedule>:setup	Error	0.00	test_vm_schedule.py
test_hostha_enable_ha_when_host_in_maintenance	Error	2101.84	test_hostha_kvm.py
test_01_migrate_vm_strict_tags_success	Error	99.22	test_vm_strict_host_tags.py
test_01_redundant_vpc_site2site_vpn	Failure	455.29	test_vpc_vpn.py
all_test_vm_lifecycle_unmanage_import	Skipped	---	test_vm_lifecycle_unmanage_import.py

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 11708

[BryanMLima]

I couldn't review thoroughly as there are a lot of changes, but I left some comments around the code. Overall, it looks good, however, as @GutoVeronezi mentioned, I think this PR would benefit being break into smaller ones, specially as not all changes are intertwined.

It will probably need a lot of testing for this much change, as to make sure no regression is introduced (that's why I think smaller PRs would make this easier to test and approve).

[shwstppr]

Thanks @BryanMLima for the review. I've addressed your suggestions and responded to two queries.
In terms of testing, yes we are already testing this exhaustively with @vladimirpetrov to prevent any regressions.

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11887

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11920

[shwstppr]

@blueorangutan test matrix

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian Build Failed (tid-11992)