tests: execute some scale tests using hirte and qm

[dougsland]

Please describe what information you were trying to find

In my journey of testing hirte, I will execute some scale tests and document how far I did.
I will use this issue for such task and later close it.

[dougsland]

Just copy and paste some results from the initial stress tests.

50 nodes and 50 QM nested containers stress test summary

[ INFO  ] Summary of tests
[ INFO  ] ==============================
[ INFO  ] ASIL environment
[ INFO  ] - 1 Controller container with:
[ INFO  ]       - hirte service: manager and agent
[ INFO  ]       - 4 systemd generated and running:
[ INFO  ]                container-safety
[ INFO  ]                container-cruise_control
[ INFO  ]                container-tires
[ INFO  ]                container-breaks

[ INFO  ] QM environment
[ INFO  ]   Nodes:
[ INFO  ]       - 50 node(s) running as containers
[ INFO  ]       - hirte service in each node: agent
[ INFO  ]       - 1 nested container running inside each node listed above

[ INFO  ]   QM Nodes (nested container):
[ INFO  ]       - 50 QM node(s) running inside in the nodes
[ INFO  ]       - hirte service in each qm node: agent
[ INFO  ]       - 4 systemd generated and running in each QM node:
[ INFO  ]                container-radio
[ INFO  ]                container-store
[ INFO  ]                container-stream_audio
[ INFO  ]                container-maps

[ INFO  ] hirtectl calls
[ INFO  ]       - 101 hirtectl calls to nodes

[ INFO  ] Running time for this script
[ INFO  ]       - 45 minutes and 28 seconds

[ INFO  ] All set!

150 nodes and 150 QM nested containers stress test summary

[ INFO  ] Summary of tests
[ INFO  ] ==============================
[ INFO  ] ASIL environment
[ INFO  ] - 1 Controller container with:
[ INFO  ]       - hirte service: manager and agent
[ INFO  ]       - 4 systemd generated and running:
[ INFO  ]                container-safety
[ INFO  ]                container-cruise_control
[ INFO  ]                container-tires
[ INFO  ]                container-breaks

[ INFO  ] QM environment
[ INFO  ]   Nodes:
[ INFO  ]       - 150 node(s) running as containers
[ INFO  ]       - hirte service in each node: agent
[ INFO  ]       - 1 nested container running inside each node listed above

[ INFO  ]   QM Nodes (nested container):
[ INFO  ]       - 150 QM node(s) running inside in the nodes
[ INFO  ]       - hirte service in each qm node: agent
[ INFO  ]       - 4 systemd generated and running in each QM node:
[ INFO  ]                container-radio
[ INFO  ]                container-store
[ INFO  ]                container-stream_audio
[ INFO  ]                container-maps

[ INFO  ] hirtectl calls
[ INFO  ]       - 301 hirtectl calls to nodes

[ INFO  ] Running time for this script
[ INFO  ]       - 150 minutes and 51 seconds

[ INFO  ] All set!

[dougsland]

Full output for 150-nodes.txt

[rhatdan]

Have you found any failures based on this testing?

[dougsland]

Have you found any failures based on this testing?

No, so far, so good.

[dougsland]

All good with 300 nodes and 300 qm nodes running as nested container.

300 nodes stress test summary

[ INFO  ] Summary of tests
[ INFO  ] ==============================
[ INFO  ] ASIL environment
[ INFO  ] - 1 Controller container with:
[ INFO  ] 	- hirte service: manager and agent
[ INFO  ] 	- 4 systemd generated and running:
[ INFO  ] 		 container-safety
[ INFO  ] 		 container-cruise_control
[ INFO  ] 		 container-tires
[ INFO  ] 		 container-breaks

[ INFO  ] QM environment
[ INFO  ]   Nodes:
[ INFO  ] 	- 300 node(s) running as containers
[ INFO  ] 	- hirte service in each node: agent
[ INFO  ] 	- 1 nested container running inside each node listed above

[ INFO  ]   QM Nodes (nested container):
[ INFO  ] 	- 300 QM node(s) running inside in the nodes
[ INFO  ] 	- hirte service in each qm node: agent
[ INFO  ] 	- 4 systemd generated and running in each QM node:
[ INFO  ] 		 container-radio
[ INFO  ] 		 container-store
[ INFO  ] 		 container-stream_audio
[ INFO  ] 		 container-maps

[ INFO  ] hirtectl calls
[ INFO  ] 	- 601 hirtectl calls to nodes

[ INFO  ] Running time for this script
[ INFO  ] 	- 375 minutes and 23 seconds

[ INFO  ] All set!

Full output 300-nodes.txt

I will execute few more tests and we are good to close this issue. Hopefully, no issues will come with such stress tests when we keep increasing the bar.

[rhatdan]

Nice work.

[dougsland]

Ok, just a feedback, 1000 nodes + 1000 qm nodes seems to work just nice. I am closing now this one now.
We are going to improve the tests to make it more faster and in the future we can give another try in such stress test.

/cc @Yarboa

[dougsland]

Steps to execute the tests for future reference:

git clone https://github.com/containers/qm && cd qm/tests/e2e
./tools/remove-containers (remove any previous created container/image, helps not count the time consumed to remove old environment)
./run-test-e2e --number-of-nodes=500 &> output-500-nodes.txt

[dougsland]

For the next tests let's make sure we address this QM issue first: containers/qm#164