Add new integration tests #267

ChrisKujawa · 2022-12-05T14:47:19Z

Add some unit tests
Add testcontainer dependency
Add new testcontainer integration tests
- First tests verifies that we can deploy our chaos models
- Second tests verifies the complete round trip, deploy models, create instance and run the workers and the related actions

In order to run the integration tests I created an "fake" experiment which just runs zbchaos version but this allows to verify whether the readExperiments worker works AND the execution of the zbchaos subcommand via the experiment definition. ✅

It seems to work 🚀


Deploy file bpmn/chaos/actionRunner.bpmn (size: 8788 bytes).
Deployed process model bpmn/chaos/actionRunner.bpmn successful with key 2251799813685249.
Deploy file bpmn/chaos/chaosExperiment.bpmn (size: 21403 bytes).
Deployed process model bpmn/chaos/chaosExperiment.bpmn successful with key 2251799813685251.
Deploy file bpmn/chaos/chaosToolkit.bpmn (size: 11031 bytes).
Deployed process model bpmn/chaos/chaosToolkit.bpmn successful with key 2251799813685253.
Create ChaosToolkit instance
Open workers: [zbchaos, readExperiments].
Handle read experiments job [key: 2251799813685265]
Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","pauses":{"after":5},"provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"}]}.
Handle zbchaos job [key: 2251799813685328]
Running command with args: [--namespace -zeebe --clientId  --clientSecret  --audience  version] 
zbchaos development (commit: HEAD)
Handle zbchaos job [key: 2251799813685374]
Running command with args: [--namespace -zeebe --clientId  --clientSecret  --audience  version] 
zbchaos development (commit: HEAD)
Handle zbchaos job [key: 2251799813685517]
Running command with args: [--namespace -zeebe --clientId  --clientSecret  --audience  version] 
zbchaos development (commit: HEAD)
Instance 2251799813685256 [definition 2251799813685253 ] completed
--- PASS: Test_ShouldBeAbleToRunExperiments (12.13s)
PASS

Process finished with the exit code 0

For the execution I used eze, instead of Zeebe to reduce execution time.

Test that chaos models can be deployed

Add new experiment definition for our integration tests

Add new integration tests which verifies the complete cycle of deploying models, creating instance and running workers

ChrisKujawa · 2022-12-06T09:01:52Z

Don't be afraid of the LOC around 900 lines are from go.sum. 🤷

deepthidevaki

Integration tests are always useful while making changes to the process model 👍 LGTM
Please remove the duplicate tests in chaos_worker_test.go.

go-chaos/worker/chaos_worker_test.go

go-chaos/integration/integration_test.go

deepthidevaki · 2022-12-06T09:29:42Z

go-chaos/cmd/worker_test.go

+	os.Stdout = out
+}
+
+func Test_ShouldRunCommand(t *testing.T) {


🙃 I think this test is not adding much value. runZbChaosCommand doesn't do much other than just delegating to the cobra command.

I would like to keep it. With this unit test failing, I can faster pin-point where an issue is, instead of having only a failed integration test.

Based on #267 (blocked by) related to #237 ------- **Fixed some smaller issues, like:** * rm one-direction from mutex * make possible to run workers against self-managed clusters * return correct errors on connect **Migrate the Deployment distribution experiment, as the first experiment, to zbchaos.** The experiment was executed and verified via the integration test against a self-managed cluster. I moved the experiment into the `chaos-experiments/camunda-cloud/test/` folder and migrated it, with that approach I was able to execute the experiment with `eze` and running against my self-managed `zell-chaos` zeebe cluster. Log output: ``` Deploy file bpmn/chaos/actionRunner.bpmn (size: 8788 bytes). Deployed process model bpmn/chaos/actionRunner.bpmn successful with key 2251799813685249. Deploy file bpmn/chaos/chaosExperiment.bpmn (size: 21403 bytes). Deployed process model bpmn/chaos/chaosExperiment.bpmn successful with key 2251799813685251. Deploy file bpmn/chaos/chaosToolkit.bpmn (size: 11031 bytes). Deployed process model bpmn/chaos/chaosToolkit.bpmn successful with key 2251799813685253. Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe deployment distribution should be fault-tolerant. Zeebe should be able to handle network outages and fail-overs and distribute the deployments after partitions are available again.","method":[{"name":"Create network partition between leaders","provider":{"arguments":["disconnect","brokers","--broker1PartitionId","1","--broker2PartitionId","3","--one-direction"],"path":"zbchaos","timeout":900,"type":"process"},"type":"action"},{"name":"Deploy different deployment versions.","provider":{"arguments":["deploy","process","--multipleVersions","10"],"path":"zbchaos","timeout":900,"type":"process"},"type":"action"},{"name":"Connect leaders again","provider":{"arguments":["connect","brokers"],"path":"zbchaos","timeout":900,"type":"process"},"type":"action"},{"name":"Create process instance of latest version on partition one","provider":{"arguments":["verify","instance-creation","--bpmnProcessId","multiVersion","--version","10","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Create process instance of latest version on partition two","provider":{"arguments":["verify","instance-creation","--bpmnProcessId","multiVersion","--version","10","--partitionId","2"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Create process instance of latest version on partition three","provider":{"arguments":["verify","instance-creation","--bpmnProcessId","multiVersion","--version","10","--partitionId","3"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe deployment distribution","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","pauses":{"after":5},"provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685376] Running command with args: [disconnect brokers --broker1PartitionId 1 --broker2PartitionId 3 --one-direction] Connecting to zell-chaos Running experiment in self-managed environment. Did not find zeebe cluster to pause reconciliation, ignoring. Patched statefulset Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-1 as LEADER for partition 1. Found Broker zell-chaos-zeebe-2 as LEADER for partition 3. Execute ["apt" "-qq" "update"] on pod zell-chaos-zeebe-1 Execute ["apt" "-qq" "install" "-y" "iproute2"] on pod zell-chaos-zeebe-1 Execute ["ip" "route" "replace" "unreachable" "10.0.4.223"] on pod zell-chaos-zeebe-1 Disconnect zell-chaos-zeebe-1 from zell-chaos-zeebe-2 Handle zbchaos job [key: 2251799813685585] Running command with args: [deploy process --multipleVersions 10] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy 10 versions of different type of models. Deployed [2/10] versions. Deployed [4/10] versions. Deployed [6/10] versions. Deployed [8/10] versions. Deployed [10/10] versions. Deployed different process models of different types and versions to zeebe! Handle zbchaos job [key: 2251799813685677] Running command with args: [connect brokers] Connecting to zell-chaos Running experiment in self-managed environment. Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-0 Error on connection Broker: zell-chaos-zeebe-0. Error: Execution exited with exit code 127 (Command not found). It is likely that the broker was not disconnected or restarted in between. Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-1 Execute ["sh" "-c" "ip route | grep -m 1 unreachable"] on pod zell-chaos-zeebe-1 Execute ["sh" "-c" "ip route del "] on pod zell-chaos-zeebe-1 Error on connection Broker: zell-chaos-zeebe-1. Error: command terminated with exit code 255 Execute ["sh" "-c" "command -v ip"] on pod zell-chaos-zeebe-2 Error on connection Broker: zell-chaos-zeebe-2. Error: Execution exited with exit code 127 (Command not found). It is likely that the broker was not disconnected or restarted in between. Handle zbchaos job [key: 2251799813685732] Running command with args: [verify instance-creation --bpmnProcessId multiVersion --version 10 --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Create process instance with BPMN process ID multiVersion and version 10 [variables: '', awaitResult: false] Created process instance with key 2251799815354503 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685779] Running command with args: [verify instance-creation --bpmnProcessId multiVersion --version 10 --partitionId 2] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Create process instance with BPMN process ID multiVersion and version 10 [variables: '', awaitResult: false] Created process instance with key 4503599628043172 on partition 2, required partition 2. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685825] Running command with args: [verify instance-creation --bpmnProcessId multiVersion --version 10 --partitionId 3] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Create process instance with BPMN process ID multiVersion and version 10 [variables: '', awaitResult: false] Created process instance with key 2251799815355181 on partition 1, required partition 3. Created process instance with key 2251799815355311 on partition 1, required partition 3. Created process instance with key 6755399441722396 on partition 3, required partition 3. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685877] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685969] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813686012] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813686157] Running command with args: [version] zbchaos development (commit: HEAD) Instance 2251799813685255 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (36.97s) PASS Process finished with the exit code 0 ```

ChrisKujawa added 9 commits December 5, 2022 13:00

test: add test for run command

9beb999

test: new chaos worker test

9c8880c

test: add first testcontainer integration test

abef514

Test that chaos models can be deployed

style: rm empty line

adf80e1

test: add new experiment definition

c31eed1

Add new experiment definition for our integration tests

refactor: add more verbose logging for the workers

1128dd3

refactor: extract worker opening

27e03ac

test: new integration test

bdc8dae

Add new integration tests which verifies the complete cycle of deploying models, creating instance and running workers

style: add license

7153c37

ChrisKujawa requested review from lenaschoenburg and deepthidevaki December 5, 2022 14:47

fix: use correct fmt

8601d0a

ChrisKujawa mentioned this pull request Dec 6, 2022

Migrate Deployment distribution experiment #268

Merged

deepthidevaki approved these changes Dec 6, 2022

View reviewed changes

test: remove duplicate test

7572a74

ChrisKujawa merged commit fc2d60a into main Dec 6, 2022

ChrisKujawa deleted the zell-tests branch December 6, 2022 09:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new integration tests #267

Add new integration tests #267

ChrisKujawa commented Dec 5, 2022 •

edited

Loading

ChrisKujawa commented Dec 6, 2022

deepthidevaki left a comment

deepthidevaki Dec 6, 2022

ChrisKujawa Dec 6, 2022

Add new integration tests #267

Add new integration tests #267

Conversation

ChrisKujawa commented Dec 5, 2022 • edited Loading

ChrisKujawa commented Dec 6, 2022

deepthidevaki left a comment

Choose a reason for hiding this comment

deepthidevaki Dec 6, 2022

Choose a reason for hiding this comment

ChrisKujawa Dec 6, 2022

Choose a reason for hiding this comment

ChrisKujawa commented Dec 5, 2022 •

edited

Loading