Async workflows integration test with kafka

[taylanisikdemir]

What changed?

Added an integration test to validate StartWorkflowExecutionAsync works. Due to the existing limitations and complicated nature of the integration test framework I had to make some enhancements.

Async WF Consumer side changes:

• Add timeout to context used in StartWorkflowExecution yarpc call because it's mandatory.
• Support for AsyncWorkflowConfig field in domain cache entry deep copy.

Test changes:

• Support per integration test dynamic config overrides. Also support passing different dynamic config overrides per service while creating in-memory cadence cluster.
• Create mock time source and pass it to domain cache and async wf consumer manager. We need this in order to update domain config during test and make sure the changes are reflected in the cache/consumer side.
• Get rid of cadence-auto-setup-with-kafka in favor of docker-compose based kafka topic creation. start.sh no longer needs kafka binaries.

Why?
Covering the new async workflows feature via an integration test to see start workflow path works e2e.

How did you test it?

1. Stop the previous run if any

docker-compose -f docker/buildkite/docker-compose-local-async-wf.yml down

2. Build the integration-test-async-wf image

docker-compose -f docker/buildkite/docker-compose-local-async-wf.yml build integration-test-async-wf

3. Run the test in the docker container

docker-compose -f docker/buildkite/docker-compose-local-async-wf.yml run --rm integration-test-async-wf

4. Full test run logs can be found at test.log file

[coveralls]

Pull Request Test Coverage Report for Build 018dcd61-abb3-4171-8ca4-27ded8ce02e4

Details

• -49 of 138 (64.49%) changed or added relevant lines in 10 files are covered.
• 50 unchanged lines in 9 files lost coverage.
• Overall coverage increased (+0.06%) to 62.798%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
host/dynamicconfig.go	2	4	50.0%
service/worker/asyncworkflow/async_workflow_consumer_manager.go	5	9	55.56%
host/integrationbase.go	0	5	0.0%
common/cache/domainCache.go	22	28	78.57%
host/onebox.go	22	54	40.74%

Files with Coverage Reduction	New Missed Lines	%
common/task/weighted_round_robin_task_scheduler.go	2	89.05%
common/persistence/nosql/nosqlplugin/cassandra/workflow.go	3	59.1%
service/history/execution/mutable_state_task_refresher.go	3	56.65%
service/history/task/fetcher.go	3	85.57%
service/matching/taskListManager.go	3	80.2%
service/frontend/wrappers/metered/metered.go	4	65.5%
service/history/task/transfer_standby_task_executor.go	4	86.6%
common/persistence/sql/workflowStateMaps.go	8	86.21%
common/persistence/nosql/nosqlplugin/cassandra/workflow_parsing_utils.go	20	79.13%

Totals
Change from base Build 018dcce8-df73-46fe-8ff7-2acba6a12910:	0.06%
Covered Lines:	92793
Relevant Lines:	147765

---

💛 - Coveralls