The LoadTests Python CoGBK Flink Batch job is flaky

[github-actions]

The LoadTests Python CoGBK Flink Batch is failing over 50% of the time.
Please visit https://github.com/apache/beam/actions/workflows/beam_LoadTests_Python_CoGBK_Flink_Batch.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs.
See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=9&var-Workflow=LoadTests%20Python%20CoGBK%20Flink%20Batch

[Amar3tto]

Culprit: #35621

[Abacn]

This test is supposed to run on Flink runner but from the logging it appears to run on PrismRunner ("INFO:apache_beam.runners.direct.direct_runner:Running pipeline with PrismRunner"). Assuming it ran on direct runner before (This means we get a load test for PrisimRunner accidentally). @shunping could you please check why it fails on Prism? @Amar3tto we should fix the test to use Flink runner (DataProc flink as other Python load tests)

[shunping]

It is due to the server timeout. Prism may be able to run it with a longer timeout pipeline option "--job_server_timeout=3600".

[shunping]

I confirmed that the runner argument is set in the gradle command but not passed into the pipeline options.

beam/sdks/python/apache_beam/testing/load_tests/build.gradle

Lines 43 to 44 in ac4b5ab

	def runnerArg = project.findProperty(runnerProperty) ?: ""
	if (runnerArg == 'DataflowRunner' || runnerArg == 'TestDataflowRunner') {

Without a runner option, we use Direct Runner by default.