Showcase tests hang indefinitely

[busunkim96]

The showcase session on recently opened PRs seems to hang #485 #470

Log

nox > Running session showcase
 
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/showcase
 
nox > pip install -e .
 
nox > curl https://github.com/googleapis/gapic-showcase/releases/download/v0.11.0/gapic-showcase-0.11.0.desc -L --output /tmp/tmp3ij41z1a/showcase.desc
 
nox > protoc --experimental_allow_proto3_optional --descriptor_set_in=/tmp/tmp3ij41z1a/showcase.desc --python_gapic_out=/tmp/tmp3ij41z1a google/showcase/v1beta1/echo.proto google/showcase/v1beta1/identity.proto google/showcase/v1beta1/messaging.proto
 
nox > pip install /tmp/tmp3ij41z1a
 
nox > pip install mock pytest pytest-asyncio
 
nox > py.test --quiet tests/system
 
......... 
 
Too long with no output (exceeded 10m0s): context deadline exceeded 

2020/06/29 20:38:42 Server Stream for Method: /google.showcase.v1beta1.Echo/Expand
2020/06/29 20:38:42     Sending Message:  content:"snails."
2020/06/29 20:38:42

I see the same behavior with a local run on master so I'm not sure what the root cause is.

[software-dov]

Running into the same issue on my end. I think @crwilcox encountered this too. Trying to reproduce locally

[software-dov]

It's the async code in test/system/test_grpc_streams.py. @lidizheng , can you assist? Did these ever pass?

[busunkim96]

https://app.circleci.com/pipelines/github/googleapis/gapic-generator-python/326/workflows/4bcdf203-1433-4559-a5bb-1851fd2f5717/jobs/13183/steps @software-dov It looks like it ran and passed on the most recent commit to master.

[busunkim96]

It looks like a new version of grpcio came out last week https://github.com/grpc/grpc/releases/tag/v1.30.0.

pytest-asyncio also had a new release, but the changelog doesn't have anything that stands out to me

[lidizheng]

@software-dov I can reproduce the error with HEAD. There could be a bug in v1.30.0. Looking into it.

[lidizheng]

The root cause is that the event loop got re-created between tests. This behavior was unexpected since the AsyncIO stack built upon the assumption that applications don't close the event loop in main thread. Unfortunately, the pytest-asyncio chose to create new loop every time (pytest-dev/pytest-asyncio#164) in the new release.

This is a known issue. I had an override (see code) to work around their official fixture, but the new change doesn't work with my fixture (see change).

A temporary fix is available at #493. I will improve the gRPC AsyncIO stack to accommodate this case. Sorry for the trouble.

@software-dov

[lidizheng]

Followup fix of the ephemeral-loop-deadlock issue in grpc/grpc#23377.