Async discoveryengine_v1 client not working

[francescov1]

Determine this is the right repository

• I determined this is the correct repository in which to report this bug.

Summary of the issue

Context
I am trying to use the DocumentServiceAsyncClient but am getting event loop errors. I believe the library is creating separate event loops.

Expected Behavior:
I can use the async client without it creating event loops.

Actual Behavior:
When I call document_client.import_documents(), i get

RuntimeError: Task <Task pending name='Task-3' coro=<_wrap_awaitable() running at /Users/francescovirga/.pyenv/versions/3.11.7/lib/python3.11/asyncio/tasks.py:694> cb=[_release_waiter(<Future pendi...ask_wakeup()]>)() at /Users/francescovirga/.pyenv/versions/3.11.7/lib/python3.11/asyncio/tasks.py:431]> got Future <Task pending name='Task-2' coro=<UnaryUnaryCall._invoke() running at /Users/francescovirga/.local/share/virtualenvs/lighthouse-ai-chat-JRzDFnJw/lib/python3.11/site-packages/grpc/aio/_call.py:577>> attached to a different loop

API client name and version

No response

Reproduction steps: code

file: main.py

from google.cloud.discoveryengine_v1 import (
    DocumentServiceAsyncClient,
    ImportDocumentsRequest,
    GcsSource,
)
from src.config import Config
from google.oauth2 import service_account

LOCATION = "global"
DATA_STORE_ID = "document-search-datastore"

credentials = service_account.Credentials.from_service_account_file(
    "vertex-ai-service-account-key.json"
)

document_client = DocumentServiceAsyncClient(credentials=credentials)
DOCUMENT_CLIENT_BRANCH_PATH = document_client.branch_path(
    project=Config.GCP_PROJECT_ID,
    location=LOCATION,
    data_store=DATA_STORE_ID,
    branch="default_branch",
)


async def import_documents(gcs_uri: str):
    """
    gcs_uri should be the file path to the jsonl metadata file in GCS
    """

    request = ImportDocumentsRequest(
        parent=DOCUMENT_CLIENT_BRANCH_PATH,
        gcs_source=GcsSource(
            input_uris=[gcs_uri],
            # Unstructured documents with custom JSONL metadata
            data_schema="custom",
        ),
        reconciliation_mode=ImportDocumentsRequest.ReconciliationMode.INCREMENTAL,
    )

    await document_client.import_documents(request=request)


if __name__ == "__main__":
    import asyncio

    async def main():
        await import_documents("gs://sec_documents/development/metadata.jsonl")

    asyncio.run(main())

Reproduction steps: supporting files

No response

Reproduction steps: actual results

terminal output

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/francescovirga/Projects/lighthouse-ai-chat/src/clients/google_discovery_engine/client.py", line 58, in <module>
    asyncio.run(main())
  File "/Users/francescovirga/.pyenv/versions/3.11.7/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/francescovirga/.pyenv/versions/3.11.7/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/francescovirga/.pyenv/versions/3.11.7/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/francescovirga/Projects/lighthouse-ai-chat/src/clients/google_discovery_engine/client.py", line 56, in main
    await import_documents("gs://sec_documents/development/metadata.jsonl")
  File "/Users/francescovirga/Projects/lighthouse-ai-chat/src/clients/google_discovery_engine/client.py", line 41, in import_documents
    operation = await document_client.import_documents(request=request)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/francescovirga/.local/share/virtualenvs/lighthouse-ai-chat-JRzDFnJw/lib/python3.11/site-packages/google/cloud/discoveryengine_v1/services/document_service/async_client.py", line 1010, in import_documents
    response = await rpc(
               ^^^^^^^^^^
  File "/Users/francescovirga/.local/share/virtualenvs/lighthouse-ai-chat-JRzDFnJw/lib/python3.11/site-packages/google/api_core/retry_async.py", line 223, in retry_wrapped_func
    return await retry_target(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/francescovirga/.local/share/virtualenvs/lighthouse-ai-chat-JRzDFnJw/lib/python3.11/site-packages/google/api_core/retry_async.py", line 121, in retry_target
    return await asyncio.wait_for(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/francescovirga/.pyenv/versions/3.11.7/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/Users/francescovirga/.pyenv/versions/3.11.7/lib/python3.11/asyncio/tasks.py", line 694, in _wrap_awaitable
    return (yield from awaitable.__await__())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/francescovirga/.local/share/virtualenvs/lighthouse-ai-chat-JRzDFnJw/lib/python3.11/site-packages/google/api_core/grpc_helpers_async.py", line 85, in __await__
    response = yield from self._call.__await__()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/francescovirga/.local/share/virtualenvs/lighthouse-ai-chat-JRzDFnJw/lib/python3.11/site-packages/grpc/aio/_call.py", line 308, in __await__
    response = yield from self._call_response
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Task <Task pending name='Task-3' coro=<_wrap_awaitable() running at /Users/francescovirga/.pyenv/versions/3.11.7/lib/python3.11/asyncio/tasks.py:694> cb=[_release_waiter(<Future pendi...ask_wakeup()]>)() at /Users/francescovirga/.pyenv/versions/3.11.7/lib/python3.11/asyncio/tasks.py:431]> got Future <Task pending name='Task-2' coro=<UnaryUnaryCall._invoke() running at /Users/francescovirga/.local/share/virtualenvs/lighthouse-ai-chat-JRzDFnJw/lib/python3.11/site-packages/grpc/aio/_call.py:577>> attached to a different loop

Reproduction steps: expected results

No error

OS & version + platform

MacOS 14.4

Python environment

Python 3.11.7

Python dependencies

Package Version

---

absl-py 2.1.0
aiofiles 23.2.1
aiohappyeyeballs 2.4.3
aiohttp 3.10.10
aiomysql 0.2.0
aiosignal 1.3.1
alembic 1.14.0
altair 5.4.1
aniso8601 9.0.1
annotated-types 0.7.0
anyio 4.6.2.post1
appnope 0.1.4
asn1crypto 1.5.1
astroid 3.3.5
asttokens 2.4.1
async-timeout 4.0.3
asyncmy 0.2.9
asyncpg 0.30.0
attrs 24.2.0
azure-core 1.32.0
azure-storage-blob 12.23.1
backoff 2.2.1
beautifulsoup4 4.12.3
black 24.10.0
blinker 1.8.2
boto3 1.35.43
botocore 1.35.43
bs4 0.0.2
CacheControl 0.14.0
cachetools 5.5.0
certifi 2024.8.30
cffi 1.17.1
cfgv 3.4.0
chardet 5.2.0
charset-normalizer 3.4.0
chex 0.1.87
click 8.1.7
cloudpickle 2.2.1
cloudscraper 1.2.71
cohere 5.11.3
comm 0.2.2
contourpy 1.3.0
cryptography 43.0.3
cycler 0.12.1
databricks-sdk 0.35.0
datasets 3.1.0
debugpy 1.8.6
decorator 5.1.1
Deprecated 1.2.14
dill 0.3.8
distlib 0.3.8
distro 1.9.0
dnspython 2.7.0
docker 7.1.0
docstring_parser 0.16
etils 1.10.0
eval_type_backport 0.2.0
execnet 2.1.1
executing 2.1.0
fake-useragent 1.5.1
fastapi 0.115.4
fastavro 1.9.7
filelock 3.16.1
firebase-admin 6.5.0
Flask 3.0.3
flax 0.10.1
fonttools 4.54.1
frozenlist 1.5.0
fsspec 2024.9.0
gcloud-aio-auth 5.3.2
gcloud-aio-storage 9.3.0
gitdb 4.0.11
GitPython 3.1.43
google-api-core 2.15.0
google-api-python-client 2.114.0
google-auth 2.26.2
google-auth-httplib2 0.2.0
google-cloud-aiplatform 1.71.1
google-cloud-bigquery 3.18.0
google-cloud-core 2.4.1
google-cloud-discoveryengine 0.13.3
google-cloud-firestore 2.14.0
google-cloud-resource-manager 1.12.3
google-cloud-storage 2.14.0
google-crc32c 1.5.0
google-pasta 0.2.0
google-resumable-media 2.7.0
googleapis-common-protos 1.62.0
graphene 3.3
graphql-core 3.2.5
graphql-relay 3.2.0
greenlet 3.1.1
grpc-google-iam-v1 0.13.0
grpcio 1.67.1
grpcio-status 1.62.3
grpcio-tools 1.62.3
gunicorn 23.0.0
h11 0.14.0
h2 4.1.0
hpack 4.0.0
html2text 2024.2.26
httpcore 1.0.6
httplib2 0.22.0
httpx 0.27.2
httpx-sse 0.4.0
huggingface-hub 0.26.2
humanize 4.11.0
hyperframe 6.0.1
identify 2.6.1
idna 3.10
importlib-metadata 6.11.0
importlib_resources 6.4.5
inflection 0.5.1
iniconfig 2.0.0
ipykernel 6.29.5
ipython 8.28.0
isodate 0.7.2
isort 5.13.2
itsdangerous 2.2.0
jax 0.4.35
jaxlib 0.4.35
jedi 0.19.1
Jinja2 3.1.4
jiter 0.7.0
jmespath 1.0.1
joblib 1.4.2
jsonpatch 1.33
jsonpointer 3.0.0
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
jupyter_client 8.6.3
jupyter_core 5.7.2
kiwisolver 1.4.7
langchain 0.3.7
langchain-core 0.3.15
langchain-text-splitters 0.3.2
langsmith 0.1.140
lxml 5.3.0
Mako 1.3.6
Markdown 3.7
markdown-it-py 3.0.0
MarkupSafe 3.0.2
matplotlib 3.9.2
matplotlib-inline 0.1.7
mccabe 0.7.0
mdurl 0.1.2
ml_dtypes 0.5.0
mlflow 2.17.0
mlflow-skinny 2.17.0
mock 4.0.3
more-itertools 10.5.0
msgpack 1.1.0
multidict 6.1.0
multiprocess 0.70.16
mypy 1.11.2
mypy-extensions 1.0.0
mysql-connector-python 9.1.0
narwhals 1.13.2
Nasdaq-Data-Link 1.0.4
nest-asyncio 1.6.0
nodeenv 1.9.1
numpy 1.26.4
openai 1.54.2
opentelemetry-api 1.27.0
opentelemetry-sdk 1.27.0
opentelemetry-semantic-conventions 0.48b0
opt_einsum 3.4.0
optax 0.2.3
orbax-checkpoint 0.8.0
orjson 3.10.11
outcome 1.3.0.post0
packaging 24.1
pandas 2.2.3
pandas_ta 0.3.14b0
parameterized 0.9.0
parso 0.8.4
pathos 0.3.2
pathspec 0.12.1
pendulum 3.0.0
pexpect 4.9.0
pg8000 1.31.2
pgvector 0.3.6
pillow 10.4.0
pip 24.2
platformdirs 4.3.6
playwright 1.48.0
pluggy 1.5.0
portalocker 2.10.1
pox 0.3.5
ppft 1.7.6.9
pre_commit 4.0.1
prometheus_client 0.21.0
prompt_toolkit 3.0.48
propcache 0.2.0
proto-plus 1.25.0
protobuf 4.25.5
psutil 6.0.0
ptyprocess 0.7.0
pure_eval 0.2.3
pyarrow 18.0.0
pyasn1 0.6.1
pyasn1_modules 0.4.0
pycparser 2.22
pydantic 2.9.2
pydantic_core 2.23.4
pydeck 0.9.1
pydeps 1.12.20
pyee 12.0.0
Pygments 2.18.0
PyJWT 2.9.0
pylint 3.3.1
pymongo 4.10.1
PyMySQL 1.1.1
pyparsing 3.2.0
PySocks 1.7.1
pytest 8.3.3
pytest-asyncio 0.24.0
pytest-rerunfailures 14.0
pytest-xdist 3.6.1
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-slugify 8.0.4
pytz 2024.2
PyYAML 6.0.2
pyzmq 26.2.0
qdrant-client 1.12.1
RapidFuzz 3.10.1
redis 5.2.0
referencing 0.35.1
regex 2024.9.11
requests 2.32.3
requests-toolbelt 1.0.0
rich 13.9.4
rpds-py 0.21.0
rsa 4.9
s3transfer 0.10.3
safetensors 0.4.5
sagemaker 2.232.2
sagemaker-core 1.0.10
sagemaker-mlflow 0.1.0
schema 0.7.7
scikit-learn 1.5.2
scipy 1.14.1
scramp 1.4.5
selenium 4.26.1
sentry-sdk 2.18.0
setuptools 75.3.0
shapely 2.0.6
shellingham 1.5.4
six 1.16.0
smdebug-rulesconfig 1.0.1
smmap 5.0.1
sniffio 1.3.1
sortedcontainers 2.4.0
soupsieve 2.6
SQLAlchemy 2.0.36
sqlparse 0.5.1
stack-data 0.6.3
starlette 0.41.2
stdlib-list 0.10.0
streamlit 1.39.0
tabulate 0.9.0
talipp 2.4.0
tblib 3.0.0
tenacity 9.0.0
tensorstore 0.1.67
text-unidecode 1.3
threadpoolctl 3.5.0
tiktoken 0.8.0
time-machine 2.16.0
together 1.3.1
tokenizers 0.20.3
toml 0.10.2
tomlkit 0.13.2
toolz 1.0.0
tornado 6.4.1
tqdm 4.67.0
traitlets 5.14.3
transformers 4.46.2
trio 0.27.0
trio-websocket 0.11.1
typer 0.12.5
types-cffi 1.16.0.20240331
types-pyOpenSSL 24.1.0.20240722
types-python-slugify 8.0.2.20240310
types-pytz 2024.2.0.20241003
types-redis 4.6.0.20241004
types-requests 2.32.0.20241016
types-retry 0.9.9.4
types-setuptools 75.1.0.20240917
typing_extensions 4.12.2
tzdata 2024.2
uritemplate 4.1.1
urllib3 2.2.3
uuid 1.30
uvicorn 0.32.0
virtualenv 20.26.6
watchdog 6.0.0
wcwidth 0.2.13
webdriver-manager 4.0.2
websocket-client 1.8.0
Werkzeug 3.0.4
wheel 0.44.0
wrapt 1.16.0
wsproto 1.2.0
xxhash 3.5.0
yarl 1.17.1
zipp 3.20.2

Additional context

No response

[francescov1]

Any news on this?

[grexrply]

Hello, I have the same problem on another client of DiscoveryEngine but it is a very similar problem. The client is ConversationalSearchServiceAsyncClient.
When i use the client with asyncio and running it on a simple python script I have the same error of @francescov1.
When I use the service using the client in my FastAPI application, I have the error:
RuntimeError: There is no current event loop in thread 'AnyIO worker thread'.

At the moment, the working solution seems to use a sync client and running the operation in a threadpool.

[vchudnov-g]

Thanks for reporting this issue! We will take a look and resolve it as soon as we can.