'request timeout' error occurred during the text segmentation and cleaning process in the knowledge base #11694

gongshaojie12 · 2024-12-16T08:49:23Z

Self Checks

This is only for bug report, if you would like to ask a question, please head to Discussions.
I have searched for existing issues search for existing issues, including closed ones.
I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
Please do not modify this template :) and fill in all the required fields.

Dify version

0.13.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I changed the UPLOAD_FILE_SIZE_LIMIT to 1024 and the UPLOAD_FILE_BATCH_LIMIT to 500, then uploaded 60 PDF files, each ranging from 20MB to 160MB in size. After the upload was completed, a 'request timeout' error occurred during the text segmentation and cleaning process. How can I resolve this? The error is as follows:

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

crazywoola · 2024-12-16T08:51:53Z

Please provide the container logs.

dosubot · 2024-12-16T08:54:19Z

I found some similar threads that might help you resolve the 'request timeout' error during the text segmentation and cleaning process:

Open Issue: Error uploading files while building the knowledge base
- Suggested solutions include checking the rate limit for the Embedding model API key, ensuring the Celery worker service is running correctly, verifying that all necessary services are up, and checking for any pending processes or configuration issues ^[1].
Closed Issue: HTTPConnectionPool(host='weaviate', port=8080): Read timed out. (read timeout=60) when creating knowledge base
- Solutions provided include checking the status of all containers, inspecting the logs of the weaviate container, verifying network configuration, considering increasing timeout settings, restarting containers, updating environment variables, and ensuring system requirements are met ^[2].

These threads suggest several steps you can take to troubleshoot and potentially resolve the timeout issue. You might want to start by checking the status of your services and ensuring that your system meets the necessary requirements for handling large file uploads.

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

gongshaojie12 · 2024-12-16T08:54:52Z

Which container logs should be provided? @crazywoola

crazywoola · 2024-12-16T08:59:54Z

api and worker

gongshaojie12 · 2024-12-16T09:33:55Z

@crazywoola
worker logs:

2024-12-16 09:28:15.377 INFO [MainThread] [strategy.py:161] - Task tasks.document_indexing_task.document_indexing_task[c054201b-e233-4685-b9e9-a59979fa80cf] received
2024-12-16 09:28:15.383 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 5c70b91a-c323-4dd4-9e58-a9c269c7b6e8
2024-12-16 09:28:15.385 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 86649d17-3dc4-411b-8f32-7096740e42bf
2024-12-16 09:28:15.387 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 3fff9cd8-c24b-4704-a91b-ae772ecb73b3
2024-12-16 09:28:15.389 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 67ffac6b-2c18-4cda-9c42-0d5f3d940700
2024-12-16 09:28:15.390 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: b1de9ae0-2094-4538-b5a5-57b9770620e6
2024-12-16 09:28:15.392 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 0ceb2248-b8f1-4ec8-90b7-819db65990f2
2024-12-16 09:28:15.393 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 5678020e-9169-4bae-9359-3b10619aee20
2024-12-16 09:28:15.395 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 123cfa98-150d-4a3b-aaaf-d0b6ad6bca93
2024-12-16 09:28:15.397 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 59d7dfa4-2768-4b5f-8ef7-8798a027b156
2024-12-16 09:28:15.398 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: c74bb8ff-8d1d-4398-89dc-bef3050dcafa
2024-12-16 09:28:15.400 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 0550f1e3-4794-4721-8a05-3645ae9b1a99
2024-12-16 09:28:15.401 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: d4f83794-6d1a-4efa-be22-f5b389bbc4d4
2024-12-16 09:28:15.403 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 4c0f2b61-be97-49dc-b976-a78851b1eeb7
2024-12-16 09:28:15.404 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 911d76e7-28cb-4ac6-aa73-23677384f8bf
2024-12-16 09:28:15.406 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 3f3cd0f8-32ea-41ee-a541-aab118624ed7
2024-12-16 09:28:15.407 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 7b4c986e-5b2c-4da8-b33e-68b73e3ee909
2024-12-16 09:28:15.409 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 539afcff-5d35-4785-97c7-07f6e36ee103
2024-12-16 09:28:15.411 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: c9d63b1e-4d0b-4495-8f67-487d42dcc36f
2024-12-16 09:28:15.412 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 7cb9cb9a-301f-497a-81e4-ae74adb32d4d
2024-12-16 09:28:15.414 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: b0158c71-55de-4ab4-b28d-f9445e335e1b
2024-12-16 09:28:15.415 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: b30a0e7b-97ce-411d-8841-2f5baff97afd
2024-12-16 09:28:15.417 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 40e9802d-3e20-4ce2-a8ab-224aca9e3fa8
2024-12-16 09:28:15.418 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 6bc2f946-dbeb-444a-9f00-e83bcdead286
2024-12-16 09:28:15.420 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 25b0719e-d885-4c34-9798-9b011045a2a2
2024-12-16 09:28:15.421 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 0cb2665c-e95d-48ee-957a-d8573c03d8e6
2024-12-16 09:28:15.423 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 6aea4187-ea88-41ce-8a2c-391626caa115
2024-12-16 09:28:15.425 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: a19e2efb-c650-4905-b23d-5bfa114d7595
2024-12-16 09:28:15.426 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: ca21ca49-ab39-4271-a7a3-9475bb96f8bd
2024-12-16 09:28:15.428 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 532a0cc1-864c-4d75-a181-f876d4a19681
2024-12-16 09:28:15.429 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: f5d85366-948b-4d04-bee2-662a03ad874d
2024-12-16 09:28:15.431 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: fd1bbf64-5778-453f-8fdd-5ae010d2b064
2024-12-16 09:28:15.432 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: bd0004e7-e3d2-4107-b259-104d70c7a8ab
2024-12-16 09:28:15.434 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 6936c65d-af63-4245-8c6d-76ae53407a01
2024-12-16 09:28:15.435 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 116f6dc8-d735-4dc6-a5c2-7d23cb1762d9
2024-12-16 09:28:15.437 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: e9786bbc-8884-49eb-868a-4c092232a015
2024-12-16 09:28:15.438 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 4253905e-a852-4f8b-8492-6929c27a002c
2024-12-16 09:28:15.440 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 70e7a76c-057b-4e56-8192-d83a96275813
2024-12-16 09:28:15.442 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 3175dd98-4660-41de-9e14-7e3222eb7ac1
2024-12-16 09:28:15.443 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 2bb53547-b5d6-4689-aa85-7a1b41165897
2024-12-16 09:28:15.445 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: fda489cb-d9e2-4b6e-8ee1-e9469bf4e586
2024-12-16 09:28:15.446 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 652cdeb5-7ec7-48b9-8636-0d15f94cabc8
2024-12-16 09:28:15.448 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: bf79e557-4abd-4e63-83e5-2c6c7253ed90
2024-12-16 09:28:15.449 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: d1c09996-0d5c-456b-b042-76e501a0e87a
2024-12-16 09:28:15.451 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 55ff51aa-3e46-44d8-aa68-f07f40d850e7
2024-12-16 09:28:15.453 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 60f36bfd-88dd-43dc-bf75-0ee21847ee1b
2024-12-16 09:28:15.454 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: e6255a38-ca7d-4499-a970-7d43afc84ad0
2024-12-16 09:28:15.456 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: b45737d2-6538-4d93-9eda-6e186f63b3f7
2024-12-16 09:28:15.457 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 3a2c7ce8-edf0-4531-a9b4-0ec3e29a59f2
2024-12-16 09:28:15.459 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: c95123b0-aea7-4d9f-9fb8-380dee43e885
2024-12-16 09:28:15.461 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 9c6c8371-c79b-4831-882f-5dfa5ec8f9b9
2024-12-16 09:28:15.462 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: ec01b069-8641-4300-9ed0-30f93a06c7bf
2024-12-16 09:28:15.464 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 64ba6d67-794e-4355-ad36-f092ef8b0266
2024-12-16 09:28:15.465 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 846307a0-1e43-49d3-8700-63387d94041f
2024-12-16 09:28:15.467 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 33c65568-e823-4a83-bc06-f9e15b4f0b52
2024-12-16 09:28:15.468 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 2cae745f-1fbe-4373-9eea-0e9a6312e45a
2024-12-16 09:28:15.470 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 7c3fc7b4-6601-485a-9907-149ab6c9b355
2024-12-16 09:28:15.471 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: f2a1192f-4be9-43ea-9d0a-332675d80443
2024-12-16 09:28:15.473 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: b3ec9bb6-66cf-4fd4-8dbf-36829e86e997
2024-12-16 09:28:15.474 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: 8275c792-de3f-4f65-8fb9-3a9c5df1c2db
2024-12-16 09:28:15.476 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: ea8abf56-961a-4f75-a23f-771fbfe7f16a
2024-12-16 09:28:15.478 INFO [Dummy-16] [document_indexing_task.py:59] - Start process document: db85524d-7f39-412f-9ec3-1d517f7ad427
2024-12-16 09:28:43.990 ERROR [ThreadPoolExecutor-137_3] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:28:43.996 ERROR [ThreadPoolExecutor-137_3] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:28:45.901 ERROR [Dummy-16] [indexing_runner.py:94] - consume document failed
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/indexing_runner.py", line 78, in run
    self._load(
  File "/app/api/core/indexing_runner.py", line 672, in _load
    tokens += future.result()
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/indexing_runner.py", line 729, in _process_chunk
    index_processor.load(dataset, chunk_documents, with_keywords=False)
  File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 57, in load
    vector.create(documents)
  File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 150, in create
    embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/embedding/cached_embedding.py", line 93, in embed_documents
    raise ex
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:04.021 ERROR [ThreadPoolExecutor-188_8] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:04.023 ERROR [ThreadPoolExecutor-188_8] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:06.093 ERROR [Dummy-16] [indexing_runner.py:94] - consume document failed
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/indexing_runner.py", line 78, in run
    self._load(
  File "/app/api/core/indexing_runner.py", line 672, in _load
    tokens += future.result()
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/indexing_runner.py", line 729, in _process_chunk
    index_processor.load(dataset, chunk_documents, with_keywords=False)
  File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 57, in load
    vector.create(documents)
  File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 150, in create
    embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/embedding/cached_embedding.py", line 93, in embed_documents
    raise ex
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:20.911 ERROR [ThreadPoolExecutor-209_3] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:20.913 ERROR [ThreadPoolExecutor-209_3] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:21.013 ERROR [ThreadPoolExecutor-209_8] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:21.022 ERROR [ThreadPoolExecutor-209_8] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:22.008 ERROR [ThreadPoolExecutor-209_0] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:22.012 ERROR [ThreadPoolExecutor-209_0] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:25.424 ERROR [Dummy-16] [indexing_runner.py:94] - consume document failed
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/indexing_runner.py", line 78, in run
    self._load(
  File "/app/api/core/indexing_runner.py", line 672, in _load
    tokens += future.result()
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/indexing_runner.py", line 729, in _process_chunk
    index_processor.load(dataset, chunk_documents, with_keywords=False)
  File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 57, in load
    vector.create(documents)
  File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 150, in create
    embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/embedding/cached_embedding.py", line 93, in embed_documents
    raise ex
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:31.932 ERROR [ThreadPoolExecutor-221_0] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:31.935 ERROR [ThreadPoolExecutor-221_0] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:32.908 ERROR [ThreadPoolExecutor-221_8] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:32.912 ERROR [ThreadPoolExecutor-221_8] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:32.952 ERROR [ThreadPoolExecutor-221_4] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:32.955 ERROR [ThreadPoolExecutor-221_4] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:32.967 ERROR [ThreadPoolExecutor-221_1] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:32.970 ERROR [ThreadPoolExecutor-221_1] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:33.039 ERROR [ThreadPoolExecutor-221_9] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:33.039 ERROR [ThreadPoolExecutor-221_2] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:33.039 ERROR [ThreadPoolExecutor-221_3] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:33.044 ERROR [ThreadPoolExecutor-221_9] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:33.045 ERROR [ThreadPoolExecutor-221_2] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:33.046 ERROR [ThreadPoolExecutor-221_3] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:34.893 ERROR [ThreadPoolExecutor-221_1] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:34.897 ERROR [ThreadPoolExecutor-221_1] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:34.981 ERROR [ThreadPoolExecutor-221_3] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:34.989 ERROR [ThreadPoolExecutor-221_3] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:34.994 ERROR [ThreadPoolExecutor-221_4] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:34.994 ERROR [ThreadPoolExecutor-221_7] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:34.999 ERROR [ThreadPoolExecutor-221_4] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:35.000 ERROR [ThreadPoolExecutor-221_7] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:37.033 ERROR [Dummy-16] [indexing_runner.py:94] - consume document failed
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/indexing_runner.py", line 78, in run
    self._load(
  File "/app/api/core/indexing_runner.py", line 672, in _load
    tokens += future.result()
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/indexing_runner.py", line 729, in _process_chunk
    index_processor.load(dataset, chunk_documents, with_keywords=False)
  File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 57, in load
    vector.create(documents)
  File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 150, in create
    embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/embedding/cached_embedding.py", line 93, in embed_documents
    raise ex
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:51.902 ERROR [ThreadPoolExecutor-243_8] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:51.905 ERROR [ThreadPoolExecutor-243_8] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:51.925 ERROR [ThreadPoolExecutor-243_5] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:51.930 ERROR [ThreadPoolExecutor-243_5] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:52.001 ERROR [ThreadPoolExecutor-243_3] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:52.003 ERROR [ThreadPoolExecutor-243_3] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:52.945 ERROR [ThreadPoolExecutor-243_6] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:52.968 ERROR [ThreadPoolExecutor-243_6] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:52.974 ERROR [ThreadPoolExecutor-243_7] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:52.988 ERROR [ThreadPoolExecutor-243_7] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.
2024-12-16 09:29:53.026 ERROR [ThreadPoolExecutor-243_1] [http_request.py:181] - Request: https://dashscope.aliyuncs.com/api/v1/services/embeddings/text-embedding/text-embedding failed, status: 429, message: Requests rate limit exceeded, please try again later.
2024-12-16 09:29:53.029 ERROR [ThreadPoolExecutor-243_1] [cached_embedding.py:92] - Failed to embed documents: %s
Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
    return self._invoke(model, credentials, texts, user, input_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 67, in _invoke
    embeddings_batch, embedding_used_tokens = self.embed_documents(
                                              ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/tongyi/text_embedding/text_embedding.py", line 141, in embed_documents
    raise ValueError("Response output is missing or does not contain embeddings.")
ValueError: Response output is missing or does not contain embeddings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 177, in invoke_text_embedding
    return self._round_robin_invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 310, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [tongyi] Error: Response output is missing or does not contain embeddings.

```


**api logs**:
```
[2024-12-16 09:28:13 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:292)
[2024-12-16 09:28:14 +0000] [1] [ERROR] Worker (pid:292) was sent SIGKILL! Perhaps out of memory?
[2024-12-16 09:28:14 +0000] [294] [INFO] Booting worker with pid: 294
```

crazywoola · 2024-12-16T11:23:42Z

It seems there is something wrong with the tongyi models and exceeds the rate limit which we can not fix this for you.
And you can see,

[2024-12-16 09:28:14 +0000] [1] [ERROR] Worker (pid:292) was sent SIGKILL! Perhaps out of memory?

Which means you need to upgrade your machine.

gongshaojie12 · 2024-12-17T02:06:35Z

It’s impossible! This machine is quite high-spec, with 128GB of memory and 4*3090 GPU, each with 24GB of VRAM. And it’s not running any other programs, only Dify.

zolgear · 2024-12-18T06:43:38Z

As indicated in the logs, there is no queuing mechanism, so increasing UPLOAD_FILE_BATCH_LIMIT enables parallel execution.

Since UPLOAD_FILE_BATCH_LIMIT has been increased 100-fold from 5 to 500, the model's limits and memory would also need to be increased 100-fold...

It is recommended to register knowledge sequentially via the API.

dosubot bot added the 🐞 bug Something isn't working label Dec 16, 2024

crazywoola added the 🌚 invalid This doesn't seem right label Dec 16, 2024

crazywoola added the ⚙️ feat:model-runtime label Dec 16, 2024

crazywoola closed this as completed Dec 16, 2024

This was referenced Dec 27, 2024

The Q&A mode always fails. #12142

Open

Web Timeout: Setting TEXT_GENERATION_TIMEOUT_MS Not Effective #12170

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'request timeout' error occurred during the text segmentation and cleaning process in the knowledge base #11694

'request timeout' error occurred during the text segmentation and cleaning process in the knowledge base #11694

gongshaojie12 commented Dec 16, 2024

crazywoola commented Dec 16, 2024

dosubot bot commented Dec 16, 2024

gongshaojie12 commented Dec 16, 2024

crazywoola commented Dec 16, 2024

gongshaojie12 commented Dec 16, 2024 •

edited

Loading

crazywoola commented Dec 16, 2024 •

edited

Loading

gongshaojie12 commented Dec 17, 2024

zolgear commented Dec 18, 2024

'request timeout' error occurred during the text segmentation and cleaning process in the knowledge base #11694

'request timeout' error occurred during the text segmentation and cleaning process in the knowledge base #11694

Comments

gongshaojie12 commented Dec 16, 2024

Self Checks

Dify version

Cloud or Self Hosted

Steps to reproduce

✔️ Expected Behavior

❌ Actual Behavior

crazywoola commented Dec 16, 2024

dosubot bot commented Dec 16, 2024

gongshaojie12 commented Dec 16, 2024

crazywoola commented Dec 16, 2024

gongshaojie12 commented Dec 16, 2024 • edited Loading

crazywoola commented Dec 16, 2024 • edited Loading

gongshaojie12 commented Dec 17, 2024

zolgear commented Dec 18, 2024

gongshaojie12 commented Dec 16, 2024 •

edited

Loading

crazywoola commented Dec 16, 2024 •

edited

Loading