Error reported when uploading knowledge base. Vector database is Milvus

[andylzming]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.5.9

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

[2024-03-21 20:43:06,020: ERROR/MainProcess] RPC error: [insert_rows], <DataNotMatchException: (code=1, message=Attempt to insert an unexpected field to collection without enabling dynamic field)>, <Time:{'RPC start': '2024-03-21 20:43:06.017347', 'RPC error': '2024-03-21 20:43:06.020646'}>
[2024-03-21 20:43:06,021: ERROR/MainProcess] Failed to insert batch starting at entity: 0/11
[2024-03-21 20:43:06,021: ERROR/MainProcess] Failed to insert batch starting at entity: 0/11
[2024-03-21 20:43:06,021: ERROR/MainProcess] consume document failed
Traceback (most recent call last):
  File "/app/api/core/indexing_runner.py", line 70, in run
    self._load(
  File "/app/api/core/indexing_runner.py", line 667, in _load
    index_processor.load(dataset, chunk_documents)
  File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 59, in load
    vector.create(documents)
  File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 121, in create
    self._vector_processor.create(
  File "/app/api/core/rag/datasource/vdb/milvus/milvus_vector.py", line 75, in create
    self.add_texts(texts, embeddings)
  File "/app/api/core/rag/datasource/vdb/milvus/milvus_vector.py", line 101, in add_texts
    raise e
  File "/app/api/core/rag/datasource/vdb/milvus/milvus_vector.py", line 95, in add_texts
    ids = self._client.insert(collection_name=self._collection_name, data=batch_insert_list)
  File "/usr/local/lib/python3.10/site-packages/pymilvus/milvus_client/milvus_client.py", line 206, in insert
    raise ex from ex
  File "/usr/local/lib/python3.10/site-packages/pymilvus/milvus_client/milvus_client.py", line 198, in insert
    res = conn.insert_rows(collection_name, insert_batch, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 127, in handler
    raise e from e
  File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 123, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 162, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 102, in handler
    raise e from e
  File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 68, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 501, in insert_rows
    request = self._prepare_row_insert_request(
  File "/usr/local/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 482, in _prepare_row_insert_request
    return Prepare.row_insert_param(
  File "/usr/local/lib/python3.10/site-packages/pymilvus/client/prepare.py", line 422, in row_insert_param
    return cls._parse_row_request(request, fields_info, enable_dynamic, entities)
  File "/usr/local/lib/python3.10/site-packages/pymilvus/client/prepare.py", line 370, in _parse_row_request
    raise DataNotMatchException(message=ExceptionsMessage.InsertUnexpectedField)
pymilvus.exceptions.DataNotMatchException: <DataNotMatchException: (code=1, message=Attempt to insert an unexpected field to collection without enabling dynamic field)>
[2024-03-21 20:43:06,028: INFO/MainProcess] Processed dataset: bd66a1d2-d871-42c4-8fe7-4275be32a591 latency: 6.201149852946401
[2024-03-21 20:43:06,064: INFO/MainProcess] Task tasks.document_indexing_task.document_indexing_task[0df93ed7-9cae-4670-842d-8032b3d5852b] succeeded in 6.23699377477169s: None

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[dosubot]

Hey @andylzming, great to see you back here! Hope you're doing well.

Based on the error message you provided, it seems like you're trying to insert a field into a collection in Milvus without enabling dynamic fields. The error message DataNotMatchException: (code=1, message=Attempt to insert an unexpected field to collection without enabling dynamic field) is raised by the Milvus client when it encounters an unexpected field in the data being inserted into a collection.

This could happen if the schema of the data you're trying to insert doesn't match the schema of the collection in Milvus. For example, if your data has an extra field that is not present in the collection schema, this error could occur.

To resolve this issue, you could check the following:

1. Data Schema: Ensure that the schema of the data you're trying to insert matches the schema of the collection in Milvus. You can check the schema of the collection in Milvus using the get_collection_info or get_collection_stats methods of the Milvus client.

2. Dynamic Field: If your application requires the insertion of dynamic fields (fields not present in the collection schema), you need to enable dynamic fields in Milvus. However, as of the current version of Milvus (2.0), dynamic fields are not supported.

3. Dify Version: You mentioned that you're using Dify version 0.5.9. The latest version of Dify is 0.5.10. If possible, consider updating to the latest version as it may contain fixes or improvements that could resolve your issue.

For more information, you can refer to the Milvus SDK documentation and the Dify source code where the Milvus client is used.

If the problem persists, please provide more details about the data you're trying to insert and the schema of the collection in Milvus. This will help in identifying the exact cause of the issue.

Sources

• api/controllers/service_api/app/file.py
• api/core/rag/datasource/vdb/milvus/milvus_vector.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[crazywoola]

Please run flask db upgrade before you proceed.