[Bug]: source collection has upsert operations, the recovered data will be less than the original data #478

zhuwenxing · 2024-12-13T08:47:52Z

Current Behavior

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:49 - INFO - ci_test]: src count: data: ["{'count()': 3000}"] , dist count: data: ["{'count()': 1497}"] (client_base.py:562)

[2024-12-13T05:23:50.877Z] ____________ TestRestoreBackup.test_milvus_restore_back_with_upsert ____________

[2024-12-13T05:23:50.877Z] 

[2024-12-13T05:23:50.877Z] self = <test_restore_backup.TestRestoreBackup object at 0x7fac2ed19c30>

[2024-12-13T05:23:50.877Z] 

[2024-12-13T05:23:50.877Z]     @pytest.mark.tags(CaseLabel.L1)

[2024-12-13T05:23:50.877Z]     def test_milvus_restore_back_with_upsert(self):

[2024-12-13T05:23:50.877Z]         self._connect()

[2024-12-13T05:23:50.877Z]         name_origin = cf.gen_unique_str(prefix)

[2024-12-13T05:23:50.877Z]         back_up_name = cf.gen_unique_str(backup_prefix)

[2024-12-13T05:23:50.877Z]         fields = [

[2024-12-13T05:23:50.877Z]             cf.gen_int64_field(name="int64", is_primary=True),

[2024-12-13T05:23:50.877Z]             cf.gen_int64_field(name="key"),

[2024-12-13T05:23:50.877Z]             cf.gen_json_field(name="json"),

[2024-12-13T05:23:50.877Z]             cf.gen_array_field(name="var_array", element_type=DataType.VARCHAR),

[2024-12-13T05:23:50.877Z]             cf.gen_array_field(name="int_array", element_type=DataType.INT64),

[2024-12-13T05:23:50.877Z]             cf.gen_float_vec_field(name="float_vector", dim=128),

[2024-12-13T05:23:50.877Z]         ]

[2024-12-13T05:23:50.877Z]         default_schema = cf.gen_collection_schema(fields)

[2024-12-13T05:23:50.877Z]         collection_w = self.init_collection_wrap(

[2024-12-13T05:23:50.877Z]             name=name_origin, schema=default_schema, active_trace=True

[2024-12-13T05:23:50.877Z]         )

[2024-12-13T05:23:50.877Z]         nb = 3000

[2024-12-13T05:23:50.877Z]         data = [

[2024-12-13T05:23:50.877Z]             [i for i in range(nb)],

[2024-12-13T05:23:50.877Z]             [i % 3 for i in range(nb)],

[2024-12-13T05:23:50.877Z]             [{f"key_{str(i)}": i} for i in range(nb)],

[2024-12-13T05:23:50.877Z]             [[str(x) for x in range(10)] for i in range(nb)],

[2024-12-13T05:23:50.877Z]             [[int(x) for x in range(10)] for i in range(nb)],

[2024-12-13T05:23:50.877Z]             [[np.float32(i) for i in range(128)] for _ in range(nb)],

[2024-12-13T05:23:50.877Z]         ]

[2024-12-13T05:23:50.877Z]         res, result = collection_w.insert(data=data)

[2024-12-13T05:23:50.877Z]         # upsert first 100 rows by pk

[2024-12-13T05:23:50.877Z]         upsert_data = [

[2024-12-13T05:23:50.877Z]             [i for i in range(100)],

[2024-12-13T05:23:50.877Z]             [i % 3 for i in range(100, 200)],

[2024-12-13T05:23:50.877Z]             [{f"key_{str(i)}": i} for i in range(100, 200)],

[2024-12-13T05:23:50.877Z]             [[str(x) for x in range(10, 20)] for _ in range(100)],

[2024-12-13T05:23:50.877Z]             [[int(x) for x in range(10)] for _ in range(100)],

[2024-12-13T05:23:50.877Z]             [[np.float32(i) for i in range(128, 128 * 2)] for _ in range(100)],

[2024-12-13T05:23:50.877Z]         ]

[2024-12-13T05:23:50.877Z]         res, result = collection_w.upsert(data=upsert_data)

[2024-12-13T05:23:50.877Z]         res = self.client.create_backup(

[2024-12-13T05:23:50.877Z]             {

[2024-12-13T05:23:50.877Z]                 "async": False,

[2024-12-13T05:23:50.877Z]                 "backup_name": back_up_name,

[2024-12-13T05:23:50.877Z]                 "collection_names": [name_origin],

[2024-12-13T05:23:50.877Z]             }

[2024-12-13T05:23:50.877Z]         )

[2024-12-13T05:23:50.877Z]         log.info(f"create_backup {res}")

[2024-12-13T05:23:50.877Z]         res = self.client.list_backup()

[2024-12-13T05:23:50.877Z]         log.info(f"list_backup {res}")

[2024-12-13T05:23:50.877Z]         if "data" in res:

[2024-12-13T05:23:50.877Z]             all_backup = [r["name"] for r in res["data"]]

[2024-12-13T05:23:50.877Z]         else:

[2024-12-13T05:23:50.877Z]             all_backup = []

[2024-12-13T05:23:50.877Z]         assert back_up_name in all_backup

[2024-12-13T05:23:50.877Z]         backup = self.client.get_backup(back_up_name)

[2024-12-13T05:23:50.877Z]         assert backup["data"]["name"] == back_up_name

[2024-12-13T05:23:50.877Z]         backup_collections = [

[2024-12-13T05:23:50.877Z]             backup["collection_name"] for backup in backup["data"]["collection_backups"]

[2024-12-13T05:23:50.877Z]         ]

[2024-12-13T05:23:50.877Z]         assert name_origin in backup_collections

[2024-12-13T05:23:50.877Z]         res = self.client.restore_backup(

[2024-12-13T05:23:50.877Z]             {

[2024-12-13T05:23:50.877Z]                 "async": False,

[2024-12-13T05:23:50.877Z]                 "backup_name": back_up_name,

[2024-12-13T05:23:50.877Z]                 "collection_names": [name_origin],

[2024-12-13T05:23:50.877Z]                 "collection_suffix": suffix,

[2024-12-13T05:23:50.877Z]             }

[2024-12-13T05:23:50.877Z]         )

[2024-12-13T05:23:50.877Z]         log.info(f"restore_backup: {res}")

[2024-12-13T05:23:50.877Z]         res, _ = self.utility_wrap.list_collections()

[2024-12-13T05:23:50.877Z]         assert name_origin + suffix in res

[2024-12-13T05:23:50.877Z]         output_fields = None

[2024-12-13T05:23:50.877Z] >       self.compare_collections(

[2024-12-13T05:23:50.877Z]             name_origin,

[2024-12-13T05:23:50.877Z]             name_origin + suffix,

[2024-12-13T05:23:50.877Z]             output_fields=output_fields,

[2024-12-13T05:23:50.877Z]             verify_by_query=True,

[2024-12-13T05:23:50.877Z]         )

[2024-12-13T05:23:50.877Z] 

[2024-12-13T05:23:50.877Z] testcases/test_restore_backup.py:1151: 

[2024-12-13T05:23:50.877Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

[2024-12-13T05:23:50.877Z] 

[2024-12-13T05:23:50.877Z] self = <test_restore_backup.TestRestoreBackup object at 0x7fac2ed19c30>

[2024-12-13T05:23:50.877Z] src_name = 'restore_backup_UrnJO7cV', dist_name = 'restore_backup_UrnJO7cV_bak'

[2024-12-13T05:23:50.877Z] output_fields = ['*'], verify_by_query = True

[2024-12-13T05:23:50.877Z] 

[2024-12-13T05:23:50.877Z]     def compare_collections(

[2024-12-13T05:23:50.877Z]         self, src_name, dist_name, output_fields=None, verify_by_query=False

[2024-12-13T05:23:50.877Z]     ):

[2024-12-13T05:23:50.877Z]         if output_fields is None:

[2024-12-13T05:23:50.877Z]             output_fields = ["*"]

[2024-12-13T05:23:50.877Z]         collection_src, _ = self.collection_wrap.init_collection(name=src_name)

[2024-12-13T05:23:50.877Z]         collection_dist, _ = self.collection_wrap.init_collection(name=dist_name)

[2024-12-13T05:23:50.877Z]         log.info(f"collection_src schema: {collection_src.schema}")

[2024-12-13T05:23:50.877Z]         log.info(f"collection_dist schema: {collection_dist.schema}")

[2024-12-13T05:23:50.877Z]         assert collection_src.schema == collection_dist.schema

[2024-12-13T05:23:50.877Z]         # get partitions

[2024-12-13T05:23:50.877Z]         partitions_src = collection_src.partitions

[2024-12-13T05:23:50.877Z]         partitions_dist = collection_dist.partitions

[2024-12-13T05:23:50.877Z]         log.info(

[2024-12-13T05:23:50.877Z]             f"partitions_src: {partitions_src}, partitions_dist: {partitions_dist}"

[2024-12-13T05:23:50.877Z]         )

[2024-12-13T05:23:50.877Z]         assert len(partitions_src) == len(partitions_dist)

[2024-12-13T05:23:50.877Z]         # get num entities

[2024-12-13T05:23:50.877Z]         src_num = collection_src.num_entities

[2024-12-13T05:23:50.877Z]         dist_num = collection_dist.num_entities

[2024-12-13T05:23:50.877Z]         log.info(f"src_num: {src_num}, dist_num: {dist_num}")

[2024-12-13T05:23:50.877Z]         if not verify_by_query:

[2024-12-13T05:23:50.877Z]             assert src_num == dist_num, f"srs_num: {src_num}, dist_num: {dist_num}"

[2024-12-13T05:23:50.877Z]             return

[2024-12-13T05:23:50.877Z]         for coll in [collection_src, collection_dist]:

[2024-12-13T05:23:50.877Z]             is_binary = self.is_binary_by_schema(coll.schema)

[2024-12-13T05:23:50.877Z]             try:

[2024-12-13T05:23:50.877Z]                 if is_binary:

[2024-12-13T05:23:50.877Z]                     coll.create_index(

[2024-12-13T05:23:50.877Z]                         ct.default_binary_vec_field_name,

[2024-12-13T05:23:50.877Z]                         ct.default_bin_flat_index,

[2024-12-13T05:23:50.877Z]                         index_name=cf.gen_unique_str(),

[2024-12-13T05:23:50.877Z]                     )

[2024-12-13T05:23:50.877Z]                 else:

[2024-12-13T05:23:50.877Z]                     coll.create_index(

[2024-12-13T05:23:50.877Z]                         ct.default_float_vec_field_name,

[2024-12-13T05:23:50.877Z]                         ct.default_index,

[2024-12-13T05:23:50.877Z]                         index_name=cf.gen_unique_str(),

[2024-12-13T05:23:50.877Z]                     )

[2024-12-13T05:23:50.877Z]             except Exception as e:

[2024-12-13T05:23:50.877Z]                 log.error(f"collection {coll.name} create index failed with error: {e}")

[2024-12-13T05:23:50.877Z]             coll.load()

[2024-12-13T05:23:50.877Z]             time.sleep(5)

[2024-12-13T05:23:50.877Z]         # get entities by count

[2024-12-13T05:23:50.877Z]         src_count = collection_src.query(expr="", output_fields=["count(*)"])

[2024-12-13T05:23:50.877Z]         dist_count = collection_dist.query(expr="", output_fields=["count(*)"])

[2024-12-13T05:23:50.877Z]         log.info(f"src count: {src_count}, dist count: {dist_count}")

[2024-12-13T05:23:50.877Z]         src_res = collection_src.query(

[2024-12-13T05:23:50.877Z]             expr=f"{ct.default_int64_field_name} >= 0", output_fields=output_fields

[2024-12-13T05:23:50.877Z]         )

[2024-12-13T05:23:50.877Z]         # log.info(f"src res: {len(src_res)}, src res: {src_res[-1]}")

[2024-12-13T05:23:50.877Z]         dist_res = collection_dist.query(

[2024-12-13T05:23:50.877Z]             expr=f"{ct.default_int64_field_name} >= 0", output_fields=output_fields

[2024-12-13T05:23:50.877Z]         )

[2024-12-13T05:23:50.877Z]         # log.info(f"dist res: {len(dist_res)}, dist res: {dist_res[-1]}")

[2024-12-13T05:23:50.878Z] >       assert len(dist_res) == len(src_res)

[2024-12-13T05:23:50.878Z] E       AssertionError

[2024-12-13T05:23:50.878Z] 

[2024-12-13T05:23:50.878Z] base/client_base.py:571: AssertionError

[2024-12-13T05:23:50.878Z] ------------------------------ Captured log setup ------------------------------

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - INFO - ci_test]: *********************************** setup *********************************** (client_base.py:39)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - INFO - ci_test]: [setup_method] Start setup test case test_milvus_restore_back_with_upsert. (client_base.py:40)

[2024-12-13T05:23:50.878Z] ------------------------------ Captured log call -------------------------------

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [Connections.connect] args: ['default'], kwargs: {'host': '10.255.80.42', 'port': 19530} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : None  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['int64', <DataType.INT64: 5>, ''], kwargs: {'is_primary': True} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : {'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['key', <DataType.INT64: 5>, ''], kwargs: {'is_primary': False} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : {'name': 'key', 'description': '', 'type': <DataType.INT64: 5>}  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['json', <DataType.JSON: 23>, ''], kwargs: {'is_primary': False} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : {'name': 'json', 'description': '', 'type': <DataType.JSON: 23>}  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['var_array', <DataType.ARRAY: 22>, ''], kwargs: {'is_primary': False, 'element_type': <DataType.VARCHAR: 21>, 'max_capacity': 2000, 'max_length': 1500} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : {'name': 'var_array', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 1500, 'max_capacity': 2000}, 'element_type': <DataType.VARCHAR: 21>}  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['int_array', <DataType.ARRAY: 22>, ''], kwargs: {'is_primary': False, 'element_type': <DataType.INT64: 5>, 'max_capacity': 2000, 'max_length': 1500} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : {'name': 'int_array', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 1500, 'max_capacity': 2000}, 'element_type': <DataType.INT64: 5>}  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['float_vector', <DataType.FLOAT_VECTOR: 101>, ''], kwargs: {'dim': 128, 'is_primary': False} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [CollectionSchema] args: [[{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'key', 'description': '', 'type': <DataType.INT64: 5>}, {'name': 'json', 'description': '', 'type': <DataType.JSON: 23>}, {'name': 'var_array', 'description': '', 'type': <DataType.ARR......, kwargs: {'primary_field': None, 'auto_id': False} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'key', 'description': '', 'type': <DataType.INT64: 5>}, {'name': 'json', 'description': '', 'type': <DataType.JSON: 23>}, {'name': 'var_a......  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : True  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['restore_backup_UrnJO7cV', {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'key', 'description': '', 'type': <DataType.INT64: 5>}, {'name': 'json', 'description': '', 'type': <DataType......, kwargs: {'consistency_level': 'Strong', 'shards_num': 2} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : <Collection>:

[2024-12-13T05:23:50.878Z] -------------

[2024-12-13T05:23:50.878Z] <name>: restore_backup_UrnJO7cV

[2024-12-13T05:23:50.878Z] <description>: 

[2024-12-13T05:23:50.878Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'key', 'description': '', 'type': <DataType.INT64: ......  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76,......, kwargs: {'timeout': 120} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_response) : (insert count: 3000, delete count: 0, upsert count: 0, timestamp: 454575357152198660, success count: 3000, err count: 0  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - INFO - ci_test]: [test][2024-12-13T05:22:51Z] [0.18706728s] restore_backup_UrnJO7cV insert -> (insert count: 3000, delete count: 0, upsert count: 0, timestamp: 454575357152198660, success count: 3000, err count: 0 (wrapper.py:32)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:51 - DEBUG - ci_test]: (api_request)  : [Collection.upsert] args: [[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76,......, kwargs: {'timeout': 120} (api_request.py:69)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:52 - DEBUG - ci_test]: (api_response) : (insert count: 100, delete count: 100, upsert count: 100, timestamp: 454575357165305857, success count: 100, err count: 0  (api_request.py:34)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:52 - INFO - ci_test]: [test][2024-12-13T05:22:51Z] [0.05387470s] restore_backup_UrnJO7cV upsert -> (insert count: 100, delete count: 100, upsert count: 100, timestamp: 454575357165305857, success count: 100, err count: 0 (wrapper.py:32)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:52 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.255.135.85:8080 (connectionpool.py:246)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:55 - DEBUG - urllib3.connectionpool]: http://10.255.135.85:8080/ "POST /api/v1/create HTTP/1.1" 200 262 (connectionpool.py:475)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:55 - DEBUG - ci_test]: 

[2024-12-13T05:23:50.878Z] method: post, 

[2024-12-13T05:23:50.878Z] url: http://10.255.135.85:8080/api/v1/create, 

[2024-12-13T05:23:50.878Z] cost time: 2.9773473739624023, 

[2024-12-13T05:23:50.878Z] header: {'Content-Type': 'application/json', 'RequestId': '4cc806be-b912-11ef-b9c5-fe032536f686'}, 

[2024-12-13T05:23:50.878Z] payload: {"async": false, "backup_name": "backup_yHPcOAJW", "collection_names": ["restore_backup_UrnJO7cV"]}, 

[2024-12-13T05:23:50.878Z] response: {"requestId":"4cc92808-b912-11ef-b312-5667c6f018e6","msg":"success","data":{"id":"4cc92808-b912-11ef-b312-5667c6f018e6","state_code":2,"start_time":1734067372059,"end_time":1734067375018,"progress":0,"name":"backup_yHPcOAJW","size":0,"milvus_version":"6ffc57c"}} (milvus_backup.py:24)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:55 - INFO - ci_test]: create_backup {'requestId': '4cc92808-b912-11ef-b312-5667c6f018e6', 'msg': 'success', 'data': {'id': '4cc92808-b912-11ef-b312-5667c6f018e6', 'state_code': 2, 'start_time': 1734067372059, 'end_time': 1734067375018, 'progress': 0, 'name': 'backup_yHPcOAJW', 'size': 0, 'milvus_version': '6ffc57c'}} (test_restore_backup.py:1125)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:55 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.255.135.85:8080 (connectionpool.py:246)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:55 - DEBUG - urllib3.connectionpool]: http://10.255.135.85:8080/ "GET /api/v1/list HTTP/1.1" 200 268 (connectionpool.py:475)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:55 - DEBUG - ci_test]: 

[2024-12-13T05:23:50.878Z] method: get, 

[2024-12-13T05:23:50.878Z] url: http://10.255.135.85:8080/api/v1/list, 

[2024-12-13T05:23:50.878Z] cost time: 0.007267475128173828, 

[2024-12-13T05:23:50.878Z] header: {'Content-Type': 'application/json', 'RequestId': '4e8e68d0-b912-11ef-b9c5-fe032536f686'}, 

[2024-12-13T05:23:50.878Z] payload: null, 

[2024-12-13T05:23:50.878Z] response: {"requestId":"4e8f7c2e-b912-11ef-b312-5667c6f018e6","msg":"success","data":[{"id":"4cc92808-b912-11ef-b312-5667c6f018e6","state_code":2,"start_time":1734067372059,"end_time":1734067375018,"progress":0,"name":"backup_yHPcOAJW","size":24204,"milvus_version":"6ffc57c"}]} (milvus_backup.py:24)

[2024-12-13T05:23:50.878Z] [2024-12-13 05:22:55 - INFO - ci_test]: list_backup {'requestId': '4e8f7c2e-b912-11ef-b312-5667c6f018e6', 'msg': 'success', 'data': [{'id': '4cc92808-b912-11ef-b312-5667c6f018e6', 'state_code': 2, 'start_time': 1734067372059, 'end_time': 1734067375018, 'progress': 0, 'name': 'backup_yHPcOAJW', 'size': 24204, 'milvus_version': '6ffc57c'}]} (test_restore_backup.py:1127)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:22:55 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.255.135.85:8080 (connectionpool.py:246)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:22:55 - DEBUG - urllib3.connectionpool]: http://10.255.135.85:8080/ "GET /api/v1/get_backup?backup_name=backup_yHPcOAJW HTTP/1.1" 200 1577 (connectionpool.py:475)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:22:55 - DEBUG - ci_test]: 

[2024-12-13T05:23:50.879Z] method: get, 

[2024-12-13T05:23:50.879Z] url: http://10.255.135.85:8080/api/v1/get_backup, 

[2024-12-13T05:23:50.879Z] cost time: 0.006150245666503906, 

[2024-12-13T05:23:50.879Z] header: {'Content-Type': 'application/json', 'RequestId': '4e8f9692-b912-11ef-b9c5-fe032536f686'}, 

[2024-12-13T05:23:50.879Z] payload: null, 

[2024-12-13T05:23:50.879Z] response: {"requestId":"4e90ad7c-b912-11ef-b312-5667c6f018e6","msg":"success","data":{"id":"4cc92808-b912-11ef-b312-5667c6f018e6","state_code":2,"start_time":1734067372059,"end_time":1734067375018,"progress":100,"name":"backup_yHPcOAJW","collection_backups":[{"id":"4cc92808-b912-11ef-b312-5667c6f018e6","state_code":0,"start_time":1734067372,"end_time":1734067375,"progress":0,"collection_id":454573997852071662,"db_name":"default","collection_name":"restore_backup_UrnJO7cV","schema":{"name":"restore_backup_UrnJO7cV","fields":[{"fieldID":100,"name":"int64","is_primary_key":true,"data_type":5},{"fieldID":101,"name":"key","data_type":5},{"fieldID":102,"name":"json","data_type":23},{"fieldID":103,"name":"var_array","data_type":22,"type_params":[{"key":"max_length","value":"1500"},{"key":"max_capacity","value":"2000"}],"element_type":21},{"fieldID":104,"name":"int_array","data_type":22,"type_params":[{"key":"max_length","value":"1500"},{"key":"max_capacity","value":"2000"}],"element_type":5},{"fieldID":105,"name":"float_vector","data_type":101,"type_params":[{"key":"dim","value":"128"}]}]},"shards_num":2,"backup_timestamp":454575357060448263,"size":24204,"has_index":false,"load_state":"NotLoad","backup_physical_timestamp":1734067372,"channel_checkpoints":{"by-dev-rootcoord-dml_6_454573997852071662v0":"CitieS1kZXYtcm9vdGNvb3JkLWRtbF82XzQ1NDU3Mzk5Nzg1MjA3MTY2MnYwEghoVAAAAAAAACCHgPDJj7y+pwY=","by-dev-rootcoord-dml_7_454573997852071662v1":"CitieS1kZXYtcm9vdGNvb3JkLWRtbF83XzQ1NDU3Mzk5Nzg1MjA3MTY2MnYxEghoVAAAAAAAACCHgPDJj7y+pwY="}}],"size":24204,"milvus_version":"6ffc57c"}} (milvus_backup.py:24)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:22:55 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.255.135.85:8080 (connectionpool.py:246)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - DEBUG - urllib3.connectionpool]: http://10.255.135.85:8080/ "POST /api/v1/restore HTTP/1.1" 200 492 (connectionpool.py:475)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - DEBUG - ci_test]: 

[2024-12-13T05:23:50.879Z] method: post, 

[2024-12-13T05:23:50.879Z] url: http://10.255.135.85:8080/api/v1/restore, 

[2024-12-13T05:23:50.879Z] cost time: 40.111679553985596, 

[2024-12-13T05:23:50.879Z] header: {'Content-Type': 'application/json', 'RequestId': '4e909510-b912-11ef-b9c5-fe032536f686'}, 

[2024-12-13T05:23:50.879Z] payload: {"async": false, "backup_name": "backup_yHPcOAJW", "collection_names": ["restore_backup_UrnJO7cV"], "collection_suffix": "_bak"}, 

[2024-12-13T05:23:50.879Z] response: {"requestId":"4e91ae80-b912-11ef-b312-5667c6f018e6","msg":"success","data":{"id":"restore_2024_12_13_05_22_55_49648246","state_code":0,"start_time":1734067375,"end_time":1734067415,"collection_restore_tasks":[{"id":"4e9251c8-b912-11ef-b312-5667c6f018e6","state_code":2,"start_time":1734067375,"target_collection_name":"restore_backup_UrnJO7cV_bak","restored_size":22681,"to_restore_size":24204,"progress":0,"target_db_name":"default"}],"restored_size":0,"to_restore_size":24204,"progress":0}} (milvus_backup.py:24)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - INFO - ci_test]: restore_backup: {'requestId': '4e91ae80-b912-11ef-b312-5667c6f018e6', 'msg': 'success', 'data': {'id': 'restore_2024_12_13_05_22_55_49648246', 'state_code': 0, 'start_time': 1734067375, 'end_time': 1734067415, 'collection_restore_tasks': [{'id': '4e9251c8-b912-11ef-b312-5667c6f018e6', 'state_code': 2, 'start_time': 1734067375, 'target_collection_name': 'restore_backup_UrnJO7cV_bak', 'restored_size': 22681, 'to_restore_size': 24204, 'progress': 0, 'target_db_name': 'default'}], 'restored_size': 0, 'to_restore_size': 24204, 'progress': 0}} (test_restore_backup.py:1147)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - DEBUG - ci_test]: (api_request)  : [list_collections] args: [20, 'default'], kwargs: {} (api_request.py:69)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - DEBUG - ci_test]: (api_response) : ['restore_backup_TL26WNbh_bak', 'restore_backup_5UolN8or_bak', 'restore_backup_yFnv3k2W_bak', 'restore_backup_HKw2ArDo_bak', 'restore_backup_VEOB1Tjx_bak', 'restore_backup_UrnJO7cV', 'restore_backup_KhvI8SYQ_bak', 'restore_backup_aLrw4rZn_bak', 'restore_backup_CaH6K9xE_bak', 'restore_backup_zaea7BfL......  (api_request.py:34)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['restore_backup_UrnJO7cV', None, 'default'], kwargs: {'consistency_level': 'Strong', 'shards_num': 2} (api_request.py:69)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - DEBUG - ci_test]: (api_response) : <Collection>:

[2024-12-13T05:23:50.879Z] -------------

[2024-12-13T05:23:50.879Z] <name>: restore_backup_UrnJO7cV

[2024-12-13T05:23:50.879Z] <description>: 

[2024-12-13T05:23:50.879Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'key', 'description': '', 'type': <DataType.INT64: ......  (api_request.py:34)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['restore_backup_UrnJO7cV_bak', None, 'default'], kwargs: {'consistency_level': 'Strong', 'shards_num': 2} (api_request.py:69)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - DEBUG - ci_test]: (api_response) : <Collection>:

[2024-12-13T05:23:50.879Z] -------------

[2024-12-13T05:23:50.879Z] <name>: restore_backup_UrnJO7cV_bak

[2024-12-13T05:23:50.879Z] <description>: 

[2024-12-13T05:23:50.879Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'key', 'description': '', 'type': <DataType.INT......  (api_request.py:34)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - INFO - ci_test]: collection_src schema: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'key', 'description': '', 'type': <DataType.INT64: 5>}, {'name': 'json', 'description': '', 'type': <DataType.JSON: 23>}, {'name': 'var_array', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 1500, 'max_capacity': 2000}, 'element_type': <DataType.VARCHAR: 21>}, {'name': 'int_array', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 1500, 'max_capacity': 2000}, 'element_type': <DataType.INT64: 5>}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}], 'enable_dynamic_field': False} (client_base.py:523)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - INFO - ci_test]: collection_dist schema: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'key', 'description': '', 'type': <DataType.INT64: 5>}, {'name': 'json', 'description': '', 'type': <DataType.JSON: 23>}, {'name': 'var_array', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 1500, 'max_capacity': 2000}, 'element_type': <DataType.VARCHAR: 21>}, {'name': 'int_array', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 1500, 'max_capacity': 2000}, 'element_type': <DataType.INT64: 5>}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}], 'enable_dynamic_field': False} (client_base.py:524)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - INFO - ci_test]: partitions_src: [{"name":"_default","collection_name":"restore_backup_UrnJO7cV","description":""}], partitions_dist: [{"name":"_default","collection_name":"restore_backup_UrnJO7cV_bak","description":""}] (client_base.py:529)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:35 - INFO - ci_test]: src_num: 3100, dist_num: 1554 (client_base.py:536)

[2024-12-13T05:23:50.879Z] [2024-12-13 05:23:49 - INFO - ci_test]: src count: data: ["{'count(*)': 3000}"] , dist count: data: ["{'count(*)': 1497}"]  (client_base.py:562)

Expected Behavior

No response

Steps To Reproduce

No response

Environment

milvus: master-20241213-6ffc57c8-amd64
backup: e22a744
failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/milvus_backup_api_test/detail/milvus_backup_api_test/101/pipeline

Anything else?

backup log: https://grafana-4am.zilliz.cc/explore?panes=%7B%22nR5%22:%7B%22datasource%22:%22vhI6Vw67k%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bpod%3D%5C%22milvus-backup-101-6f77d7f984-f9sc5%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22vhI6Vw67k%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221734061880847%22,%22to%22:%221734067692253%22%7D%7D%7D&schemaVersion=1&orgId=1

The text was updated successfully, but these errors were encountered:

bigsheeper · 2024-12-16T09:25:04Z

There are two segments, but Milvus received an import request for only one segment.

bigsheeper · 2024-12-16T10:01:49Z

During the backup, a stats task was completed, but the newly stats-ed segment did not trigger the import.

zhuwenxing assigned huanghaoyuanhhy Dec 16, 2024

bigsheeper mentioned this issue Dec 16, 2024

[Bug]: The number of entities after backup recovery is inconsistent with the number of entities in the source collection, and no delete or upsert operations have been performed on the collection #475

Closed

huanghaoyuanhhy mentioned this issue Dec 17, 2024

use segment info after flush #482

Merged

zhuwenxing closed this as completed in #482 Dec 17, 2024

huanghaoyuanhhy mentioned this issue Dec 18, 2024

[Bug]: After backup\restore no rows in collections #479

Closed

huanghaoyuanhhy mentioned this issue Jan 17, 2025

[Bug]: After an upsert operation on the source collection, the count of the collection obtained through backup and recovery is not equal to the count of the source collection. #456

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: source collection has upsert operations, the recovered data will be less than the original data #478

[Bug]: source collection has upsert operations, the recovered data will be less than the original data #478

zhuwenxing commented Dec 13, 2024

bigsheeper commented Dec 16, 2024

bigsheeper commented Dec 16, 2024

[Bug]: source collection has upsert operations, the recovered data will be less than the original data #478

[Bug]: source collection has upsert operations, the recovered data will be less than the original data #478

Comments

zhuwenxing commented Dec 13, 2024

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

Anything else?

bigsheeper commented Dec 16, 2024

bigsheeper commented Dec 16, 2024