[Bug]: After backup\restore no rows in collections #479
Comments
|
Thank you for reporting the issue! Please try updating to the latest release, and let us know if you're still encountering any problems with missing rows during data restoration. We're here to help! |
|
I tested with new version: I tried from scratch recreate backup and restore, still in restored cluster 0 rows in collections. |
|
I reviewed the logs you provided and noticed that the row count during the restoration process is indeed 0. Could you please check the Additionally, I carefully examined the data insertion script you shared and found that it might not be inserting data into the test database as intended. According to the PyMilvus documentation on managing databases, the database selection can be done either by using the Let us know your findings so we can assist you further. |
|
@huanghaoyuanhhy I checked our scripts, and it seems I shared our old version of creating script which didn't have right connecting to db. But we already fixed this and all of the testes above was done with correct connection method. I can also confirm, that num_of_rows from backup file has right number of rows. jq . full_meta.json | grep num_of_row
"num_of_rows": 5,
"num_of_rows": 5,
"num_of_rows": 5,
"num_of_rows": 5,
"num_of_rows": 5,Our latest python scripts which has right connection: Create:### RUN THIS IN PYTHON
# Import python modules
import numpy as np
import random
import string
from pymilvus import connections, db, utility, Collection, FieldSchema, DataType, CollectionSchema
### CREATE A DATABASE WITH RANDOM DATA ###
db_name='test_db'
print("Connecting to Milvus server 127.0.0.1:19530 ...")
connections.connect(host="127.0.0.1",port="19530",alias="default")
print(f"Creating database {db_name} if it doesn't exist ...")
if not db_name in db.list_database():
db.create_database(db_name)
print(f"Database {db_name} created")
else:
print(f"Database {db_name} already exists")
print("")
print(f"Connecting to database {db_name} ...")
connections.connect(host="127.0.0.1",port="19530",alias=db_name,db_name=db_name)
print(f"Collecting database info ...")
existing_collections = utility.list_collections(using=db_name)
if existing_collections:
for existing_collection in existing_collections:
ec = Collection(existing_collection,using=db_name, consistency_level="Strong")
print(f"Existing collection {ec.name} has {ec.num_entities} rows")
print("")
else:
print("No collections exist")
print("")
num_collections = 5
print(f"Creating {num_collections} new collections in database {db_name} ...")
collections = []
primary_key = FieldSchema(name="id",dtype=DataType.INT64,is_primary=True)
vector = FieldSchema(name="vector",dtype=DataType.FLOAT_VECTOR,dim=768)
schema = CollectionSchema(fields = [primary_key, vector])
index_params = {"index_type": "IVF_FLAT","metric_type": "IP","params": {"nlist": 128}}
for i in range(num_collections):
collection_name = 'test_col_' + ''.join(random.choice(string.ascii_lowercase) for i in range(7))
collection = Collection(name=collection_name,schema=schema,using=db_name)
collection.create_index(field_name="id")
collection.create_index(field_name="vector",index_params=index_params,timeout=None)
collections.append(collection)
print(f"Collection {collection_name} created")
num_rows = 5
print(f"Inserting {num_rows} new random data rows into new collections ...")
for collection in collections:
data = []
for i in range(num_rows):
data.append({"id": i, "vector": np.random.random(768)})
collection.insert(data)
collection.flush()
print(f"{num_rows} rows inserted into collection {collection.name}")
print("Done")Validate:from pymilvus import connections, db, utility, Collection
db_name = 'test_db'
print("Connecting to Milvus server 127.0.0.1:19530 ...")
connections.connect(host="127.0.0.1",port="19530",alias="default",db_name=db_name)
print(f"Checking if database {db_name} exists ...")
if db_name in db.list_database():
print(f"Database {db_name} exists")
else:
print(f"Database {db_name} does NOT exist")
exit(1)
print(f"Connecting to database {db_name} ...")
connections.connect(host="127.0.0.1",port="19530",alias=db_name,db_name=db_name)
collection_names = utility.list_collections(using=db_name)
print(f"Database {db_name} has {len(collection_names)} collections in it")
for collection_name in collection_names:
c=Collection(collection_name,using=db_name)
c.flush()
print(f"Collection {c.name} has {c.num_entities} rows in it") |
|
This is our backup: |
|
I tested with the new creating script you provided, and everything works fine—backup and restore are both good. However, when I restored from the backup you uploaded, I was able to reproduce the issue with the row count being 0. I noticed the
|
|
@huanghaoyuanhhy Also I can confirm that we don't use So it seems we have some issue with backup creation part. log/app # ./milvus-backup create
v0.4.28 (Built on 2024-12-18T03:43:07Z from Git SHA 6b29a3a)
config:backup.yaml
[2024/12/20 21:15:38.727 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/12/20 21:15:38.727 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2024/12/20 21:15:38.727 +00:00] [INFO] [core/backup_impl_create_backup.go:32] ["receive CreateBackupRequest"] [requestId=8fab045e-bf17-11ef-9f68-e61e7a4ebc10] [backupName=] [collections="[]"] [databaseCollections=] [async=false] [force=false] [metaOnly=false]
[2024/12/20 21:15:38.727 +00:00] [INFO] [core/backup_context.go:117] ["{Base:0xc0000e8dc0 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://localhost:9091/}"]
[2024/12/20 21:15:38.728 +00:00] [INFO] [core/backup_context.go:118] ["{Base:0xc0000e8dc0 Enabled:true DebugMode:false SimpleResponse:true}"]
[2024/12/20 21:15:38.760 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=pcz-milvus-backup] [root=backup]
[2024/12/20 21:15:38.777 +00:00] [WARN] [core/backup_impl_create_backup.go:601] ["Pause Milvus GC Error:This warn won't fail the backup process. Pause GC can protect data not to be GCed during backup, it is necessary to backup very large data(cost more than a hour)."] [error="Get \http://localhost:9091/management/datacoord/garbage_collection/pause?pause_seconds=7200\: dial tcp [::1]:9091: connect: connection refused"]
[2024/12/20 21:15:38.783 +00:00] [INFO] [core/backup_impl_create_backup.go:667] ["collections to backup"] [collections="[test_db.test_col_ihroxqt,test_db.test_col_wnzevip,test_db.test_col_ybosnxc,test_db.test_col_yzpbkct,test_db.test_col_vmymifi]"]
[2024/12/20 21:15:38.783 +00:00] [INFO] [core/backup_impl_create_backup.go:231] ["start backup collection"] [db=test_db] [collection=test_col_ihroxqt]
[2024/12/20 21:15:38.785 +00:00] [INFO] [core/backup_impl_create_backup.go:231] ["start backup collection"] [db=test_db] [collection=test_col_ybosnxc]
[2024/12/20 21:15:38.786 +00:00] [INFO] [core/backup_impl_create_backup.go:231] ["start backup collection"] [db=test_db] [collection=test_col_yzpbkct]
[2024/12/20 21:15:38.786 +00:00] [INFO] [core/backup_impl_create_backup.go:231] ["start backup collection"] [db=test_db] [collection=test_col_wnzevip]
[2024/12/20 21:15:38.787 +00:00] [INFO] [core/backup_impl_create_backup.go:274] ["try to get index"] [collection_name=test_col_ihroxqt]
[2024/12/20 21:15:38.791 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_ihroxqt] [field_name=id] ["index info"="[{}]"]
[2024/12/20 21:15:38.794 +00:00] [INFO] [core/backup_impl_create_backup.go:274] ["try to get index"] [collection_name=test_col_ybosnxc]
[2024/12/20 21:15:38.796 +00:00] [INFO] [core/backup_impl_create_backup.go:274] ["try to get index"] [collection_name=test_col_yzpbkct]
[2024/12/20 21:15:38.798 +00:00] [INFO] [core/backup_impl_create_backup.go:274] ["try to get index"] [collection_name=test_col_wnzevip]
[2024/12/20 21:15:38.802 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_ihroxqt] [field_name=vector] ["index info"="[{}]"]
[2024/12/20 21:15:38.807 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_ybosnxc] [field_name=id] ["index info"="[{}]"]
[2024/12/20 21:15:38.811 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_yzpbkct] [field_name=id] ["index info"="[{}]"]
[2024/12/20 21:15:38.815 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_wnzevip] [field_name=id] ["index info"="[{}]"]
[2024/12/20 21:15:38.821 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_ybosnxc] [field_name=vector] ["index info"="[{}]"]
[2024/12/20 21:15:38.825 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_yzpbkct] [field_name=vector] ["index info"="[{}]"]
[2024/12/20 21:15:38.829 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_wnzevip] [field_name=vector] ["index info"="[{}]"]
[2024/12/20 21:15:38.843 +00:00] [INFO] [core/backup_impl_create_backup.go:380] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=test_db] [collectionName=test_col_ihroxqt] [segmentNumBeforeFlush=1]
[2024/12/20 21:15:38.857 +00:00] [INFO] [core/backup_impl_create_backup.go:408] ["flush segments"] [databaseName=test_db] [collectionName=test_col_ihroxqt] [newSealedSegmentIDs="[]"] [flushedSegmentIDs="[454569269145510753]"] [timeOfSeal=1734729338] [BackupTimestamp=0] [channelCPs="{\"by-dev-rootcoord-dml_1_454569269145110233v0\":{\"channel_name\":\"by-dev-rootcoord-dml_1_454569269145110233v0\",\"msgID\":\"/z6Mjmz0TgY=\",\"msgGroup\":\"datanode-2-by-dev-rootcoord-dml_1_454569269145110233v0-true\",\"timestamp\":454748876637732865}}"]
[2024/12/20 21:15:38.859 +00:00] [INFO] [core/backup_impl_create_backup.go:380] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=test_db] [collectionName=test_col_ybosnxc] [segmentNumBeforeFlush=1]
[2024/12/20 21:15:38.862 +00:00] [INFO] [core/backup_impl_create_backup.go:380] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=test_db] [collectionName=test_col_yzpbkct] [segmentNumBeforeFlush=1]
[2024/12/20 21:15:38.864 +00:00] [INFO] [core/backup_impl_create_backup.go:380] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=test_db] [collectionName=test_col_wnzevip] [segmentNumBeforeFlush=1]
[2024/12/20 21:15:38.866 +00:00] [INFO] [core/backup_impl_create_backup.go:429] ["GetPersistentSegmentInfo after flush from milvus"] [databaseName=test_db] [collectionName=test_col_ihroxqt] [segmentIDsBeforeFlush="[454569269145510753]"] [segmentIDsAfterFlush="[454569269145510753]"] [newL0SegmentsIDs="[]"]
[2024/12/20 21:15:38.866 +00:00] [INFO] [core/backup_impl_create_backup.go:497] ["finish build partition info"] [collectionName=test_col_ihroxqt] [partitionNum=1] [l0SegmentsNum=0]
[2024/12/20 21:15:38.866 +00:00] [INFO] [core/backup_impl_create_backup.go:231] ["start backup collection"] [db=test_db] [collection=test_col_vmymifi]
[2024/12/20 21:15:38.870 +00:00] [INFO] [core/backup_impl_create_backup.go:408] ["flush segments"] [databaseName=test_db] [collectionName=test_col_ybosnxc] [newSealedSegmentIDs="[]"] [flushedSegmentIDs="[454569269145511617]"] [timeOfSeal=1734729338] [BackupTimestamp=0] [channelCPs="{\"by-dev-rootcoord-dml_3_454569269145110270v0\":{\"channel_name\":\"by-dev-rootcoord-dml_3_454569269145110270v0\",\"msgID\":\"9D60r2z0TgY=\",\"msgGroup\":\"datanode-2-by-dev-rootcoord-dml_3_454569269145110270v0-true\",\"timestamp\":454748876637732865}}"]
[2024/12/20 21:15:38.873 +00:00] [INFO] [core/backup_impl_create_backup.go:408] ["flush segments"] [databaseName=test_db] [collectionName=test_col_yzpbkct] [newSealedSegmentIDs="[]"] [flushedSegmentIDs="[454569269145512044]"] [timeOfSeal=1734729338] [BackupTimestamp=0] [channelCPs="{\"by-dev-rootcoord-dml_4_454569269145110288v0\":{\"channel_name\":\"by-dev-rootcoord-dml_4_454569269145110288v0\",\"msgID\":\"7z5YwGz0TgY=\",\"msgGroup\":\"datanode-2-by-dev-rootcoord-dml_4_454569269145110288v0-true\",\"timestamp\":454748876637732865}}"]
[2024/12/20 21:15:38.876 +00:00] [INFO] [core/backup_impl_create_backup.go:408] ["flush segments"] [databaseName=test_db] [collectionName=test_col_wnzevip] [newSealedSegmentIDs="[]"] [flushedSegmentIDs="[454569269145511184]"] [timeOfSeal=1734729338] [BackupTimestamp=0] [channelCPs="{\"by-dev-rootcoord-dml_2_454569269145110252v0\":{\"channel_name\":\"by-dev-rootcoord-dml_2_454569269145110252v0\",\"msgID\":\"+T4on2z0TgY=\",\"msgGroup\":\"datanode-2-by-dev-rootcoord-dml_2_454569269145110252v0-true\",\"timestamp\":454748876637732865}}"]
[2024/12/20 21:15:38.879 +00:00] [INFO] [core/backup_impl_create_backup.go:274] ["try to get index"] [collection_name=test_col_vmymifi]
[2024/12/20 21:15:38.881 +00:00] [INFO] [core/backup_impl_create_backup.go:429] ["GetPersistentSegmentInfo after flush from milvus"] [databaseName=test_db] [collectionName=test_col_ybosnxc] [segmentIDsBeforeFlush="[454569269145511617]"] [segmentIDsAfterFlush="[454569269145511617]"] [newL0SegmentsIDs="[]"]
[2024/12/20 21:15:38.881 +00:00] [INFO] [core/backup_impl_create_backup.go:497] ["finish build partition info"] [collectionName=test_col_ybosnxc] [partitionNum=1] [l0SegmentsNum=0]
[2024/12/20 21:15:38.887 +00:00] [INFO] [core/backup_impl_create_backup.go:429] ["GetPersistentSegmentInfo after flush from milvus"] [databaseName=test_db] [collectionName=test_col_yzpbkct] [segmentIDsBeforeFlush="[454569269145512044]"] [segmentIDsAfterFlush="[454569269145512044]"] [newL0SegmentsIDs="[]"]
[2024/12/20 21:15:38.888 +00:00] [INFO] [core/backup_impl_create_backup.go:497] ["finish build partition info"] [collectionName=test_col_yzpbkct] [partitionNum=1] [l0SegmentsNum=0]
[2024/12/20 21:15:38.890 +00:00] [INFO] [core/backup_impl_create_backup.go:429] ["GetPersistentSegmentInfo after flush from milvus"] [databaseName=test_db] [collectionName=test_col_wnzevip] [segmentIDsBeforeFlush="[454569269145511184]"] [segmentIDsAfterFlush="[454569269145511184]"] [newL0SegmentsIDs="[]"]
[2024/12/20 21:15:38.890 +00:00] [INFO] [core/backup_impl_create_backup.go:497] ["finish build partition info"] [collectionName=test_col_wnzevip] [partitionNum=1] [l0SegmentsNum=0]
[2024/12/20 21:15:38.894 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_vmymifi] [field_name=id] ["index info"="[{}]"]
[2024/12/20 21:15:38.898 +00:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=test_col_vmymifi] [field_name=vector] ["index info"="[{}]"]
[2024/12/20 21:15:38.905 +00:00] [INFO] [core/backup_impl_create_backup.go:380] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=test_db] [collectionName=test_col_vmymifi] [segmentNumBeforeFlush=1]
[2024/12/20 21:15:38.908 +00:00] [INFO] [core/backup_impl_create_backup.go:408] ["flush segments"] [databaseName=test_db] [collectionName=test_col_vmymifi] [newSealedSegmentIDs="[]"] [flushedSegmentIDs="[454569269145510326]"] [timeOfSeal=1734729338] [BackupTimestamp=0] [channelCPs="{\"by-dev-rootcoord-dml_0_454569269145110215v0\":{\"channel_name\":\"by-dev-rootcoord-dml_0_454569269145110215v0\",\"msgID\":\"BD+4fWz0TgY=\",\"msgGroup\":\"datanode-2-by-dev-rootcoord-dml_0_454569269145110215v0-true\",\"timestamp\":454748876637732865}}"]
[2024/12/20 21:15:38.911 +00:00] [INFO] [core/backup_impl_create_backup.go:429] ["GetPersistentSegmentInfo after flush from milvus"] [databaseName=test_db] [collectionName=test_col_vmymifi] [segmentIDsBeforeFlush="[454569269145510326]"] [segmentIDsAfterFlush="[454569269145510326]"] [newL0SegmentsIDs="[]"]
[2024/12/20 21:15:38.911 +00:00] [INFO] [core/backup_impl_create_backup.go:497] ["finish build partition info"] [collectionName=test_col_vmymifi] [partitionNum=1] [l0SegmentsNum=0]
[2024/12/20 21:15:38.911 +00:00] [INFO] [core/backup_impl_create_backup.go:701] ["Finish prepare all collections meta"]
[2024/12/20 21:15:38.911 +00:00] [INFO] [core/backup_impl_create_backup.go:706] ["before backupCollectionExecute"] [collectionID=454569269145110288] [collection=test_col_yzpbkct]
[2024/12/20 21:15:38.911 +00:00] [INFO] [core/backup_impl_create_backup.go:706] ["before backupCollectionExecute"] [collectionID=454569269145110252] [collection=test_col_wnzevip]
[2024/12/20 21:15:38.911 +00:00] [INFO] [core/backup_impl_create_backup.go:706] ["before backupCollectionExecute"] [collectionID=454569269145110215] [collection=test_col_vmymifi]
[2024/12/20 21:15:38.911 +00:00] [INFO] [core/backup_impl_create_backup.go:706] ["before backupCollectionExecute"] [collectionID=454569269145110233] [collection=test_col_ihroxqt]
[2024/12/20 21:15:38.911 +00:00] [INFO] [core/backup_impl_create_backup.go:706] ["before backupCollectionExecute"] [collectionID=454569269145110270] [collection=test_col_ybosnxc]
[2024/12/20 21:15:38.911 +00:00] [INFO] [core/backup_impl_create_backup.go:512] [backupCollectionExecute] [collectionMeta="id:\"8fab045e-bf17-11ef-9f68-e61e7a4ebc10\" start_time:1734729338 collection_id:454569269145110252 db_name:\"test_db\" collection_name:\"test_col_wnzevip\" schema:<name:\"test_col_wnzevip\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"768\" > > > shards_num:1 consistency_level:Bounded backup_timestamp:454748876637732865 has_index:true index_infos:<field_name:\"id\" index_name:\"id\" > index_infos:<field_name:\"vector\" index_name:\"vector\" index_type:\"IVF_FLAT\" params:<key:\"index_type\" value:\"IVF_FLAT\" > params:<key:\"metric_type\" value:\"IP\" > params:<key:\"params\" value:\"{\\\"nlist\\\":128}\" > > load_state:\"NotLoad\" backup_physical_timestamp:1734729338 channel_checkpoints:<key:\"by-dev-rootcoord-dml_2_454569269145110252v0\" value:\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8yXzQ1NDU2OTI2OTE0NTExMDI1MnYwEgj5PiifbPROBho7ZGF0YW5vZGUtMi1ieS1kZXYtcm9vdGNvb3JkLWRtbF8yXzQ1NDU2OTI2OTE0NTExMDI1MnYwLXRydWUggYCAsJr25acG\" > "]
[2024/12/20 21:15:38.913 +00:00] [INFO] [core/backup_impl_create_backup.go:512] [backupCollectionExecute] [collectionMeta="id:\"8fab045e-bf17-11ef-9f68-e61e7a4ebc10\" start_time:1734729338 collection_id:454569269145110288 db_name:\"test_db\" collection_name:\"test_col_yzpbkct\" schema:<name:\"test_col_yzpbkct\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"768\" > > > shards_num:1 consistency_level:Bounded backup_timestamp:454748876637732865 has_index:true index_infos:<field_name:\"id\" index_name:\"id\" > index_infos:<field_name:\"vector\" index_name:\"vector\" index_type:\"IVF_FLAT\" params:<key:\"index_type\" value:\"IVF_FLAT\" > params:<key:\"metric_type\" value:\"IP\" > params:<key:\"params\" value:\"{\\\"nlist\\\":128}\" > > load_state:\"NotLoad\" backup_physical_timestamp:1734729338 channel_checkpoints:<key:\"by-dev-rootcoord-dml_4_454569269145110288v0\" value:\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF80XzQ1NDU2OTI2OTE0NTExMDI4OHYwEgjvPljAbPROBho7ZGF0YW5vZGUtMi1ieS1kZXYtcm9vdGNvb3JkLWRtbF80XzQ1NDU2OTI2OTE0NTExMDI4OHYwLXRydWUggYCAsJr25acG\" > "]
[2024/12/20 21:15:38.913 +00:00] [INFO] [core/backup_impl_create_backup.go:512] [backupCollectionExecute] [collectionMeta="id:\"8fab045e-bf17-11ef-9f68-e61e7a4ebc10\" start_time:1734729338 collection_id:454569269145110233 db_name:\"test_db\" collection_name:\"test_col_ihroxqt\" schema:<name:\"test_col_ihroxqt\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"768\" > > > shards_num:1 consistency_level:Bounded backup_timestamp:454748876637732865 has_index:true index_infos:<field_name:\"id\" index_name:\"id\" > index_infos:<field_name:\"vector\" index_name:\"vector\" index_type:\"IVF_FLAT\" params:<key:\"index_type\" value:\"IVF_FLAT\" > params:<key:\"metric_type\" value:\"IP\" > params:<key:\"params\" value:\"{\\\"nlist\\\":128}\" > > load_state:\"NotLoad\" backup_physical_timestamp:1734729338 channel_checkpoints:<key:\"by-dev-rootcoord-dml_1_454569269145110233v0\" value:\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8xXzQ1NDU2OTI2OTE0NTExMDIzM3YwEgj/PoyObPROBho7ZGF0YW5vZGUtMi1ieS1kZXYtcm9vdGNvb3JkLWRtbF8xXzQ1NDU2OTI2OTE0NTExMDIzM3YwLXRydWUggYCAsJr25acG\" > "]
[2024/12/20 21:15:38.915 +00:00] [INFO] [core/backup_impl_create_backup.go:512] [backupCollectionExecute] [collectionMeta="id:\"8fab045e-bf17-11ef-9f68-e61e7a4ebc10\" start_time:1734729338 collection_id:454569269145110215 db_name:\"test_db\" collection_name:\"test_col_vmymifi\" schema:<name:\"test_col_vmymifi\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"768\" > > > shards_num:1 consistency_level:Bounded backup_timestamp:454748876637732865 has_index:true index_infos:<field_name:\"id\" index_name:\"id\" > index_infos:<field_name:\"vector\" index_name:\"vector\" index_type:\"IVF_FLAT\" params:<key:\"index_type\" value:\"IVF_FLAT\" > params:<key:\"metric_type\" value:\"IP\" > params:<key:\"params\" value:\"{\\\"nlist\\\":128}\" > > load_state:\"NotLoad\" backup_physical_timestamp:1734729338 channel_checkpoints:<key:\"by-dev-rootcoord-dml_0_454569269145110215v0\" value:\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8wXzQ1NDU2OTI2OTE0NTExMDIxNXYwEggEP7h9bPROBho7ZGF0YW5vZGUtMi1ieS1kZXYtcm9vdGNvb3JkLWRtbF8wXzQ1NDU2OTI2OTE0NTExMDIxNXYwLXRydWUggYCAsJr25acG\" > "]
[2024/12/20 21:15:38.928 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=pcz-milvus-a] [root=file]
[2024/12/20 21:15:38.930 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=pcz-milvus-a] [root=file]
[2024/12/20 21:15:38.937 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=pcz-milvus-a] [root=file]
[2024/12/20 21:15:38.937 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=pcz-milvus-a] [root=file]
[2024/12/20 21:15:38.957 +00:00] [INFO] [core/backup_impl_create_backup.go:547] ["Begin copy data"] [dbName=test_db] [collectionName=test_col_wnzevip] [collectionID=454569269145110252] [partitionID=454569269145110253]
[2024/12/20 21:15:38.957 +00:00] [INFO] [core/backup_impl_create_backup.go:800] ["copy segment"] [collection_id=454569269145110252] [partition_id=454569269145110253] [segment_id=454569269145511184] [group_id=454569269145511184] [backupBinlogPath=backup/backup_2024_12_20_21_15_38_728105727/binlogs]
[2024/12/20 21:15:38.957 +00:00] [INFO] [core/backup_impl_create_backup.go:587] ["Finish copy data"] [dbName=test_db] [collectionName=test_col_wnzevip]
[2024/12/20 21:15:38.957 +00:00] [INFO] [core/backup_impl_create_backup.go:512] [backupCollectionExecute] [collectionMeta="id:\"8fab045e-bf17-11ef-9f68-e61e7a4ebc10\" start_time:1734729338 collection_id:454569269145110270 db_name:\"test_db\" collection_name:\"test_col_ybosnxc\" schema:<name:\"test_col_ybosnxc\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"768\" > > > shards_num:1 consistency_level:Bounded backup_timestamp:454748876637732865 has_index:true index_infos:<field_name:\"id\" index_name:\"id\" > index_infos:<field_name:\"vector\" index_name:\"vector\" index_type:\"IVF_FLAT\" params:<key:\"index_type\" value:\"IVF_FLAT\" > params:<key:\"metric_type\" value:\"IP\" > params:<key:\"params\" value:\"{\\\"nlist\\\":128}\" > > load_state:\"NotLoad\" backup_physical_timestamp:1734729338 channel_checkpoints:<key:\"by-dev-rootcoord-dml_3_454569269145110270v0\" value:\"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8zXzQ1NDU2OTI2OTE0NTExMDI3MHYwEgj0PrSvbPROBho7ZGF0YW5vZGUtMi1ieS1kZXYtcm9vdGNvb3JkLWRtbF8zXzQ1NDU2OTI2OTE0NTExMDI3MHYwLXRydWUggYCAsJr25acG\" > "]
[2024/12/20 21:15:38.958 +00:00] [INFO] [core/backup_impl_create_backup.go:547] ["Begin copy data"] [dbName=test_db] [collectionName=test_col_vmymifi] [collectionID=454569269145110215] [partitionID=454569269145110216]
[2024/12/20 21:15:38.975 +00:00] [INFO] [core/backup_impl_create_backup.go:800] ["copy segment"] [collection_id=454569269145110215] [partition_id=454569269145110216] [segment_id=454569269145510326] [group_id=454569269145510326] [backupBinlogPath=backup/backup_2024_12_20_21_15_38_728105727/binlogs]
[2024/12/20 21:15:38.975 +00:00] [INFO] [core/backup_impl_create_backup.go:547] ["Begin copy data"] [dbName=test_db] [collectionName=test_col_ihroxqt] [collectionID=454569269145110233] [partitionID=454569269145110234]
[2024/12/20 21:15:38.983 +00:00] [INFO] [core/backup_impl_create_backup.go:587] ["Finish copy data"] [dbName=test_db] [collectionName=test_col_vmymifi]
[2024/12/20 21:15:38.993 +00:00] [INFO] [core/backup_impl_create_backup.go:800] ["copy segment"] [collection_id=454569269145110233] [partition_id=454569269145110234] [segment_id=454569269145510753] [group_id=454569269145510753] [backupBinlogPath=backup/backup_2024_12_20_21_15_38_728105727/binlogs]
[2024/12/20 21:15:38.994 +00:00] [INFO] [core/backup_impl_create_backup.go:547] ["Begin copy data"] [dbName=test_db] [collectionName=test_col_yzpbkct] [collectionID=454569269145110288] [partitionID=454569269145110289]
[2024/12/20 21:15:39.003 +00:00] [INFO] [core/backup_impl_create_backup.go:547] ["Begin copy data"] [dbName=test_db] [collectionName=test_col_ybosnxc] [collectionID=454569269145110270] [partitionID=454569269145110271]
[2024/12/20 21:15:39.004 +00:00] [INFO] [core/backup_impl_create_backup.go:587] ["Finish copy data"] [dbName=test_db] [collectionName=test_col_ihroxqt]
[2024/12/20 21:15:39.013 +00:00] [INFO] [core/backup_impl_create_backup.go:800] ["copy segment"] [collection_id=454569269145110270] [partition_id=454569269145110271] [segment_id=454569269145511617] [group_id=454569269145511617] [backupBinlogPath=backup/backup_2024_12_20_21_15_38_728105727/binlogs]
[2024/12/20 21:15:39.014 +00:00] [INFO] [core/backup_impl_create_backup.go:587] ["Finish copy data"] [dbName=test_db] [collectionName=test_col_ybosnxc]
[2024/12/20 21:15:39.033 +00:00] [INFO] [core/backup_impl_create_backup.go:800] ["copy segment"] [collection_id=454569269145110288] [partition_id=454569269145110289] [segment_id=454569269145512044] [group_id=454569269145512044] [backupBinlogPath=backup/backup_2024_12_20_21_15_38_728105727/binlogs]
[2024/12/20 21:15:39.034 +00:00] [INFO] [core/backup_impl_create_backup.go:587] ["Finish copy data"] [dbName=test_db] [collectionName=test_col_yzpbkct]
[2024/12/20 21:15:39.035 +00:00] [INFO] [core/backup_meta_manager.go:501] ["Get backup"] [state=BACKUP_SUCCESS] [backupedSize=0] [totalSize=0] [progress=100]
[2024/12/20 21:15:41.416 +00:00] [INFO] [core/backup_impl_create_backup.go:787] ["finish writeBackupInfoMeta"] [path=backup/backup_2024_12_20_21_15_38_728105727/] [backupName=backup_2024_12_20_21_15_38_728105727] ["backup meta"="{\"id\":\"8fab045e-bf17-11ef-9f68-e61e7a4ebc10\",\"state_code\":2,\"start_time\":1734729338776,\"end_time\":1734729339035,\"progress\":100,\"name\":\"backup_2024_12_20_21_15_38_728105727\",\"size\":0,\"milvus_version\":\"v2.5.0-beta\"}"]
[2024/12/20 21:15:41.416 +00:00] [INFO] [core/backup_impl_create_backup.go:744] ["finish backup all collections"] [requestId=8fab045e-bf17-11ef-9f68-e61e7a4ebc10] [backupName=backup_2024_12_20_21_15_38_728105727] [collections="[]"] [async=false]
[2024/12/20 21:15:41.416 +00:00] [WARN] [core/backup_impl_create_backup.go:619] ["Resume Milvus GC Error:This warn won't fail the backup process. Pause GC can protect data not to be GCed during backup, it is necessary to backup very large data(cost more than a hour)."] [error="Get \[http://localhost:9091/management/datacoord/garbage_collection/resume\](http://localhost:9091/management/datacoord/garbage_collection/resume/): dial tcp [::1]:9091: connect: connection refused"]
success
duration:3 s
/app #backup itselfbackup_2024_12_20_21_15_38_728105727.zip milvus backup config file with cut s3 credentials/app/configs # cat backup.yaml
# Configures the system log output.
log:
level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
console: true # whether print log to console
file:
rootPath: "logs/backup.log"
http:
simpleResponse: true
# milvus proxy address, compatible to milvus.yaml
milvus:
address: milvus-a
port: 19530
authorizationEnabled: false
# tls mode values [0, 1, 2]
# 0 is close, 1 is one-way authentication, 2 is two-way authentication.
tlsMode: 0
user: "root"
password: "Milvus"
# Related configuration of minio, which is responsible for data persistence for Milvus.
minio:
# Milvus storage configs, make them the same with milvus config
storageType: "s3" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
address: # Address of MinIO/S3
port: 8082 # Port of MinIO/S3
accessKeyID: # accessKeyID of MinIO/S3
secretAccessKey: "" # MinIO/S3 encryption string
useSSL: true # Access to MinIO/S3 with SSL
useIAM: false
iamEndpoint: ""
bucketName: "" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
rootPath: file # Milvus storage root path in MinIO/S3, make it the same as your milvus instance
# Backup storage configs, the storage you want to put the backup data
backupStorageType: "s3" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
backupAddress: # Address of MinIO/S3
backupPort: 8082 # Port of MinIO/S3
backupAccessKeyID: # accessKeyID of MinIO/S3
backupSecretAccessKey: "" # MinIO/S3 encryption string
backupBucketName: "" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
backupRootPath: "backup" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath
# If you need to back up or restore data between two different storage systems, direct client-side copying is not supported.
# Set this option to true to enable data transfer through Milvus Backup.
# Note: This option will be automatically set to true if `minio.storageType` and `minio.backupStorageType` differ.
# However, if they are the same but belong to different services, you must manually set this option to `true`.
crossStorage: "false"
backup:
maxSegmentGroupSize: 2G
parallelism:
# collection level parallelism to backup
backupCollection: 4
# thread pool to copy data. reduce it if blocks your storage's network bandwidth
copydata: 128
# Collection level parallelism to restore
restoreCollection: 2
# keep temporary files during restore, only use to debug
keepTempFiles: false
# Pause GC during backup through Milvus Http API.
gcPause:
enable: true
seconds: 7200
address: http://localhost:9091/app/configs #We created one more backup with And again reproduced that we don't have |
|
Thanks for the additional information. |
|
After reviewing the configuration and logs, and doing some testing, I suspect the issue might be related to the If this is the cause, it’s worth noting that milvus-backup didn’t catch this during the process. I’ll work on adding validation to address this in future updates. Could you try updating the configuration and let me know if it resolves the issue? |
|
@huanghaoyuanhhy Initially we took example from default values file of milvus helm chart: externalS3:
enabled: true
rootPath: ""So our After we set So definitely milvus backup should check whether it can copy valid bing log folder and throw error when it is missing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Current Behavior
We have DB with 5 collections and 5 rows in each collection, after backup\restore we can't see rows in restored cluster.
Expected Behavior
We expect to see the same rows after restore as in the source cluster
Steps To Reproduce
We have two just deployed milvus clusters with latest milvus version: source and destination
0. Scripts
0.1. Create DB python script:
0.2. Check DB python script:
1. Checking source before backup, which shows DB test_db has 5 collections in it with 5 rows in each
2 Creating backup at source cluster:
3. Restoring backup at destination cluster
4. Checking destination after restore shows 0 rows:
Environment
The text was updated successfully, but these errors were encountered: