Could not fetch frame data from the server (A manifest file not exists) #3119
Comments
|
@nstolyarov , Could you please describe in more detail the current task, from what data did you create this task, what parameters did you use? |
|
Some update: Seems it has happened after some update with |
|
I created this task from connected file share images with And yes, it was available earlier. Guess I created this task on the following commit: But I'm not sure, maybe it was on later commit. |
|
Update x2: It affects on this task and all the previous tasks. |
|
Could I ask you to check the data folder |
|
Hi, @Marishka17 ! There is nothing in this folder for this task and the same situation is for all previous tasks. But for other tasks everything is ok. |
|
|
|
Would generation a Manifest file on the fly if it does not exist be an option to fix? |
|
Follow below steps to create missing manifest files:
import glob
import os
from re import search
from django.conf import settings
from django.db import migrations
from cvat.apps.engine.models import DimensionType, StorageChoice,StorageMethodChoice
from utils.dataset_manifest import ImageManifestManager, VideoManifestManager
from cvat.apps.engine.models import Data
query_set = Data.objects.filter(storage_method=StorageMethodChoice.CACHE)
with open("exceptions_log.txt", "w") as log_file:
for db_data in query_set:
try:
upload_dir = '{}/{}/raw'.format(settings.MEDIA_DATA_ROOT, db_data.id)
if os.path.exists(os.path.join(upload_dir, 'meta_info.txt')):
os.remove(os.path.join(upload_dir, 'meta_info.txt'))
else:
for path in glob.glob(f'{upload_dir}/dummy_*.txt'):
os.remove(path)
if os.path.exists(os.path.join(upload_dir, 'manifest.jsonl')):
continue
data_dir = upload_dir if db_data.storage == StorageChoice.LOCAL else settings.SHARE_ROOT
if hasattr(db_data, 'video'):
media_file = os.path.join(data_dir, db_data.video.path)
manifest = VideoManifestManager(manifest_path=upload_dir)
meta_info = manifest.prepare_meta(media_file=media_file)
manifest.create(meta_info)
manifest.init_index()
else:
manifest = ImageManifestManager(manifest_path=upload_dir)
sources = []
if db_data.storage == StorageChoice.LOCAL:
for (root, _, files) in os.walk(data_dir):
sources.extend([os.path.join(root, f) for f in files])
sources.sort()
else:
sources = [os.path.join(data_dir, db_image.path) for db_image in db_data.images.all().order_by('frame')]
if any(list(filter(lambda x: x.dimension==DimensionType.DIM_3D, db_data.tasks.all()))):
content = []
for source in sources:
name, ext = os.path.splitext(os.path.relpath(source, upload_dir))
content.append({
'name': name,
'extension': ext
})
else:
meta_info = manifest.prepare_meta(sources=sources, data_dir=data_dir)
content = meta_info.content
if db_data.storage == StorageChoice.SHARE:
def _get_frame_step(str_):
match = search("step\s*=\s*([1-9]\d*)", str_)
return int(match.group(1)) if match else 1
step = _get_frame_step(db_data.frame_filter)
start = db_data.start_frame
stop = db_data.stop_frame + 1
images_range = range(start, stop, step)
result_content = []
for i in range(stop):
item = content.pop(0) if i in images_range else dict()
result_content.append(item)
content = result_content
manifest.create(content)
manifest.init_index()
except Exception as ex:
log_file.write(str(ex))
|
|
Hi, @Marishka17! Thank you, the proposed script solved this issue. |
My actions before raising this issue
When I try to enter into any job in concrete task, I can't see any images and I get the following error:
I've found the following logs in cvat container for this task:
I've found that it is still possible to dump annotations but I can't export the full dataset.
Expected Behaviour
Images in the job should be loaded properly.
Current Behaviour
Images are not loaded in the task with error notification.
Context
Due to this issue I can't load the full dataset
Your Environment
git log -1):docker version(e.g. Docker 17.0.05): 20.10.05NopeThe text was updated successfully, but these errors were encountered: