Upgrading from 2.22.0 to 2.23.1 fails with broken assets. #8803

ggozad · 2024-12-09T13:55:30Z

Actions before raising this issue

I searched the existing issues and did not find anything similar.
I read/searched the docs

Steps to Reproduce

Created a backup following https://docs.cvat.ai/docs/administration/advanced/backup_guide/
Updated from github from 2.22.0 to the latest being 2.23.0 following the postgres upgrade just in case described in https://docs.cvat.ai/docs/administration/advanced/upgrade_guide/

Images and video data are missing. Reverting to 2.22.0 and restoring from backup works fine.

Expected Behavior

I would expect to be able upgrade without loosing all the images/videos.

Possible Solution

No response

Context

No response

Environment

Local installation.
- Git was on `v2.22.0` tag as well as the `v2.23.1` tag.
- Ubuntu 24
- Updated from github from 2.22.0 to the latest being 2.23.0

bsekachev · 2024-12-09T14:23:43Z

Images and video data are missing

Could you please clarify what you mean?

ggozad · 2024-12-09T15:08:09Z

Images and video data are missing

Could you please clarify what you mean?

Sorry, that wasn't clear.

In the task list all images that used to be video previews are replaced with placeholders.
When trying to open an annotation task, I get the error message

Could not receive image data

<!doctype html> <html lang="en"> <head> <title>Server Error (500)</title> </head> <body> <h1>Server Error (500)</h1><p></p> </body> </html>

The video is simply gone from the UI.

bsekachev · 2024-12-09T15:09:19Z

docker logs cvat_server?

descilla · 2024-12-10T00:44:42Z

Hi,
I've the same Problem. In my case this only affects tasks which were imported trough the share path feature. Probably it's the same cause in your setup.

The "problem" was introduced by this Pull Request: Prepare chunks in a worker process. Since they use a dedicated container for chunk generation now this container also needs to have access to the shared volume: Example

Updating my docker-compose.override.yml by these lines made the images visible in the annotation view angain itself. Unfortunately the preview images are still replaced by placeholders. While the logging of the cvat_worker_chunks container remains clean the log output of the cvat_server container reports a file not found exception like the following (version 2.23.0):

2024-12-06 15:27:37,259 DEBG 'uvicorn-1' stderr output:
[2024-12-06 15:27:37,256] ERROR django.request: Internal Server Error: /api/projects/46/preview
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 518, in thread_handler
    raise exc_info[1]
  File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
  File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
  File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 468, in __call__
    ret = await asyncio.shield(exec_coro)
  File "/opt/venv/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 522, in thread_handler
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/django/cvat/apps/engine/views.py", line 652, in preview
    return data_getter()
  File "/home/django/cvat/apps/engine/views.py", line 708, in __call__
    data = frame_provider.get_preview()
  File "/home/django/cvat/apps/engine/frame_provider.py", line 265, in get_preview
    return self._get_segment_frame_provider(0).get_preview()
  File "/home/django/cvat/apps/engine/frame_provider.py", line 564, in get_preview
    preview, mime = cache.get_or_set_segment_preview(self._db_segment)
  File "/home/django/cvat/apps/engine/cache.py", line 454, in get_or_set_segment_preview
    self._get_or_set_cache_item(
  File "/home/django/cvat/apps/engine/cache.py", line 196, in _get_or_set_cache_item
    return self._create_cache_item(
  File "/home/django/cvat/apps/engine/cache.py", line 264, in _create_cache_item
    wait_for_rq_job(rq_job)
  File "/home/django/cvat/apps/engine/cache.py", line 117, in wait_for_rq_job
    raise exc_type(*exc_args)
FileNotFoundError: [Errno 2] No such file or directory

PS: The documentation currently does not mention the cvat_worker_chunks container within the share path section. I "discovered" this required configuration change by reading the pull request itself.

ggozad · 2024-12-10T07:23:13Z

I am not using anything extra, this is just a default setup.
Here's the traceback

[2024-12-10 07:20:57,274] INFO cvat.apps.engine.cache: Starting to prepare chunk: key segment_18_chunk_0_0

2024-12-10 07:20:58,082 DEBG 'uvicorn-0' stderr output:
[2024-12-10 07:20:58,081] ERROR django.request: Internal Server Error: /api/jobs/18/data
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 518, in thread_handler
    raise exc_info[1]
  File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
  File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
  File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 468, in __call__
    ret = await asyncio.shield(exec_coro)
  File "/opt/venv/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 522, in thread_handler
    return func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
    return view_func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 124, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/django/cvat/apps/engine/views.py", line 2239, in data
    return data_getter()
  File "/home/django/cvat/apps/engine/views.py", line 817, in __call__
    data = frame_provider.get_chunk(
  File "/home/django/cvat/apps/engine/frame_provider.py", line 652, in get_chunk
    return super().get_chunk(chunk_number, quality=quality)
  File "/home/django/cvat/apps/engine/frame_provider.py", line 571, in get_chunk
    chunk_data, mime = self._loaders[quality].read_chunk(chunk_number)
  File "/home/django/cvat/apps/engine/frame_provider.py", line 120, in read_chunk
    return self.get_chunk(chunk_id)
  File "/home/django/cvat/apps/engine/frame_provider.py", line 478, in <lambda>
    get_chunk_callback=lambda chunk_idx: cache.get_or_set_segment_chunk(
  File "/home/django/cvat/apps/engine/cache.py", line 372, in get_or_set_segment_chunk
    item = self._get_or_set_cache_item(
  File "/home/django/cvat/apps/engine/cache.py", line 197, in _get_or_set_cache_item
    return self._create_cache_item(
  File "/home/django/cvat/apps/engine/cache.py", line 270, in _create_cache_item
    wait_for_rq_job(rq_job)
  File "/home/django/cvat/apps/engine/cache.py", line 118, in wait_for_rq_job
    raise exc_type(*exc_args)
IndexError: tuple index out of range

bsekachev · 2024-12-10T08:39:22Z

PS: The documentation currently does not mention the cvat_worker_chunks container within the share path section. I "discovered" this required configuration change by reading the pull request itself.

You are right, we definitely need to update the documentation.

descilla · 2024-12-10T10:35:30Z

@ggozad Ok, looks like a slightly different problem. Should I therefore file a separate issue for my problem or will it suitable to handle both problems in this issue?

sriramsowmithri9807 · 2024-12-11T04:30:54Z

Hi @ggozad,

I’m encountering the same issue you described, and it seems to be related to tasks imported through the share path feature. I suspect it might be caused by the changes introduced in the Pull Request that prepared chunks in a worker process.

As you mentioned, since the chunk generation now occurs in a dedicated container, it’s crucial that this container has access to the shared volume. I made similar updates to my docker-compose.override.yml to ensure the images are visible in the annotation view again. However, I’m still facing issues with the preview images being replaced by placeholders.

Here’s the relevant error log from my setup (version 2.23.0):

Verify

Open In Editor
Run
Copy code

2024-12-06 15:27:37,259 DEBG 'uvicorn-1' stderr output:
[2024-12-06 15:27:37,256] ERROR django.request: Internal Server Error: /api/projects/46/preview
Traceback (most recent call last):
  ...
FileNotFoundError: [Errno 2] No such file or directory

Additionally, I noticed that the documentation does not currently mention the cvat_worker_chunks container in the share path section. I discovered this required configuration change by reviewing the pull request itself.

I also observed a similar traceback when trying to access job data:

Verify

Open In Editor
Run
Copy code

2024-12-10 07:20:58,082 DEBG 'uvicorn-0' stderr output:
[2024-12-10 07:20:58,081] ERROR django.request: Internal Server Error: /api/jobs/18/data
Traceback (most recent call last):

  ...
IndexError: tuple index out of range

It seems like there might be a deeper issue with how the chunk data is being handled in the new setup.

If anyone has insights or suggestions on how to resolve this, I would greatly appreciate it. Thank you for your help!

Best,
Sowmithri Sriram.
sriramsowmithri9807 --> github

azhavoro · 2024-12-12T13:39:08Z

@ggozad please provide logs from cvat_worker_chunks container as well

ggozad · 2024-12-12T14:28:39Z

Here you go:

[2024-12-12 14:23:53,120] ERROR rq.worker: [Job chunks:prepare-item-segment_19_preview]: exception raised while executing (__call__)
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/rq/worker.py", line 1431, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 1280, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 1317, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/engine/cache.py", line 169, in __call__
    return self._callable(*self._args, **self._kwargs)
  File "/home/django/cvat/apps/engine/cache.py", line 213, in _drop_return_value
    func(*args, **kwargs)
  File "/home/django/cvat/apps/engine/cache.py", line 223, in _create_and_set_cache_item
    item_data = create_callback()
  File "/home/django/cvat/apps/engine/cache.py", line 169, in __call__
    return self._callable(*self._args, **self._kwargs)
  File "/home/django/cvat/apps/engine/cache.py", line 827, in _prepare_segment_preview
    preview = segment_frame_provider.get_frame(
  File "/home/django/cvat/apps/engine/frame_provider.py", line 595, in get_frame
    frame, frame_name, reader_class = self._get_raw_frame(frame_number, quality=quality)
  File "/home/django/cvat/apps/engine/frame_provider.py", line 582, in _get_raw_frame
    chunk_reader = loader.load(chunk_number)
  File "/home/django/cvat/apps/engine/frame_provider.py", line 72, in load
    [self.read_chunk(chunk_id)[0]],
  File "/home/django/cvat/apps/engine/frame_provider.py", line 120, in read_chunk
    return self.get_chunk(chunk_id)
  File "/home/django/cvat/apps/engine/frame_provider.py", line 478, in <lambda>
    get_chunk_callback=lambda chunk_idx: cache.get_or_set_segment_chunk(
  File "/home/django/cvat/apps/engine/cache.py", line 372, in get_or_set_segment_chunk
    item = self._get_or_set_cache_item(
  File "/home/django/cvat/apps/engine/cache.py", line 197, in _get_or_set_cache_item
    return self._create_cache_item(
  File "/home/django/cvat/apps/engine/cache.py", line 249, in _create_cache_item
    item = self._create_and_set_cache_item(
  File "/home/django/cvat/apps/engine/cache.py", line 233, in _create_and_set_cache_item
    if cached_item is not None and timestamp <= cached_item[3]:
IndexError: tuple index out of range

azhavoro · 2024-12-12T14:32:27Z

@ggozad try to clear the cache with docker exec cvat_redis_ondisk redis-cli -p 6666 flushall

ggozad · 2024-12-12T14:34:22Z

@azhavoro this seems to resolve the issue! Thank you!
Should it be perhaps part of the upgrade?

azhavoro · 2024-12-12T14:40:47Z

I don't think so, it should be handled correctly in the code. Unfortunately, I missed this point in the PR. In any case, it can be fixed either by clearing the cache, or it would work without any action in no more than 24 hours, after the cache item would have expired

ggozad · 2024-12-12T14:56:24Z

I don't think so, it should be handled correctly in the code. Unfortunately, I missed this point in the PR. In any case, it can be fixed either by clearing the cache, or it would work without any action in no more than 24 hours, after the cache item would have expired

I did verify on another mirror I had set up for testing that indeed after 24h the cache keys are expired and everything works fine.

Prehani · 2024-12-12T17:49:05Z

fwiw, we are experiencing the same issue--clearing the cache as described did not work; I will spin up a mirror and see if it self resoles within 24 hours

ggozad added the bug Something isn't working label Dec 9, 2024

bsekachev added the documentation Documentation should be updated label Dec 10, 2024

bsekachev mentioned this issue Dec 10, 2024

Updated documentation: need to mount shared volume to cvat_worker_chunks #8812

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading from 2.22.0 to 2.23.1 fails with broken assets. #8803

Upgrading from 2.22.0 to 2.23.1 fails with broken assets. #8803

ggozad commented Dec 9, 2024

bsekachev commented Dec 9, 2024

ggozad commented Dec 9, 2024

bsekachev commented Dec 9, 2024

descilla commented Dec 10, 2024 •

edited

Loading

ggozad commented Dec 10, 2024

bsekachev commented Dec 10, 2024

descilla commented Dec 10, 2024

sriramsowmithri9807 commented Dec 11, 2024

azhavoro commented Dec 12, 2024

ggozad commented Dec 12, 2024

azhavoro commented Dec 12, 2024

ggozad commented Dec 12, 2024

azhavoro commented Dec 12, 2024

ggozad commented Dec 12, 2024

Prehani commented Dec 12, 2024

Upgrading from 2.22.0 to 2.23.1 fails with broken assets. #8803

Upgrading from 2.22.0 to 2.23.1 fails with broken assets. #8803

Comments

ggozad commented Dec 9, 2024

Actions before raising this issue

Steps to Reproduce

Expected Behavior

Possible Solution

Context

Environment

bsekachev commented Dec 9, 2024

ggozad commented Dec 9, 2024

bsekachev commented Dec 9, 2024

descilla commented Dec 10, 2024 • edited Loading

ggozad commented Dec 10, 2024

bsekachev commented Dec 10, 2024

descilla commented Dec 10, 2024

sriramsowmithri9807 commented Dec 11, 2024

azhavoro commented Dec 12, 2024

ggozad commented Dec 12, 2024

azhavoro commented Dec 12, 2024

ggozad commented Dec 12, 2024

azhavoro commented Dec 12, 2024

ggozad commented Dec 12, 2024

Prehani commented Dec 12, 2024

descilla commented Dec 10, 2024 •

edited

Loading