Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validation management core #8471

Merged
merged 343 commits into from
Oct 7, 2024
Merged

Validation management core #8471

merged 343 commits into from
Oct 7, 2024

Conversation

zhiltsov-max
Copy link
Contributor

@zhiltsov-max zhiltsov-max commented Sep 25, 2024

Motivation and context

Depends on #8348
Includes / depends on #8510

  • Added support for updating chunks in jobs
  • Added reporting for chunk update date in jobs
  • Added an option to change validation frames in tasks with honeypots (validation mode == gt_pool)
  • Added server API to view and manage current validation frame setup:
    • GET /api/tasks/{id}/validation_layout - allows getting validation configuration in the task
    • GET /api/jobs/{id}/validation_layout - allows getting validation configuration in the job (available only to the task assignee and above)
    • PATCH /api/tasks/{id}/validation_layout - allows exclusion of validation frames and changing real frames for honeypots in the task
    • PATCH /api/jobs/{id}/validation_layout - allows changing real frames for honeypots in the job (available only to the task assignee and above)

How has this been tested?

Checklist

  • I submit my changes into the develop branch
  • I have created a changelog fragment
  • I have updated the documentation accordingly
  • I have added tests to cover my changes
  • I have linked related issues (see GitHub docs)
  • I have increased versions of npm packages if it is necessary
    (cvat-canvas,
    cvat-core,
    cvat-data and
    cvat-ui)

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.

Summary by CodeRabbit

  • New Features

    • Enhanced quality reports with tracking of last assignee update date.
    • Added new options in quality settings for better customization.
    • Introduced sorting method handling in the advanced configuration form.
  • Bug Fixes

    • Improved logic for calculating active and excluded frame counts in quality management.
  • Refactor

    • Updated terminology from ValidationMethod to ValidationMode across various components.
    • Streamlined frame data processing in the allocation table.
  • Chores

    • Removed unused properties related to frame deletion in UI components.


def init_chunks_updated_date(apps, schema_editor):
# The 0083 migration changed data distribution by chunks
migration_0083_date = _get_0083_move_to_segment_chunks_migration_date(apps, schema_editor)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use current datetime?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or just task_created_date as done below

Copy link
Contributor Author

@zhiltsov-max zhiltsov-max Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because this will be incorrect.
If the task was created before the migration, it's chunks were updated when the migration was applied (maybe several days before the current migration is applied).
If the task was created after the migration, it's chunks were updated at the moment of task creation.
Neither of this is now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is correct enough, as the feature to upgrade chunks (shuffle honeypots) was not merged yet.

chunks were updated when the migration was applied

How exactly these chunks were updated in the migration? They only were re-numerated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How exactly these chunks were updated in the migration? They only were re-numerated.

No, the chunks on the end of the segment could be truncated, and for videos with overlap there were entirely new chunks.

cvat/apps/engine/rules/jobs.rego Outdated Show resolved Hide resolved
cvat/apps/engine/rules/tasks.rego Outdated Show resolved Hide resolved
@@ -115,6 +116,9 @@

_UPLOAD_PARSER_CLASSES = api_settings.DEFAULT_PARSER_CLASSES + [MultiPartParser]

_DATA_CHECKSUM_HEADER_NAME = 'X-Checksum'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this checksum supposed to be used?

Copy link
Contributor Author

@zhiltsov-max zhiltsov-max Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be used to check if the downloaded chunk has the same checksum as the one cached, if there are any locally cached ones. Some job chunks may be unaffected by the latest update.

@zhiltsov-max
Copy link
Contributor Author

zhiltsov-max commented Oct 4, 2024

Need to integrate tasks with honeypots with Review mode in UI. Basically, need to retrieve job validation_layout and get honeypot_real_frames from there, if the task validation_mode is gt_pool.


# Guarantee uniformness by using a known distribution
# overall task honeypot distribution is not guaranteed though
rng = random.Generator(random.MT19937())

Check notice

Code scanning / SonarCloud

Results that depend on random number generation should be reproducible Low

Provide a seed for this random generator. See more on SonarCloud
Copy link

sonarcloud bot commented Oct 7, 2024

@bsekachev bsekachev merged commit 0572fa1 into develop Oct 7, 2024
35 checks passed
@cvat-bot cvat-bot bot mentioned this pull request Oct 10, 2024
@bsekachev bsekachev deleted the zm/validation-management-core branch October 24, 2024 05:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants