Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Direct registration of files to database #1

Merged
merged 39 commits into from
Jun 3, 2024
Merged

Direct registration of files to database #1

merged 39 commits into from
Jun 3, 2024

Conversation

jennydaman
Copy link
Contributor

@jennydaman jennydaman commented May 30, 2024

We observed a performance bottleneck with the django backend of CUBE. FNNDSC/ChRIS_ultron_backEnd#546

This PR registers files to the database directly, instead of via CUBE's API. This is advantageous for several reasons:

  • file registration can be done in bulk: specifically, two SQL commands to register 100 files
    • First SQL adds the AE title of PACS to the pacsfiles_pacs table, second SQL registers the files in bulk.
    • ChRIS_ultron_backEnd registers files one-by-one using ORM, which probably means 200-400 SQL commands for registering 100 files.
  • Handling of duplicate rows in pacsfiles_pacs is achieved using native PostgreSQL features (INSERT INTO ... ON CONFLICT DO NOTHING)
  • Using SQL transactions guarantees atomicity, making it safer to run multiple workers/replicas
  • the whole process is now free of slow Python code!

Other changes:

Not yet implemented: oxidicom custom metadata spec.

@jennydaman jennydaman merged commit f2198bd into master Jun 3, 2024
2 checks passed
@jennydaman jennydaman deleted the direct2db branch June 3, 2024 07:56
jennydaman added a commit to FNNDSC/miniChRIS-docker that referenced this pull request Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant