Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(server): faster metadata extraction #14277

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

mertalev
Copy link
Contributor

@mertalev mertalev commented Nov 21, 2024

Description

  • Groups async calls where possible rather than doing them sequentially
  • Moves some checks to be done synchronously without await
  • Extracts live photo logic into a private method that metadata extraction runs
  • No longer queues live photo job as part of upload pipeline

Besides being significantly faster, it also makes job progress more intuitive as the number of waiting jobs goes down as one would expect without other jobs being queued in their place.

Testing

metadata-extraction-rate

The y-axis is jobs/s. The first part is main and the second is this PR. The jump toward the end is when the queueing job finished queueing assets (the job is somewhat intensive with a large table). There is an increase on main as well if you look closely, but it's more subtle. I canceled both when their progress rate stabilized after the queueing job finished.

Copy link
Contributor

@jrasm91 jrasm91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two parts of a live photo can be uploaded at the same time and won't be linked if their metadata extraction jobs are processed in parallel. That was why it was moved to a separate job in the first place.

@jrasm91
Copy link
Contributor

jrasm91 commented Nov 21, 2024

It might be sufficient to save the CID value to the database prior to running the link logic part of the job though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants