Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[performance] move thumbnail generation to go code where possible #3183

Conversation

NyaaaWhatsUpDoc
Copy link
Member

@NyaaaWhatsUpDoc NyaaaWhatsUpDoc commented Aug 8, 2024

Description

While ffmpeg is great for thumbnail generation, and we do like using webps (pee hehe), it comes with a lot of extra overhead. This updates our media processing code to use ffmpeg as a fallback, which ends up being with the majority of media types, but primarily resorting to our own Go thumbnail generation code if possible. this results in JPEG, PNG, GIF, Webps without transparency layers being thumbnailed using Go code, and all else resorting to ffmpeg. unfortunately there is no Go support for Webp encoding, so those thumbnailed with Go code are encoded to JPEGs, hence the restriction on transparency layers.

This also updates ffmpeg to maintain the pixel colorspace where possible (credit largely to @tsmethurst for this, even if the investigation ended a bit oddly 🥲), updates our metadata cleaning to fallback to ffmpeg when exif-terminator fails (thank you @tsmethurst), updates our handling of image rotation to instead use the "orientation" flag which is both much more flexible (supporting mirroring) and much easier to work with, reduces the number of frames we page through with the ffmpeg thumbnail filter (which reduces memory usage, marginally), and finally updates our thumbnail quality setting to the defaults for both webp / jpeg (75%) which looks much nicer on the frontend while not causing much of a size increase (once again, thank you @tsmethurst)

This PR is a result of multiple days of digging and experimentation by both myself and @tsmethurst, this absolutely was a joint effort :p

Checklist

  • I/we have read the GoToSocial contribution guidelines.
  • I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
  • I/we have not leveraged AI to create the proposed changes.
  • I/we have performed a self-review of added code.
  • I/we have written code that is legible and maintainable by others.
  • I/we have commented the added code, particularly in hard-to-understand areas.
  • I/we have made any necessary changes to documentation.
  • I/we have added tests that cover new code.
  • I/we have run tests and they pass locally with the changes.
  • I/we have run go fmt ./... and golangci-lint run.

internal/media/ffmpeg.go Outdated Show resolved Hide resolved
internal/media/ffmpeg.go Outdated Show resolved Hide resolved
}

// Apply orientation
// BEFORE any resize.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for this btw? Cheaper?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's just a case of our image dimensions already taking orientation into account, so we need to reorient the image first before applying the new dimensions :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i added a code comment to make it a bit clearer too

@tsmethurst
Copy link
Contributor

Looks good :) :) :)

@NyaaaWhatsUpDoc NyaaaWhatsUpDoc marked this pull request as ready for review August 8, 2024 15:59
@NyaaaWhatsUpDoc NyaaaWhatsUpDoc merged commit f770051 into superseriousbusiness:main Aug 8, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants