Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Asset optimization #2828

Merged
merged 7 commits into from
Nov 10, 2020
Merged

Asset optimization #2828

merged 7 commits into from
Nov 10, 2020

Conversation

jesseleite
Copy link
Member

Though there is still room for performance optimizations, this should fix timeouts in asset browser when working with thousands of files (both locally and in S3).

Closes #2145
Closes #1188

Also possibly related...?

References #2644
References #2279

jasonvarga and others added 5 commits November 10, 2020 11:41
There's a lot of overhead involved in converting paths to asset instances and hydrating them (reading the meta files). Especially on folders with thousands of assets. Even worse on S3 when it has to do a ton of API requests.

Use paths and natural limiting/offset unless we need asset instances, eg. when ordering or filtering with wheres.
Though it's still a bit slow, this makes navigating between pages in the asset browser a bit faster.  The downside is if any assets are added or deleted, it may take up to 1 minute to propagate.
@jasonvarga
Copy link
Member

More details:

  • The asset meta data gets cached to prevent on extra file reads (API calls on S3, DO, etc) and YAML parsing.
  • Folder listings get cached. An API call to S3 to get the contents is what takes a chunk of time.
  • We're only converting paths to assets (and getting meta data) for what's on the paginated page, where before we were doing it for everything in the folder. That's where the main slow down was.
  • As soon as you add a where clause or orderBy, we need to convert all the paths to assets, so it'll be slow again. In the asset browser, we don't do either of those things, so we're good.
  • If you perform a search, it would be a where clause, so it'll be slow. The workaround (which was already a thing you'd want to be doing anyway) is to have an actual search index containing the assets. That's fast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

S3 asset container list view caching S3 Assets Showing Container Is Empty
2 participants