Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: store collection volume changes #576

Merged
merged 19 commits into from
Jan 18, 2024

Conversation

crnkovic
Copy link
Contributor

@crnkovic crnkovic commented Dec 19, 2023

Summary

https://app.clickup.com/t/86dqrnjn8

So, for this PR, I went with the approach of storing daily volume changes of collections and then averaging that value over a 1d/7d/30d period. We can use that average for sorting on the collections page. In #548, Alfy went with the route of automatically taking 1d/7d/30d volume averages from Mnemonic (by making 3 parallel API requests) when we fetch volume. Both approaches have pros and cons, which I'll try to highlight here.

After giving it some thought, this may or may not be the best way to do this, depending on what we want. A while back, we removed scheduled updating of collection metadata (owners, volume, traits). Because of that and the way this PR works, I brought back the scheduled updating of collection volume -- just because we calculate average based on the volume data from our database. If we take an example number of 1000 collections in our database, this would automatically put us at 30k API requests a month to Mnemonic. Mnemonic segregates API requests by type, and free plan allows 50k requests a month for this type of request.

From what I see in the code, we don't really use Mnemonic a lot anymore. We use it to update: collection banner, owners, traits, volume and collection activity. All of these (other than activity) are ran on-demand when you open the collection page and not on schedule. Collection activity fetching is a type of its own and that endpoint has 50k requests/month limit as well and is of a different type than volume.

This gives us basically very little Mnemonic usage at the moment, so that we could go back to scheduled for volume and monitor API usage. The pro of this approach is that we'll always have the latest volume changes over 1d/7d/30d period, unlike in #548, where it consumes less API quota for Mnemonic, but we can get old data since the 1d/7d/30d volume is updated on-demand (when you open the collection page) and not on a schedule, therefore sorting would not really "work" if you don't update volume for a lot of collections.

Update: I have updated the code to handle situations where we don't have enough data in our database to calculate average weekly/monthly volume. Now, whenever a new collection is added to the database, we'll dispatch the jobs to fetch average volumes for a collection. This would basically be 3 requests per newly-added collection, but run only once for a new collection. Then, we'll reuse that data until we fetch the daily volume enough times to calculate averages.

Checklist

  • I checked my UI changes against the design and there are no notable differences
  • I checked my UI changes for any responsiveness issues
  • I checked my (code) changes for obvious issues, debug statements and commented code
  • I opened a corresponding card on Clickup for any remaining TODOs in my code
  • I added a short description on how to test this PR (if necessary)
  • I added a storybook entry for the component that was added (if necessary)
  • Documentation (if necessary)
  • Tests (if necessary)
  • Ready to be merged

@crnkovic crnkovic mentioned this pull request Dec 19, 2023
9 tasks
@ItsANameToo ItsANameToo changed the base branch from feat/collections-overview to develop December 20, 2023 09:55
@goga-m goga-m added this to the 0.13.0 milestone Dec 21, 2023
@ItsANameToo
Copy link
Contributor

what's left for this @crnkovic

@crnkovic
Copy link
Contributor Author

@ItsANameToo finished the code but have updated the PR description with more context on this implementation

app/Jobs/FetchCollectionVolume.php Outdated Show resolved Hide resolved
app/Jobs/FetchCollectionVolume.php Outdated Show resolved Hide resolved
@crnkovic crnkovic marked this pull request as ready for review January 17, 2024 15:26
@ItsANameToo ItsANameToo merged commit dd7ebab into develop Jan 18, 2024
12 checks passed
@ItsANameToo ItsANameToo deleted the refactor/log-volume-changes branch January 18, 2024 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants