Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: improve sync command performance #3766

Merged
merged 10 commits into from
May 27, 2024
Merged

perf: improve sync command performance #3766

merged 10 commits into from
May 27, 2024

Conversation

olaszakos
Copy link
Contributor

@olaszakos olaszakos commented May 22, 2024 •
edited
Loading

Description

This PR has the following changes:

  1. Make chunk creation faster, by increasing parallelization 4=>25, improves performance significantly for lots of small assets
  2. Make asset properties query faster by parallelization (major time save for canisters that have a ton of assets, the internetcomputer.org canister has 11k+ assets)
  3. icx-asset: Support log levels, and default to info

How Has This Been Tested?

I have not added any new tests, we are using this already with icx-asset sync for dfinity/portal PR preview deployments.

Deployment time has gone down from ~40m to <10m.

Checklist:

  • The title of this PR complies with Conventional Commits.
  • I have edited the CHANGELOG accordingly.
  • I have made corresponding changes to the documentation.

@olaszakos olaszakos mentioned this pull request May 22, 2024
Merged
@olaszakos olaszakos changed the title feat(icx-asset)!: improve upload/sync commands feat(icx-asset)!: improve sync command performance May 23, 2024
@olaszakos olaszakos changed the title feat(icx-asset)!: improve sync command performance feat(icx-asset): improve sync command performance May 23, 2024
@olaszakos olaszakos marked this pull request as ready for review May 23, 2024 10:04
@olaszakos olaszakos requested review from chenyan-dfinity and a team as code owners May 23, 2024 10:04
@olaszakos olaszakos changed the title feat(icx-asset): improve sync command performance feat: improve sync command performance May 24, 2024
ericswanson-dfinity
ericswanson-dfinity approved these changes May 24, 2024
View reviewed changes
Copy link
Member

@ericswanson-dfinity ericswanson-dfinity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a note to the changelog. This kind of performance improvement is exciting!

@olaszakos olaszakos changed the title feat: improve sync command performance perf: improve sync command performance May 27, 2024
@olaszakos olaszakos merged commit f3c5b68 into master May 27, 2024
285 checks passed
@olaszakos olaszakos deleted the akos/asset-upload-strategies branch May 27, 2024 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericswanson-dfinity ericswanson-dfinity ericswanson-dfinity approved these changes

@chenyan-dfinity chenyan-dfinity Awaiting requested review from chenyan-dfinity

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@olaszakos @ericswanson-dfinity