Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up git destination with mostly excluded files #191

Closed

Conversation

aaron-skydio
Copy link

When migrating into a git destination with excluded files, copybara calls git add on all of these excluded files individually. For cases where the destination repo is large, but most of the files are excluded, this drastically slows down the import process and is by far the slowest part of the import, taking minutes when the rest of the import takes seconds.

This change looks for directories where all files are excluded (i.e. we plan to call git add with every file in a directory), and instead calls git add with just the directory. It also does this recursively, i.e. on entire directory trees of this form. With this change, this part of the code is no longer the bottleneck in this type of migration.

There is potentially some overlap here with #112 - however, if I understand correctly, that PR would only work for CHANGE_REQUEST workflows, while this implementation will speed up all workflows with git destinations. And, I don't see any reason why these two changes are mutually exclusive.

@google-cla
Copy link

google-cla bot commented Mar 1, 2022

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

For more information, open the CLA check for this pull request.

@mikelalcon
Copy link
Collaborator

We had to rollback the change in e2a64da due to some bug. We are investigating.

@mikelalcon mikelalcon reopened this May 27, 2022
@copybara-staging copybara-staging bot closed this in f08fb8c Jun 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants