feat: use symlink instead of copy for some build operations #374
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
pr/internal
area/workflow/node_npm
area/workflow/python_pip
labels
mndeveci
commented
Jul 15, 2022
mndeveci
commented
Jul 15, 2022
mildaniel
reviewed
Jul 18, 2022
sriram-mv
requested changes
Jul 22, 2022
| "Symlink operation is failed, falling back to copying files", | ||
| exc_info=ex if LOG.isEnabledFor(logging.DEBUG) else None, | ||
| ) | ||
| copytree(source, destination) |
Contributor
There was a problem hiding this comment.
Am I right that this is backwards compatible, in the sense that if we cannot symlink we fall back to copying which was the existing behavior anyways?
| # Auto dependency layer disabled, first build | ||
| actions += [esbuild_with_deps, CopyDependenciesAction(source_dir, scratch_dir, self.dependencies_dir)] | ||
| actions += [ | ||
| esbuild_with_deps, |
Contributor
There was a problem hiding this comment.
Does this require an experimental check? Python requires it and node es build does not?
Contributor
There was a problem hiding this comment.
This was a conscious decision. Since is esbuild is still in beta, we figured it would be redundant to have two beta flags for it.
sriram-mv
approved these changes
Jul 22, 2022
mildaniel
approved these changes
Jul 25, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Companion PR to this: aws/aws-sam-cli#3935
Related Issues
aws/aws-sam-cli#3093
aws/aws-sam-cli#2688
Description of changes:
During the incremental build, we do an extra copy operation to persist dependencies into a different folder so that next time the build can re-use them instead of re-downloading if the manifest file has not changed.
But that extra copy operation might take some time especially for a build where there are a lot of small files.
This PR addresses this issue by replacing copy operation with symlink, since symlinking those files will be faster than copying them.
Note: This improvement can't be applied to regular nodejs build & ruby build yet since local invocation for these runtimes will be broken.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.