-
Notifications
You must be signed in to change notification settings - Fork 152
refactor: esbuild refactor for readability #433
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| minimum_version_required=MINIMUM_VERSION_FOR_EXTERNAL, | ||
| working_directory=self._working_directory, | ||
| subprocess_esbuild=self._subprocess_esbuild, | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This used to be a separate action. Since we need to check the version every time that we want to mark things as --external then I think we should do the check when/before marking things as external (here), instead of having a separate action that would always have to be appended before this one. This cleans things up a bit, but it's also safer as we don't have to remember that we need a check action before this bundle action.
| working_directory: str, | ||
| output_directory: str, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
renamed to make more general
| NodeJS NPM Workflow using the esbuild bundler | ||
| """ | ||
|
|
||
| import logging |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file changed quite a bit. Might be easier to open the new and the old file in different windows (without the file diffs), and comparing.
| osutils, | ||
| self.options, | ||
| self.actions.append( | ||
| CopySourceAction(source_dir=self.source_dir, dest_dir=self.scratch_dir, excludes=self.EXCLUDED_FILES) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think using named arguments can be more readable in cases where it's not immediately obvious what the parameter(s) would be for. E.g. here we would immediately know which folder we copy from and which one we copy to, without having to check the CopySourceAction definition.
| osutils=self.osutils, | ||
| subprocess_esbuild=self.subprocess_esbuild, | ||
| manifest=self.manifest_path, | ||
| skip_deps=not self.combine_dependencies, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
combine_dependencies determines whether we should bundle dependencies or not
| self.actions += [ | ||
| NodejsNpmWorkflow.get_install_action( | ||
| source_dir=source_dir, | ||
| artifacts_dir=self.scratch_dir, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it would be better if this argument was named install_dir, as it actually controls where dependencies are installed. Will rename in another PR, when I introduce building-in-source to esbuild
| ) | ||
|
|
||
| self.assertEqual(len(workflow.actions), 4) | ||
| self.assertEqual(len(workflow.actions), 3) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a couple tests will have one action less because I moved the action to check esbuild version into the bundling action itself.
| self.assertIsInstance(workflow.actions[2], EsbuildBundleAction) | ||
| self.assertIsInstance(workflow.actions[3], CleanUpAction) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the order did change, to simplify the code logic. There should be no impact in the end-result, however. We just need to clean up the dependencies dir before we move the updated dependencies in there, but the bundling itself happens on another folder.
mndeveci
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the effort! Left some comments but will definitely do another round.
mildaniel
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Rup, this is a great QoL change. Left a couple small comments but generally agree with the direction you took combining the bundle action and version check action since they are always bundled together.
| # if we're reusing dependencies, then we need to symlink them before bundling | ||
| self.actions += [ | ||
| LinkSourceAction(self.dependencies_dir, self.scratch_dir), | ||
| bundle_action, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems that the bundle action is already included in line 103, so this would run the bundler twice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is one part that can lead to confusion and wasn't able to figure out how to make it better. It would never bundle twice because this bundle on this line happens when self.download_dependencies is False, while the one in line 103 happens when self.download_dependencies is True
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in other words, if we're reusing dependencies, then we didn't download them and we should first symlink and then bundle. If we downloaded dependencies, then we should bundle after downloading, (and we won't symlink or bundle again)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm seeing it if it's true for both cases:
if self.download_dependencies:
self.actions += [
NodejsNpmWorkflow.get_install_action(
source_dir=source_dir,
artifacts_dir=self.scratch_dir,
subprocess_npm=self.subprocess_npm,
osutils=self.osutils,
build_options=self.options,
),
bundle_action,
]
if not self.dependencies_dir:
# we need a dependencies dir for the logic below
return
if self.download_dependencies:
# if we downloaded dependencies, we have to update dependencies_dir
self.actions += [
CleanUpAction(self.dependencies_dir),
MoveDependenciesAction(self.source_dir, self.scratch_dir, self.dependencies_dir),
]
else:
# if we're reusing dependencies, then we need to symlink them before bundling
self.actions += [
LinkSourceAction(self.dependencies_dir, self.scratch_dir),
bundle_action,
]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think your indentation might be off? 😅 . The first if in the code block you're posting should be in the same level as the other two if's below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or otherwise not sure what you mean
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just made an additional change related to this, hopefully it's clearer now.
…lambda-builders into refactor_esbuild_workflow
|
@mndeveci added tests to increase coverage and changed to numpy docstrings |
mndeveci
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the effort that you put here @torresxb1! Approving the changes since we confirm that expected actions are been setup with possible combinations through tests.
* feat: support rust via cargo * make black reformat * use os.path.join in path assertion tests to address windows paths * address pylint ci errors * fix reformatted list comprehensions * try passing rust-lld linker on windows * update rust cargo design doc. windows support was added and tested * add test for cargo workspaces project * reformat test sources * build with musl on linux because glibc may differ on lambda * update linux copy and bin paths * update make test again, use the version appveyor is complaining about * update integration tests for rust cargo with latest aws rust runtime interfaces * align TestCustomMakeWorkflow integ test assumptions with appveyor reality * make x86_64-unknown-linux-musl a const for the rust cargo workflow * add integ test for failing cargo rust build * Update Rust workflow to use cargo-lambda Cargo-lambda takes care of cross compiling using Zig as linker. This works on Windows, Linux, and MacOS natively. Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix deprecation warnings. Signed-off-by: David Calavera <david.calavera@gmail.com> * Install Zig on Windows manually Signed-off-by: David Calavera <david.calavera@gmail.com> * Print Zig version on Windows Signed-off-by: David Calavera <david.calavera@gmail.com> * Update version of cargo-lambda Signed-off-by: David Calavera <david.calavera@gmail.com> * Run windows tests in powershell Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix powershell env notation Signed-off-by: David Calavera <david.calavera@gmail.com> * Print clang version on windows Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix env variable name Signed-off-by: David Calavera <david.calavera@gmail.com> * Try new version of zigbuild that fixes some linker issues on Windows. Signed-off-by: David Calavera <david.calavera@gmail.com> * Update releases URL. Signed-off-by: David Calavera <david.calavera@gmail.com> * Upgrade LLVM and clang Signed-off-by: David Calavera <david.calavera@gmail.com> * Update visual studio image To check if that makes any difference. Signed-off-by: David Calavera <david.calavera@gmail.com> * Change the visual studio image everywhere. Signed-off-by: David Calavera <david.calavera@gmail.com> * Revert upgrade changes They didn't fix the problem Signed-off-by: David Calavera <david.calavera@gmail.com> * Print environment Signed-off-by: David Calavera <david.calavera@gmail.com> * Update to Visual Studio 2022 Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix python variable Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix package urls Signed-off-by: David Calavera <david.calavera@gmail.com> * Update missing vs 2019 reference. Signed-off-by: David Calavera <david.calavera@gmail.com> * Change windows package suffix Signed-off-by: David Calavera <david.calavera@gmail.com> * Update cargo-lambda to version 0.9.0 Signed-off-by: David Calavera <david.calavera@gmail.com> * Go back to the original VS version. Signed-off-by: David Calavera <david.calavera@gmail.com> * Cleanup options - Use architecture to setup the right build target. - Don't require `--bin` flag, the default behaviour should work for the majority of functions. - Add flags option to provide a list of additional flags for projects that need extra configuration, like projects within a workspace. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add integration test with cargo_lambda_flags Signed-off-by: David Calavera <david.calavera@gmail.com> * Detect binaries when the project only includes one function. Signed-off-by: David Calavera <david.calavera@gmail.com> * Bring back handler as an optional argument. It saves some duplicated flags for working with workspaces. Signed-off-by: David Calavera <david.calavera@gmail.com> * Ignore errors if directory doesn't exist. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add debug logs Set RUST_LOG=debug when debug is enabled. Signed-off-by: David Calavera <david.calavera@gmail.com> * Log the artifact destination path. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add missing comma Signed-off-by: David Calavera <david.calavera@gmail.com> * Print out and err in the log when debug is enabled Signed-off-by: David Calavera <david.calavera@gmail.com> * Only set RUST_LOG when it's not already set Log it's value, so users know what's set at. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add experimentalCargoLambda feature flag. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add integration test for multi-function projects. Signed-off-by: David Calavera <david.calavera@gmail.com> * Remove type hints They're causing false positives in Python 3.9 with pylint. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add new CI steps for GHA Signed-off-by: David Calavera <david.calavera@gmail.com> * Add build_in_source_support Signed-off-by: David Calavera <david.calavera@gmail.com> * Test rust logger Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix assertion Signed-off-by: David Calavera <david.calavera@gmail.com> * Don't fail fast Signed-off-by: David Calavera <david.calavera@gmail.com> * fix: Fix failing esbuild integration tests (#423) * fix: Fix failing esbuild integration tests * Replace npm ci with npm install * Remove default shell Signed-off-by: David Calavera <david.calavera@gmail.com> * Revert "Remove default shell" This reverts commit 478c3b6. * Add check to ensure that Cargo Lambda is installed. Include a link to the gettings started guide that gives direct installation instructions based on the platform. Signed-off-by: David Calavera <david.calavera@gmail.com> * feat: Add support for mjs files with esbuild (#427) * feat: Add support for mjs files with esbuild * Black reformat * Test Cargo Lambda check Signed-off-by: David Calavera <david.calavera@gmail.com> * Don't redefine which Signed-off-by: David Calavera <david.calavera@gmail.com> * Organize code in more modules This structure follows other workflows, and provides better testeability. Signed-off-by: David Calavera <david.calavera@gmail.com> * Add more documentation Signed-off-by: David Calavera <david.calavera@gmail.com> * Format code Signed-off-by: David Calavera <david.calavera@gmail.com> * Capture exception Signed-off-by: David Calavera <david.calavera@gmail.com> * chore: Remove type/bug label for Bug Issue Template (#425) Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> * Clean design doc and tests Signed-off-by: David Calavera <david.calavera@gmail.com> * chore: Version bump to 1.25.0 (#429) * refactor: assign each workflow a default build directory (#428) * fix: remove unused symlinking (#432) * chore: Move to ruff from pylint (#435) When we started the project we defaulted to use pylint. Pylint has served it's purpose but it pretty slow. Ruff is a newer linter in the python ecosystem but is written in Rust. This makes Ruff was faster than pylint. On my machine (while testing in SAM CLI), pylint took about 70s to lint the repo but with ruff it took .04s. Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> * chore: Enable pylint within ruff (#436) Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> * refactor: esbuild refactor for readability (#433) * feat: use build_dir in esbuild workflow to support building in source (#437) * Fix formatting Signed-off-by: David Calavera <david.calavera@gmail.com> * Update build in source settings Signed-off-by: David Calavera <david.calavera@gmail.com> * fix: remove python3.6 support (#434) * chore: bump version to 1.26.0 (#441) * Remove default value for subprocess_cargo_lambda Signed-off-by: David Calavera <david.calavera@gmail.com> * feat: Pin ruff version, add dependabot config (#442) * feat: Pin ruff version, add dependabot config to keep our dependencies up to date * Exlude isort and flake8 from dependabot updates * feat: Add sources content flag to supported esbuild options (#439) * Better process management Signed-off-by: David Calavera <david.calavera@gmail.com> * Make release mode the default Signed-off-by: David Calavera <david.calavera@gmail.com> * Remove already default None Signed-off-by: David Calavera <david.calavera@gmail.com> * Turn debug message into warning Signed-off-by: David Calavera <david.calavera@gmail.com> * Update documentation format Signed-off-by: David Calavera <david.calavera@gmail.com> * Fix formatting Signed-off-by: David Calavera <david.calavera@gmail.com> * Preserve binary permissions Use copy2 instead of copyfile Signed-off-by: David Calavera <david.calavera@gmail.com> --------- Signed-off-by: David Calavera <david.calavera@gmail.com> Co-authored-by: softprops <d.tangren@gmail.com> Co-authored-by: Daniel Mil <84205762+mildaniel@users.noreply.github.com> Co-authored-by: Jacob Fuss <32497805+jfuss@users.noreply.github.com> Co-authored-by: Jacob Fuss <jfuss@users.noreply.github.com> Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Ruperto Torres <86501267+torresxb1@users.noreply.github.com>
Issue #, if available:
Description of changes:
I originally intended to add build-in-source support for the esbuild workflow but I found it hard with the way it was currently set up. I'm hoping my changes make it more readable and easier to work with in general, but let me know if you disagree with my changes or you think there could be further improvements.
Main changes done:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.