Commits on May 23, 2023

1. Implement layout="zip" for Lambda/GCF, deprecating lambdex (#19076) …

   This fixes #18879 by allowing the `python_awslambda` and
   `python_google_cloud_function` FaaS artefacts to be generated in
   "simple" format, using the `pex3 venv create --layout=flat-zipped`
   functionality recently added in PEX 2.1.135
   (https://github.com/pantsbuild/pex/releases/tag/v2.1.135).
   
   This format is just: put everything at the top-level. For instance, the
   zip contains `cowsay/__init__.py` etc., rather than
   `.deps/cowsay-....whl`. This avoids the need to do the dynamic PEX
   initialisation/venv creation.
   
   This shifts the dynamic dependency computation/extraction/layout from
   run-time to build-time, relying on the FaaS environment to be generally
   consistent. It shouldn't change what actually happens after
   initialisation. This can:
   
   - reduce cold-starts noticeably: for instance, some of our lambdas spend
   1s doing PEX/Lambdex start up.
   - reduce package size somewhat (the PEX `.bootstrap/` folder seems to be
   about 2MB uncompressed, ~1MB compressed).
   - increase build times.
    
   For instance, for one Python 3.9 Lambda in our codebase:
   
   | metric | before | after |
   |---|---|---|
   | init time on cold start | 2.3-2.5s | 1.3-1.4s (-1s) |
   | compressed size |  24.6MB | 23.8MB (-0.8MB) |
   | uncompressed size | 117.8MB | 115.8MB (-2.0MB) |
   | PEX-construction build time | ~5s | ~5s |
   | PEX-postprocessing build time | 0.14s | 4.8s |
   
   (The PEX-postprocessing time metric is specifically the time to run the
   `Setting up handler` (lambdex) or `Build python_awslambda` (`pex3 venv
   create`) process, computed by running `pants --keep-sandboxes=always
   package ...` for each layout, and then `hyperfine -r3 -w1
   path/to/first/__run.sh path/to/second/__run.sh`. This _doesn't_ include
   the time to construct the input PEX, which is the same for both.)
   
   ---
   
   This functionality is driven by adding a new option to the
   `[lambdex].layout` option added in #19074. In #19074 (targeted for
   2.17), it defaults `lambdex` (retaining the current code paths). This PR
   flips the default to the new option `zip`, which keys into the
   functionality above. I've tried to keep the non-lambdex implementation
   generally separate to the lambdex one, rather than reusing all of the
   code that happens to be common currently, because it'd make sense to
   deprecate/remove the lambdex functionality and thus I feel it's best for
   this new functionality to be mostly a fresh start.
   
   This PR's commits can be reviewed independently. 
   
   I _think_ this is an acceptable MVP for this functionality, but there's
   various bits of follow-up:
   
   - add a warning about `files` being loaded into these packages, which
   has been temporarily lost (#19027)
   - adjust documentation #19067
   - other improvements like #18195 and #18880 
   - improve performance, e.g. potentially `pex3 venv create ...` could use
   the lock file and sources to directly compute the appropriate files,
   without having to materialise a normal pex first
   
   This is a re-doing of #19022 with a simpler approach to deprecation, as
   discussed in
   #19074 (comment)
   and
   #19032 (comment).
   The phasing will be:
   
   | release | supports lambdex? | supports zip? | default layout |
   deprecation warnings |
   |---|---|---|---|---|
   | 2.17 (this PR) | ✅ | ✅ | lambdex | if `layout = "lambdex"` is
   implicit, tell people to set it: recommend `zip`, but allow `lambdex` if
   they have to |
   | 2.18 | ✅ | ✅ | zip | if `layout = "lambdex"` is set at all, tell
   people to remove it and switch to `zip` |
   | 2.19 | ❌ | ✅ | zip | none, migration over (or maybe just about
   removing the `[lambdex]` section entirely) |

   

   huonw authored and WorkerPants committed May 23, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for b16be63
   • Browse repository at this point

   Copy the full SHA

   b16be63 View commit details

   Browse the repository at this point in the history