Imports of large Python library (aws-cdk-lib) extremely slow

[rix0rrr]

🐛 Bug Report

Affected Languages

• TypeScript or Javascript
• Python
• Java
• .NET (C#, F#, ...)
• Go

General Information

• JSII Version: N/A
• Platform: MacOS

What is the problem?

Originally reported as aws/aws-cdk#19000

---

I noticed that simple commands such as cdk ls would take very long (sometimes above 20 seconds, rarely much less) to complete. This in a project created using cdk init sample-app --language python.

Reproduction Steps

# setup:
$ cdk init sample-app --language python
$ source .venv/bin/activate
$ pip install -r requirements.txt
# reproducing issue:
$ time python -c "import aws_cdk as cdk"
python -c "import aws_cdk as cdk"  5.74s user 4.20s system 58% cpu 17.113 total

What did you expect to happen?

A simple import should not take longer than 1 second (ideally even less, but there are constrains that make that diffucult here I understand).

What actually happened?

As seen above, it takes ~17 seconds. This is quite consistent, however first or second time after initing the project, it can take 20-25 seconds.

CDK CLI Version

2.12.0 (build c9786db)

Framework Version

Node.js Version

v16.14.0

OS

MacOS 11.6.2 (20G314) - Intel Macbook

Language

Python

Language Version

Python (3.8.9, 3.9.10)

Other information

To better understand where the time went, I debugged the issue with some print statements:

❯ time python -c "import aws_cdk as cdk"
process.py start: 0.00
self._process.args=['node', '--max-old-space-size=4069', '/var/folders/9j/cszr_w557ns8q4bk2ctzygx40000gp/T/tmp9_s6gne5/bin/jsii-runtime.js']
>>> b'{"hello":"@jsii/runtime@1.54.0"}\n'
process.py before self._next_message(): 0.25
>>> b'{"ok":{"assembly":"constructs","types":10}}\n'
process.py after self._next_message(): 0.33
process.py before self._next_message(): 0.33
>>> b'{"ok":{"assembly":"aws-cdk-lib","types":7639}}\n'
process.py after self._next_message(): 12.95
python -c "import aws_cdk as cdk"  5.72s user 4.21s system 58% cpu 16.923 total

The offending method seems to be (from Python's point of view) jsii/_kernel/providers/process.py: _NodeProcess.send

I understand that this communicates with a node process, so I assume it to be something going on there that takes very long.

[kbakk]

Hi, original reporter of aws/aws-cdk#19000 here. 🙂

I wonder if it's something wrong with my environment. I didn't originally try out CDK TypeScript and now wanted to check how the experience is there. To my surprise, it's quite slow...

In a directory initialized with cdk init sample-app --language typescript:

$ time cdk ls --debug
CdkTsWorkshopStack
cdk ls --debug  9.44s user 1.52s system 86% cpu 12.634 total

Any suggestions to what I should look into?

[kbakk]

Did another test, running in Docker (to exclude my Macbook from the equation), and the results are quite different:

Setup TypeScript:

$ docker run -it --rm --workdir /cdk_workshop node:16-bullseye bash
# inside container:
$ npm install -g cdk@2.12.0
$ cdk init sample-app --language typescript

Test:

$ time cdk ls
CdkWorkshopStack

real	0m6.648s
user	0m7.601s
sys	0m1.378s

Setup Python:

$ docker run --rm -it --workdir /cdk_workshop python:3.9-bullseye bash
# inside container:
$ curl -fsSL https://deb.nodesource.com/setup_16.x | bash - && apt-get install -y nodejs
$ npm install -g cdk@2.12.0
$ cdk init sample-app --language python
$ source .venv/bin/activate
$ pip install -r requirements.txt

Test:

$ time cdk ls
cdk-workshop

real	0m6.195s
user	0m4.823s
sys	0m1.832s

$ time python -c "import aws_cdk as cdk"

real	0m6.495s
user	0m5.019s
sys	0m1.954s

Edit: updated with Python import timing result

[RomainMuller]

Removing the bug label as this is more of a performance issue than a literal bug.

It seems like this report suggests that: importing aws-cdk-lib (JavaScript) is taking ~6s.

Note that importing aws_cdk (Python) equates importing the JavaScript + the python, so I would expect it takes ~6 seconds, too (the Python module tax would be relatively low).

This perhaps has to do more with aws-cdk-lib's sheer size than with how we generate Python bindings?

---

I'm going to try to come up with a "somewhat scientific" measurement protocol here (across all supported languages), to try and better qualify where the problem might originate from.

[RomainMuller]

Okay so far I do see there is a significant delta between the various "minimal apps":

JavaScript =>    1.62s user 0.22s system 112% cpu 1.628 total
C#         =>    6.52s user 1.03s system 115% cpu 6.545 total
Go         =>    0.61s user 0.15s system 16% cpu 4.559 total
Java       =>    4.58s user 0.43s system 62% cpu 8.056 total
Python     =>    5.63s user 1.18s system 107% cpu 6.358 total

I can see affected languages would be: C#, Python, and Java. Although this has a sample-size of 1.
System time looks to be a contributor for C# and Python (although not the majority). Maybe different file system access patterns. There are still 4 to 6 user-land seconds that have to be accounted for now.

Interestingly, Go has lower user + system times, but higher wall time than JavaScript (this is somewhat surprising -- where do the extra 3 seconds go?)

[RomainMuller]

Focusing on Python... suing cProfile and pstats (on my laptop), it appears to have this suspicious entry:

Tue Feb 22 15:50:17 2022    profile

         1908213 function calls (1862684 primitive calls) in 6.822 seconds

   Ordered by: internal time
   List reduced from 9880 to 30 due to restriction <30>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        6    4.863    0.811    4.863    0.811 {method 'readline' of '_io.BufferedReader' objects}

That is ~70% of the runtime accounted by reading from the child process' output stream... That might be time spent waiting for the node process to return data... but I guess the python-side profile doesn't quite tell the full picture on this.

[RomainMuller]

I've actually gotten conflict data on the JavaScript execution speed - on similar hardware I got the aws-cdk-lib import clocking at ~6 seconds. It appears some other factors are at play in the performance numbers we see here and so far I can't quite identify what causes the vast discrepancies I see.

[kbakk]

On a different machine – 2016 MacBook Pro 13 (Intel i7), quite a big step down from the 2019 MacBook Pro 16 that I initially did the testing on, it actually behaves faster:

$ time npx cdk@2 ls
cdk-19000
npx cdk@2 ls  6.90s user 2.84s system 106% cpu 9.164 total
$ time python3 -c 'import aws_cdk'
python3 -c 'import aws_cdk'  5.40s user 2.53s system 104% cpu 7.556 total
$ npx cdk@2 --version        
2.13.0 (build b0b744d)

~7 sec is an improvement to ~17 sec, so if I could get the same performance on my main machine...

As mentioned, I get better performance when running in a docker container – I'm contemplating setting up my dev environment for CDK to run in Docker...

[kbakk]

As a workaround, I think this is something I'll be able to use:

docker-compose.yaml

version: '3'
services:
  cdk:
    build: .
    working_dir: /cdk
    volumes:
    - ./:/cdk
    entrypoint: ["bash", "-c", "trap exit INT TERM; while :; do sleep 1 & wait; done;"]

Dockerfile

FROM python:3.9-bullseye
RUN curl -fsSL https://deb.nodesource.com/setup_16.x | bash - && apt-get install -y nodejs && \
    npm install -g cdk@2.12.0

Then these steps:

1. docker compose up -d
2. docker compose exec cdk bash
3. inside container pip install -r requirements.txt
4. Then run the CDK commands (the executions are much faster, as mentioned the other day)

[kbakk]

Regarding Docker Compose (not strictly on-topic, but I think this might be helpful for others coming here):

The docker-compose approach didn't work for me in the end. For cdk synth, cdk ls it works, but if you'd like to do cdk diff, cdk deploy etc. it may be problematic if you depend on bundling Lambdas using Docker. There might be some workarounds there. See perhaps aws/aws-cdk#9348.

[JJSphar]

@kbakk did you ever find a resolution for this?

On my win10 machine, CDK commands execute painfully slowly (over 60s for a cdk synth on the sample-app). The import seems to be major chunk of that time:

> time python -c "import aws_cdk as cdk"

real    0m50.914s
user    0m0.015s
sys     0m0.062s

I see similar slowness running cdk synth on a TS project from my base machine:

> time cdk synth -q

real    0m47.562s
user    0m0.153s
sys     0m0.339s

Interestingly, I get much better performance when using an Ubuntu 20.04 WSL instance from the same machine:

> time python -c "import aws_cdk as cdk"

real    0m10.985s
user    0m9.548s
sys     0m1.731s

When I profiled app.py, I also saw a significant portion of time being taken up in method 'readline' of '_io.BufferedReader' objects. @RomainMuller, I can provide the .pstat file if it would be helpful.

[kbakk]

No, still slow. I'm currently running these commands Linux VM (hosted in Google Cloud, can't award AWS for this 😄).

[rafalkrupinski]

might be related #3365

[pinzyte]

I've just installed cdk on fedora35 and am using python. "cdk ls" in a new project is taking >6s.

The node process which is spawned by "import aws_cdk as cdk" is taking most of the time, and I notice that it's populating 7000+ files each time in a new directory /tmp/jsii-kernel-rAnDoM under subdirs: node_modules/{aws-cdk-lib,constructs}. I can watch it taking seconds to do this. (The directory is usually destroyed at the end of each command.)

I don't know much about node. I tried installing aws-cdk-lib globally with npm (having originally installed aws-cdk as per the docs) but the behaviour is unchanged.

Everything works, but just with this overhead on each command.

[RichiCoder1]

@kbakk did you ever find a resolution for this?

On my win10 machine, CDK commands execute painfully slowly (over 60s for a cdk synth on the sample-app). The import seems to be major chunk of that time:

> time python -c "import aws_cdk as cdk"

real    0m50.914s
user    0m0.015s
sys     0m0.062s

I see similar slowness running cdk synth on a TS project from my base machine:

> time cdk synth -q

real    0m47.562s
user    0m0.153s
sys     0m0.339s

Interestingly, I get much better performance when using an Ubuntu 20.04 WSL instance from the same machine:

> time python -c "import aws_cdk as cdk"

real    0m10.985s
user    0m9.548s
sys     0m1.731s

When I profiled app.py, I also saw a significant portion of time being taken up in method 'readline' of '_io.BufferedReader' objects. @RomainMuller, I can provide the .pstat file if it would be helpful.

Whenever I see Windows be slower than WSL, the first thing that comes to mind is Windows Defender. Have you excluded the containing folder from Defender to count that out?

[kbakk]

@RomainMuller has there been any progress from your side (or anyone else at AWS) on this? Is there anything that can be done to ensure this gets attention?

[du291]

Hello, we observe the same issue.

Simple python file with the content from aws_cdk import aws_ec2 takes about 7 seconds to run. It seems to run a node process that takes most of the CPU. It looks like it is trying to compile a large chunk of javascript code each time. On CDK1 this didn't happen, because you could import selectively smaller code bases, here it seems it's trying to compile everything at once.
It is a real productivity killer for test-driven development where sub-second test-runs are preferred.
Is it possible to have a pre-cached code or load on demand system?

[du291]

We spent some time debugging the CDK load process. It is somewhat unwieldy due to two major issues/design decisions that contribute to the problem. I will outline it here for the general public, as I assume that AWS people decided on these tradeoffs knowingly.

The first issue is with the JSII distribution itself. Unlike python modules, where the source code is already present in the python library directory, JSII package comes as a tarball containing the typescript files. On each run of the python program, the tarball is extracted to a temporary directory and executed from the temp dir. This is further slowed down by the fact that the file is gzipped, so CPU time must be expended to decompress. Finally, the decompression and unpacking is performed in javascript, which is much slower that the native 'tar' command.

We see no user benefit in having the tarball decompressed on each program run. Instead, we worked around this by unpacking the files once and re-using the same path for each application run. Getting rid of the repeated unpacking already saves almost half of the load time -- we were able to get to about 4 seconds instead of 7.

The other issue is with the fact that aws_cdk is now packaged as myriad modules for each of the aws service (EC2, SSM, ...) altogether. When the JSII initializes, it loads and interprets all the code in one go, as opposed to lazy loading of what is actually required. This would be advantageous if majority of the modules were actually required, but in a realistic project only a handful of the 200+ modules is used.

This results in a terrible waste as we are actually loading thousands of callable methods that are never called. Unfortunately, working around that isn't very easy as it requires editing the huge (50MB+) .jsii manifest which we couldn't do by hand. However, we suspect that this factor is responsible for majority of the rest of the load times.

I'd like to stress that I find it not acceptable to add 7+ seconds to any program's load time without a very good justification. It strikes me as odd that this was not noticed or given weight during internal testing. I do like to work with CDK and I am open to discussing possible solutions, rather sooner than later.

[RomainMuller]

This is odd though, I have only seen complaints about this from Python developers at this stage... The bundling/loading is done in the exact same way in all languages... so why is Python the only one experiencing the slowness?

Or am I missing signals in other languages, too?

[RichiCoder1]

.NET is also painfully slow, to the point where we dropped it in favor of TypeScript. Even TypeScript isn't exactly snappy, though it's better than Python and .NET by a margin.

[RomainMuller]

I have dug deeper on this subject and found out that (sample size = 1):

• ( 65%) 3.331s => tar.extract (while macOS' tar zxf takes 1.764s)
• ( 13%) 0.695s => loadAssemblyFromPath (mostly from parsing a massive JSON document)
• ( 21%) 1.106s => require (consistent with node -e 'require("aws-cdk-lib")')
• ... some milliseconds here and there of other stuff ...
• (100%) 5.160s Total

From this perspective:

1. Un-tar is definitely a top contributor (gunzip is however not, I ran experiments and decompressing the tarball ahead of time does not make much of a difference)
2. The very large jsii assembly processing is heavy, but pivoting to an alternate format can easily become a breaking change... it can be improved, but it'll take time & care
3. The library takes a solid second to load in "pure Node" due to the sheer amount of code that needs parsing.

---

Specifically on some of @du291 claims (by the way, thank you for the detailed writeup):

This is further slowed down by the fact that the file is gzipped, so CPU time must be expended to decompress.

My experimentations demonstrated that the gz pass is immaterial when using macOS' tar command (1.764s vs 1.720s, so 44ms - less than 2%). I have a sample size of 1 however, and tested this on only one platform. If you have data from other platforms/environment where the gap is more substantial, please let me know so I can see to add them to my benchmark posture.

Finally, the decompression and unpacking is performed in javascript, which is much slower that the native 'tar' command.

I initially did not believe the difference would be big enough to be material. When we chose this mechanism about 4/5 years ago, we had run some benchmarks and the npm library performed similar to the tar command... But it turns out at that time, the packages were much smaller... and it turns out the JS version is about 90% slower.

We see no user benefit in having the tarball decompressed on each program run.

Unpacking on every run was deemed safer (guarantees the files on-disk have not been tampered with, removes some race condition risks, etc...). It also consumes less disk space at rest.

Instead, we worked around this by unpacking the files once and re-using the same path for each application run.

Given the research above that confirms your findings, I guess we can easily improve the situation a lot by doing something similar... I don't know how you got around to do your deed here, but if it makes sense & you're able and willing to file a PR with what you have... this might give us a headstart.

This would be advantageous if majority of the modules were actually required, but in a realistic project only a handful of the 200+ modules is used.

This is correct. I would note though that in this particular area, using TypeScript gives you no edge. The way aws-cdk-lib is architected means you can't quite avoid loading the entire library (since App, Stack, etc... are at the package root, and loading that causes all other submodules to be loaded as well).

I'd like to stress that I find it not acceptable to add 7+ seconds to any program's load time without a very good justification.

It's obviously hard to disagree with this statement. In an ideal world, the load performance would be identical (in the same ballpark) regardless of your language of choice (after all, the premise of jsii is that you should be free to choose your language independently of other considerations), and I will treat any excessive "jsii tax" as a defect.

Immediate next steps here are:

• Short term: Remove tar.extract from the hot path
• The CDK team is working on reducing the module size (RFC 39)... we may consider stretching the scope of this work with tactical items that could improve the require time for aws-cdk-lib across all languages if we can identify opportunities there

Longer term (these are non-trivial and risk incurring breaking changes):

• Design a strategy that would allow avoiding eager loading of the entire library
• Remove the need to load/parse the jsii assembly file at runtime

[du291]

Hi @RomainMuller , thank you for the reply and action plan. It's very appreciated! Please find some responses below...

1. Un-tar is definitely a top contributor (gunzip is however not, I ran experiments and decompressing the tarball ahead of time does not make much of a difference)

This is my measurement... I guess every platform is different. One way to reach a compromise can be lz4...

$ time tar tf aws-cdk-lib\@2.38.1.jsii.tgz >/dev/null

real	0m0.659s
user	0m0.649s
sys	0m0.160s

$ time tar tf aws-cdk-lib\@2.38.1.jsii.tar >/dev/null

real	0m0.024s
user	0m0.017s
sys	0m0.007s

$ mkdir _tmp1 _tmp2
$ time tar xf aws-cdk-lib\@2.38.1.jsii.tgz -C _tmp1

real	0m0.880s
user	0m0.729s
sys	0m0.484s
$ time tar xf aws-cdk-lib\@2.38.1.jsii.tar -C _tmp2

real	0m0.429s
user	0m0.030s
sys	0m0.396s

$ uname -a
Linux box 5.17.9-hardened1-1-hardened #1 SMP PREEMPT Thu, 19 May 2022 19:12:41 +0000 x86_64 GNU/Linux

$ grep -i model /proc/cpuinfo 
model		: 44
model name	: Intel(R) Core(TM) i7 CPU         970  @ 3.20GHz

2. The very large `jsii` assembly processing is heavy, but pivoting to an alternate format can easily become a breaking change... it can be improved, but it'll take time & care

Yes, I agree, most of the problems are heavily magnified by the library size. I think jsii has hit a scaling problem-- the discussion should be, do we want to ship all these 200+ modules together? Then we need to massively optimize it... Or do we want people to hand pick the 10 they use, like in CDK1, and then none of that matters... I only speak for myself, I didn't mind having 10 imports and pip requirements.

We see no user benefit in having the tarball decompressed on each program run.

Unpacking on every run was deemed safer (guarantees the files on-disk have not been tampered with, removes some race condition risks, etc...). It also consumes less disk space at rest.

I haven't really thought about tampering here ... I guess all the pip packages in python library (including the CDK python files) are suspect to tampering as well? But what is the risk?

Instead, we worked around this by unpacking the files once and re-using the same path for each application run.

Given the research above that confirms your findings, I guess we can easily improve the situation a lot by doing something similar... I don't know how you got around to do your deed here, but if it makes sense & you're able and willing to file a PR with what you have... this might give us a headstart.

See patch below... it's not in any shape to be pulled into the repo, but gives you the idea. Seeing that you already implemented a cache, it might be moot.

The original idea was to keep the unpacked files with the pip package... (that would take care of old versions, etc). When we tried that, we run into some issues regarding the directory structure that is expected (there needs to be "node-modules", and under it "constructs" and "aws-cdk-lib") and willing not to do symlink shenanigans for the sake of simple measurement, we settled on having a persistent unpacked copy in /tmp (which has the same structure as the on-the-fly copy that would normally appear under a random name there).

The first hunk covers the check of "already-loaded assembly" which possibly incorrectly assumes that from existence of a directory, rather than checking the actual assemblies thing.

Then, we remove the need for mkdir and tar.extract. No science there.

The hunk at 5170 fixates the load directory and removes the hook to delete it.

Finally the last two at 5145 and 5348 have to do with a different thing. We attempted to reduce the load times by commenting out unneeded modules in index.js of the unpacked CDK. This helped a bit, but the loader needed to be made aware that not everything from the .jsii manifest would be loaded. As you wrote elsewhere, probably even more reduction would be achieved by (machine-)editing the manifest, but that's for another day.

This patch is against the webpack version, because we didn't (still don't) have much insight into what's going on so we used quite a lot of reverse engineering on the installed pip module -- YMMV.

--- ./venv/lib/python3.10/site-packages/jsii/_embedded/jsii/lib__program.js	2022-08-24 15:43:38.182513143 +0200
+++ ./venv/lib/python3.10/site-packages/jsii/_embedded/jsii/lib__program.js	2022-08-24 15:43:38.182513143 +0200
@@ -4923,7 +4923,7 @@
                 var _a, _b;
                 if (this._debug("load", req), "assembly" in req) throw new Error('`assembly` field is deprecated for "load", use `name`, `version` and `tarball` instead');
                 const pkgname = req.name, pkgver = req.version, packageDir = this._getPackageDir(pkgname);
-                if (fs.pathExistsSync(packageDir)) {
+                if (this.assemblies && this.assemblies.has(pkgname)) {
                     const epkg = fs.readJsonSync(path.join(packageDir, "package.json"));
                     if (epkg.version !== pkgver) throw new Error(`Multiple versions ${pkgver} and ${epkg.version} of the package '${pkgname}' cannot be loaded together since this is unsupported by some runtime environments`);
                     this._debug("look up already-loaded assembly", pkgname);
@@ -4933,17 +4933,8 @@
                         types: Object.keys(null !== (_a = assm.metadata.types) && void 0 !== _a ? _a : {}).length
                     };
                 }
-                fs.mkdirpSync(packageDir);
                 const originalUmask = process.umask(18);
                 try {
-                    tar.extract({
-                        cwd: packageDir,
-                        file: req.tarball,
-                        strict: !0,
-                        strip: 1,
-                        sync: !0,
-                        unlink: !0
-                    });
                 } finally {
                     process.umask(originalUmask);
                 }
@@ -5145,10 +5136,12 @@
                       case spec.TypeKind.Class:
                       case spec.TypeKind.Enum:
                         const constructor = this._findSymbol(fqn);
+                        if (constructor) {
                         (0, objects_1.tagJsiiConstructor)(constructor, fqn);
                     }
                 }
             }
+            }
             _findCtor(fqn, args) {
                 if (fqn === wire.EMPTY_OBJECT_FQN) return {
                     ctor: Object
@@ -5170,9 +5163,9 @@
                 }
             }
             _getPackageDir(pkgname) {
-                return this.installDir || (this.installDir = fs.mkdtempSync(path.join(os.tmpdir(), "jsii-kernel-")), 
+                return this.installDir || (this.installDir = '/tmp/jsii-cdk/'),
                 this.require = (0, module_1.createRequire)(this.installDir), fs.mkdirpSync(path.join(this.installDir, "node_modules")), 
-                this._debug("creating jsii-kernel modules workdir:", this.installDir), onExit.removeSync(this.installDir)), 
+                this._debug("creating jsii-kernel modules workdir:", this.installDir),
                 path.join(this.installDir, "node_modules", pkgname);
             }
             _create(req) {
@@ -5348,6 +5341,9 @@
                 for (;parts.length > 0; ) {
                     const name = parts.shift();
                     if (!name) break;
+                    if (!curr) {
+                        return null;
+                    }
                     curr = curr[name];
                 }
                 if (!curr) throw new Error(`Could not find symbol ${fqn}`);

[RichiCoder1]

Yes, I agree, most of the problems are heavily magnified by the library size. I think jsii has hit a scaling problem-- the discussion should be, do we want to ship all these 200+ modules together? Then we need to massively optimize it... Or do we want people to hand pick the 10 they use, like in CDK1, and then none of that matters... I only speak for myself, I didn't mind having 10 imports and pip requirements.

To be fair, part of the issue is the CDK is a super package at both the package and the code level. The CDK could still be a "super" package (e.g., only one pip/npm install to get all the things you need), but not be tightly linked together like it is today (import what you need only). The download size would still be large, but JSII compression and the proposed lambda layer changes make that more tolerable as a one-time cost.

[pauldraper]

The bundling/loading is done in the exact same way in all languages... so why is Python the only one experiencing the slowness?

Loading other languages are also slow, but Python loading is ~3x the others.

And it's not a difference of 50ms vs 150ms. It's 3s vs 9s.

[mcouthon]

Hi, I reached this issue after encountering slow import times for cdk based libraries (we're using cdktf, but the "culprit" is jsii).
Once I saw the experimental caching solution, I immediately tried to use it, but couldn't get it to work (nothing appears in the chosen directory, and I'm not seeing any time improvement).

I've started this discussion to try to understand why it isn't working. So if anyone comes to this issue and is having the same problem, follow the discussion there.

[matthewpick]

I've also noticed the same issue on my Windows machine, import time is slightly faster running on Ubuntu WSL2.

My current workaround for this issue is to avoid installing the entire aws-cdk-lib, and instead only pip install what I need.
For example:

# old-requirements.txt
aws-cdk-lib==2.73.0

# new-requirements.txt
aws_cdk.core==1.200.0
aws_cdk.aws_lambda==1.200.0

This brings my aws_cdk.* import time down by 10x, which makes writing unit tests much faster. My import time went from 30 seconds down to 3-5 seconds on average. Huge improvement. 3-5 seconds is still pretty bad, but it is manageable.

[ermanno]

@matthewpick Yes, but this way you are using CDKv1 which is on maintenance mode (only receiving critical bug fixes and security patches) since June 1, 2022 and will stop receiving any support on June 1, 2023.

[matthewpick]

Yes, but this way you are using CDKv1 which is on maintenance mode...

@ermanno Thanks for the clarification. You are correct, I recently started using CDK and wasn't aware that module-based import/installation was only a thing in v1, which is unfortunate. I guess I'll just put up with the ~30s import time for the time being. Definitely slows down incremental development of new infrastructure via cdk deploy.

[mcouthon]

Note that caching has been implemented, and allows for much quicker subsequent runs.

[rirze]

Note that caching has been implemented, and allows for much quicker subsequent runs.

This is great for development, but my use case is CI/CD where it's often a cold start or just one execution of import App ... per pipeline. The same delay will persist for those use cases...

[RomainMuller]

#4181 will provide improvements to the JavaScript side of the load problem.

---

There is still an apparent issue where loading any part of the Python generated code results in ALL of it being loaded, which can be slow due to the sheer amount of code this amounts to. There may be avenues to generate code differently to avoid this particular behavior, but I'm not entirely sure how and this requires further research to avoid causing breaking changes.

---

Note that #3895 (comment) has been implemented, and allows for much quicker subsequent runs.
This is great for development, but my use case is CI/CD where it's often a cold start or just one execution of import App ... per pipeline. The same delay will persist for those use cases...

Your CI/CI can persist the cache location and re-use it in between runs. Generally speaking, CI/CD also can afford to be a little slower as there's normally no human patiently waiting for it to be done before they can do some work... As far as I'm aware, we're talking about seconds here (maybe enough to make a couple of minutes), not hours...

[rirze]

Your CI/CI can persist the cache location and re-use it in between runs. Generally speaking, CI/CD also can afford to be a little slower as there's normally no human patiently waiting for it to be done before they can do some work... As far as I'm aware, we're talking about seconds here (maybe enough to make a couple of minutes), not hours...

I would be fine it was seconds. Removing the part that imports all modules from __init__.py saves a minute from each run. It reduces the runtime from 2 mins+ to about 30 secs. If you add up 10s of executions per day, the wasted hours over a month start to look depressing. Our CDKTF stack is about 1000 resources, which honestly is not a very large environment. We also work in a secure environment where devs can't run cdk's cli commands locally, so they are waiting patiently for the CI/CD run to finish to get feedback.

Needless to say, I speculate there's a very great runtime optimization waiting to be implemented here. Similar to how Python's typing.TYPE_CHECKING is used, maybe JSII could utilize a branch to skip loading all the modules in runtime scenarios? I think in another thread, a contributor mentioned that the imports are necessary for loading all the type information across modules, but my limited testing in removing the imports shows that they are not necessary. The application compiles fine without importing all modules first. This could be a Python peculiarity afaik that allows this to work.

[mcouthon]

We've implemented caching in CI and it's working reasonably well. It's a little ugly, but it works — we send an archive of the local cache (on main branch only) to S3, and pull it on branch runs.

[rirze]

We have since fixed the caching issue as well-- albeit the first run after any update seems to choke the memory of the instance it's running on. Had to upgrade the instance type to 8gb get any reliable runs. Kinda ridiculous that 8gb is the minimum needed but I guess that's the kind of applications jsii and its derivatives are...

[AmirTNinja]

Hey, can anyone pls update regarding this issue?

We still getting this performance issue delay of ~30seconds in our test suites and its really unbearable.
we currently using "aws-cdk-lib=2.151.0" version, which is quite new and looks at the release notes of the most new ones and didn't notice any fix on that issue.

important notes - we're using WINDOWS Platform not MAC

Please assist !

[AmirTNinja]

Hey, can anyone pls update regarding this issue?

We still getting this performance issue delay of ~30seconds in our test suites and its really unbearable. we currently using "aws-cdk-lib=2.151.0" version, which is quite new and looks at the release notes of the most new ones and didn't notice any fix on that issue.

important notes - we're using WINDOWS Platform not MAC

Please assist !

anyone?

[hanwen-cluster]

This issue is preventing AWS ParallelCluster from migrating to AWS CDK v2. Could you please raise the priority?