JsiiErrors / EAGAIN when using node>=13.2 with Python, Java & .NET #5187
Assignees
Labels
bug
This issue is a bug.
language/python
Related to Python bindings
p2
package/tools
Related to AWS CDK Tools or CLI
Comments
DrLuke
added
bug
SomayaB
added
language/python
|
@DrLuke - Can you do a |
shivlaks
added
the
response-requested
|
I tried, but I get the same error message. |
|
@DrLuke - can you share the output with a |
|
@shivlaks Like this? |
|
This happened to me with a project that has already been deployed successfully multiple times before upgrading to |
|
@DrLuke - gotcha that's good to know, what version was the app created with? I'd like to get full reproduction steps so I can dig deeper |
|
@shivlaks Probably around |
|
@DrLuke - got it. Does your |
|
No, it's empty |
|
@DrLuke - can you run |
|
Nope |
|
@DrLuke - seems like a bug. we'll investigate further. really appreciate the additional information. It should help us get to the root cause sooner |
|
Thank you! Let me know if you need any further details. |
|
Looks like a Jsii runtime issue. @RomainMuller can you take a look? |
|
Looks like the jsii Kernel (JS) gets EAGAIN trying time read from its own STDIN. It might help to know the version of node that you re using, too ( |
|
So I have tried running a reproduction and haven't managed to hit the issue to this point... In order to avoid us going back and forth asking for additional information (you already provided a lot, and that's great!), let me ask for all the stuff that comes to mind now and could play a role in this problem you're having:
That's all I could think of at this point which could contribute to causing your issues... Hopefully this helps give us a better understanding of what is happening. |
|
I encountered the same error. Here are my reproduction steps and information on my environment. Reproduction steps$ mkdir -p cdk_error && cd $_
$ cdk init app --language=python
Applying project template app for python
Initializing a new git repository...
Executing Creating virtualenv...
(...snip...)
Enjoy!
$ source .env/bin/activate
(.env)
$ pip install -r requirements.txt
(...snip...)
Successfully installed attrs-19.3.0 aws-cdk.core-1.18.0 aws-cdk.cx-api-1.18.0 cattrs-0.9.0 cdk-error jsii-0.20.8 publication-0.0.3 python-dateutil-2.8.1 six-1.13.0 typing-extensions-3.7.4.1
(.env)
$ JSII_DEBUG=1 cdk synth -v &> error.log
(.env)
$ cat error.log
CDK toolkit version: 1.18.0 (build bc924bc)
Command line arguments: {
_: [ 'synth' ],
v: true,
verbose: true,
'ignore-errors': false,
ignoreErrors: false,
json: false,
j: false,
ec2creds: undefined,
i: undefined,
'version-reporting': undefined,
versionReporting: undefined,
'path-metadata': true,
pathMetadata: true,
'asset-metadata': true,
assetMetadata: true,
'role-arn': undefined,
r: undefined,
roleArn: undefined,
staging: true,
'no-color': false,
noColor: false,
'$0': '/usr/local/bin/cdk'
}
Determining whether we're on an EC2 instance.
Does not look like EC2 instance.
cdk.json: {
"app": "python3 app.py"
}
cdk.context.json: {
"@aws-cdk/core:enableStackNameDuplicates": "true"
}
merged settings: {
versionReporting: true,
pathMetadata: true,
output: 'cdk.out',
app: 'python3 app.py',
context: {},
tags: [],
assetMetadata: true,
toolkitBucket: {},
staging: true
}
Setting "CDK_DEFAULT_REGION" environment variable to ap-northeast-1
Resolving default credentials
Retrieved account ID XXXXXXXXXXXX from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to XXXXXXXXXXXX
context: {
'@aws-cdk/core:enableStackNameDuplicates': 'true',
'aws:cdk:enable-path-metadata': true,
'aws:cdk:enable-asset-metadata': true
}
outdir: cdk.out
env: {
CDK_DEFAULT_REGION: 'ap-northeast-1',
CDK_DEFAULT_ACCOUNT: 'XXXXXXXXXXXX',
CDK_CONTEXT_JSON: '{"@aws-cdk/core:enableStackNameDuplicates":"true","aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
CDK_OUTDIR: 'cdk.out',
CDK_CLI_ASM_VERSION: '1.16.0',
CDK_CLI_VERSION: '1.18.0'
}
> {"name":"@aws-cdk/cx-api","version":"1.18.0","tarball":"/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/cx_api/_jsii/cx-api@1.18.0.jsii.tgz","api":"load"}
[jsii-kernel] load {
name: '@aws-cdk/cx-api',
version: '1.18.0',
tarball: '/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/cx_api/_jsii/cx-api@1.18.0.jsii.tgz',
api: 'load'
}
[jsii-kernel] creating jsii-kernel modules workdir: /var/folders/3x/vrvl2bs171l55csv_l2zb1hh0000gn/T/jsii-kernel-NvSBHH
[jsii-kernel] removing staging directory: /var/folders/3x/vrvl2bs171l55csv_l2zb1hh0000gn/T/jsii-kernel-install-staging-c3OO11
< {"ok":{"assembly":"@aws-cdk/cx-api","types":29}}
> {"name":"@aws-cdk/core","version":"1.18.0","tarball":"/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/core/_jsii/core@1.18.0.jsii.tgz","api":"load"}
[jsii-kernel] load {
name: '@aws-cdk/core',
version: '1.18.0',
tarball: '/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/core/_jsii/core@1.18.0.jsii.tgz',
api: 'load'
}
[jsii-kernel] removing staging directory: /var/folders/3x/vrvl2bs171l55csv_l2zb1hh0000gn/T/jsii-kernel-install-staging-BZtDkO
< {"ok":{"assembly":"@aws-cdk/core","types":108}}
[jsii-kernel] removing install dir /var/folders/3x/vrvl2bs171l55csv_l2zb1hh0000gn/T/jsii-kernel-NvSBHH
internal/fs/utils.js:220
throw err;
^
Error: EAGAIN: resource temporarily unavailable, read
at Object.readSync (fs.js:516:3)
at SyncStdio.readLine (/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13322:25)
at InputOutput.read (/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:13272:34)
at KernelHost.run (/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7228:32)
at Immediate.<anonymous> (/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7236:37)
at processImmediate (internal/timers.js:439:21) {
errno: -35,
syscall: 'read',
code: 'EAGAIN'
}
Traceback (most recent call last):
File "app.py", line 8, in <module>
app = core.App()
File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_runtime.py", line 66, in __call__
inst = super().__call__(*args, **kwargs)
File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/aws_cdk/core/__init__.py", line 3443, in __init__
jsii.create(App, self, [props])
File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 223, in create
interfaces=[iface.__jsii_type__ for iface in getattr(klass, "__jsii_ifaces__", [])],
File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 333, in create
return self._process.send(request, CreateResponse)
File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 310, in send
self._next_message(), _ProcessResponse_R
File "/Users/skatsuta/tmp/cdk_error/.env/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 261, in _next_message
return json.loads(self._process.stdout.readline(), object_hook=ohook)
File "/Users/skatsuta/.pyenv/versions/anaconda3-2019.03/lib/python3.7/json/__init__.py", line 361, in loads
return cls(**kw).decode(s)
File "/Users/skatsuta/.pyenv/versions/anaconda3-2019.03/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Users/skatsuta/.pyenv/versions/anaconda3-2019.03/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Subprocess exited with error 1
Error: Subprocess exited with error 1
at ChildProcess.<anonymous> (/usr/local/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:115:23)
at ChildProcess.emit (events.js:210:5)
at ChildProcess.EventEmitter.emit (domain.js:478:20)
at Process.ChildProcess._handle.onexit (internal/child_process.js:270:12)
(.env)Our AWS account ID is masked here for security reason, but actually the correct ID is set. Environment# OS
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.13.6
BuildVersion: 17G9016
# AWS CDK
$ cdk --version
1.18.0 (build bc924bc)
# Node
$ node --version
v13.2.0
# Python
$ python --version
Python 3.7.3
# Python packages
$ pip list
Package Version Location
----------------- ------- ---------------------------------------
attrs 19.3.0
aws-cdk.core 1.18.0
aws-cdk.cx-api 1.18.0
cattrs 0.9.0
cdk-error 0.0.1 /Users/skatsuta/tmp/cdk_error/cdk_error
jsii 0.20.8
pip 19.0.3
publication 0.0.3
python-dateutil 2.8.1
setuptools 40.8.0
six 1.13.0
typing-extensions 3.7.4.1
(.env)
# Env vars related to Node
$ env | grep NODE
(.env)
# Memory
# I'm using macOS, so I show the output of vmstat and top instead of `cat /proc/meminfo`
$ vm_stat
Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free: 95560.
Pages active: 1512991.
Pages inactive: 1138637.
Pages speculative: 142651.
Pages throttled: 0.
Pages wired down: 1067127.
Pages purgeable: 50024.
"Translation faults": 1267941904.
Pages copy-on-write: 45647892.
Pages zero filled: 608317362.
Pages reactivated: 68208285.
Pages purged: 29002867.
File-backed pages: 529691.
Anonymous pages: 2264588.
Pages stored in compressor: 6440947.
Pages occupied by compressor: 237143.
Decompressions: 98763361.
Compressions: 132396336.
Pageins: 58923959.
Pageouts: 335282.
Swapins: 136352534.
Swapouts: 141541295.
$ top -l 1 -s 0 | grep PhysMem
PhysMem: 15G used (4258M wired), 711M unused.Thank you for your sincere support. Hope it helps! |
|
|
I am having the same issue. Tried downgrading CDK dependencies to 1.16.3, but it still failed with 1.17.1 cli and latest node. Having downgraded node to 12.13.1 helped me. |
|
Had exactly the same issue (on 2 different machines, local & CI/CD). Tried downgrading CDK & dependencies to previous version but still saw the same issue. |
|
I have prepared a Dockerfile which reproduces the issue. In the unzipped directory run: (Warning, python3.7 has to be compiled when building the image, might take a while) I hope this helps to narrow down the issue. |
|
Using the Dockerimage I can confirm that downgrading node to version |
|
same here. Downgraded to stable node version: 12.3.1 fixed the issue I was on 13.3 |
I had the same issue. After downgraded node version: 12.3.1. problems was solved. |
4 tasks
RomainMuller
added a commit
to aws/jsii
that referenced
this issue
Dec 19, 2019
When using node >= 13.2, attempts to synchronously read from STDIN often result in `EAGAIN` being raised. This is due to the STDIN file descriptor being opened with `O_NONBLOCK`. This introduces a temporary workaround that catches the EAGAIN exception and re-attempts reading from STDIN "immediately". While this is not the ideal solution, it can be done right away and appears to have minimal impact on the overall performance of applications. A longer term fix is to move away from synchronous IO operations, but this is a significantly mroe involved fix and migth be a while before this can be delivered. The long term solution is tracked in #1142. Related to aws/aws-cdk#5187
RomainMuller
added a commit
to aws/jsii
that referenced
this issue
Dec 19, 2019
When using node >= 13.2, attempts to synchronously read from STDIN often result in `EAGAIN` being raised. This is due to the STDIN file descriptor being opened with `O_NONBLOCK`. This introduces a temporary workaround that catches the EAGAIN exception and re-attempts reading from STDIN "immediately". While this is not the ideal solution, it can be done right away and appears to have minimal impact on the overall performance of applications. A longer term fix is to move away from synchronous IO operations, but this is a significantly mroe involved fix and migth be a while before this can be delivered. The long term solution is tracked in #1142. Related to aws/aws-cdk#5187
RomainMuller
added a commit
to aws/jsii
that referenced
this issue
Dec 19, 2019
…1143) When using node >= 13.2, attempts to synchronously read from STDIN often result in `EAGAIN` being raised. This is due to the STDIN file descriptor being opened with `O_NONBLOCK`. This introduces a temporary workaround that catches the EAGAIN exception and re-attempts reading from STDIN "immediately". While this is not the ideal solution, it can be done right away and appears to have minimal impact on the overall performance of applications. A longer term fix is to move away from synchronous IO operations, but this is a significantly mroe involved fix and migth be a while before this can be delivered. The long term solution is tracked in #1142. Related to aws/aws-cdk#5187
|
Tested aws-cdk Python 1.20.0 now works with node v13.6.0 |
|
Tested aws-cdk Python So I guess it's once again a regression from what @chefren says? It now yields this on my OSX machine: |
|
@chefren can you confirm that node node v13.6.0 is still working for you with python cdk v1.22.0? As far as I know we haven't made any changes that would lead to node v13.6.0 working. |
|
Yes, confirmed - note: I'm only checking Python and using brew to handle node Brew: had to follow this to manage brew toggling correctly node versions. Also upgraded CDK again (which included node upgrade) and retested OK |
|
@chefren Did you test |
|
yes, version is just a reference |
@brainstorm I noted you are not using pip and venv , but miniconda, this is suspicious in your stack trace, maybe not pointing at the correct libraries? Check the python libraries versions you are using as well as node and cdk ;) |
|
Indeed @chefren , that was the culprit, I fixed it like this: Since I have quite a few of aws-cdk modules on that virtualenv (miniconda, but same thing)... IMHO, there should be a toplevel aws-cdk metapackage that updates all submodules in lockstep by just running:
As I raised on #3406 (comment) and also mentioned on issue #972 ... this issue will keep coming up folks. The current xargs hack above is not pythonic and should be fixed if you would like to avoid more issues/requests/bugs from the python community ;) Cheers! |
|
@brainstorm this is definitely under discussion. Keep an eye out for updates. For others watching this issue: aws/jsii#1143 implemented a short term solution to retry failed reads from stdin immediately after aws/jsii#1142 is the long term fix but it requires significant investigation. |
|
I am also experiencing an issue with the latest I filed an issue #8288 but it might be the same as this one. |
RomainMuller
added a commit
to aws/jsii
that referenced
this issue
Jun 8, 2020
…eadLine When using node `>= 12.17`, `EAGAIN` errors consistently occur when trying to read from `stdin` when there is no available data. The retry mechanism for this was to recursively call back `SyncStdio.readLine`, which could evtnually lead to a "Maximum call stack size exceeded" error to occur (possibly hidden from the consumer, and later causing a "Stream closed" error). This changes how the retry mechanism works so it operates in a `while` loop instead of making a recursive call, completely avoiding to run into the growing stack issue. Fixes aws/aws-cdk#8288 Fixes aws/aws-cdk#5187
|
Linking this for posterity: nodejs/help#2663, it appears to contain a lot of interesting information. Definitely reinforces me thinking it's likely due to how the pipe is set-up, triggering |
|
Hey what if we used a custom FD? So not |
|
@rix0rrr - I have tried opening As I'm saying in the comment on my PR, we could set up our own channel & control it in however way we want, but cross-platform is challenging (named pipes, and the likes... Windows could be problematic) |
mergify bot
pushed a commit
to aws/jsii
that referenced
this issue
Jun 11, 2020
…eadLine (#1717) When using node `>= 12.17`, `EAGAIN` errors consistently occur when trying to read from `stdin` when there is no available data. The retry mechanism for this was to recursively call back `SyncStdio.readLine`, which could evtnually lead to a "Maximum call stack size exceeded" error to occur (possibly hidden from the consumer, and later causing a "Stream closed" error). This changes how the retry mechanism works so it operates in a `while` loop instead of making a recursive call, completely avoiding to run into the growing stack issue. Fixes aws/aws-cdk#8288 Fixes aws/aws-cdk#5187 Fixes aws/aws-cdk#8397 --- By submitting this pull request, I confirm that my contribution is made under the terms of the [Apache 2.0 license]. [Apache 2.0 license]: https://www.apache.org/licenses/LICENSE-2.0
|
I have node 18.17.1 and I have this error: Error: ENOTEMPTY: directory not empty, rmdir '\?\C:\Users\ronyt\AppData\Local\Temp\jsii-kernel-etkuhF\node_modules@aws-cdk\asset-awscli-v1\projenrc' |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
When deploying, some file is temporarily unavailable.
Reproduction Steps
Then deploy
$ cdk deploy --app "python stack.py"Error Log
Workaround for the time being
Downgrade node to version
12.13.1Environment
This is 🐛 Bug Report
The text was updated successfully, but these errors were encountered: