tests/Frontend/ember-build-tests: Unhandled error event #2260
Comments
briancain
changed the title
|
My guess is that some aspect of the UI build pipeline has a memory leak, leading it to exhaust the available memory in the CI container. See angular/angular-cli#8331 (angular-cli is a fork of ember-cli). I don’t have a ton of evidence for this, just a hunch. |
briancain
changed the title
|
The key message to look at here is the |
|
Let’s work backwards from the key Though it appears in multiple places, it’s always a dependency of ember-cli-typescript, so it seems reasonable to focus our investigation on ember-cli-typescript. Can we upgrade ember-cli-typescript? What version(s) are we running right now? We’re using five different version of ember-cli-typescript across the dependency tree. Thanks NPM 🤦♀️ What version(s) of TypeScript itself are we using? That’s more reasonable at least. Current working theory: Next step: |
|
Alternative next step: |
|
There’s no guarantee #2264 fixes this. So let’s keep an eye in things and, if this flapper recurs, re-open the issue and continue the investigation. |
|
Hmm, I've seen this fail a couple of times already today. 😢 I think I'll reopen this. |
|
@briancain could you point me in the direction of a failure? Would be good to see if the output has changed. |
|
I think the next relatively cheap thing to try is bumping the version of Node we use. |
|
Managed to invalidate the Node 14 hypothesis very quickly 😅 Next step: Alternative: |
|
I have some workflows running with |
|
Managed to get a 137 exit code from running Next step: |
|
Ran with It certainly seems like problems occur in conjunction with the two large files: Next step: |
|
Additional avenue of investigation: |
|
With A thought: if the problem is the typescript type checker running out of memory, can we just disable the check during a build? ember-cli-typescript doesn’t actually use typescript to compile the files, just to type check them. Babel does the compilation. I’m not sure there’s an option for this, still reading the ember-cli-typescript source. |
|
On a successful built, we get this output with Followed by a bunch of other stuff. Trying to trigger a failing build to see if those messages are the same. Note to self: still investigating ember-cli-typescript because it’s the only thing that uses stagehand. |
|
I’m now in the delightful situation where about five builds in a row have succeeded, without changing anything other than the DEBUG setting 😬 |
|
OK, here we go: So for all the world it appears that type checking completed successfully, and quickly. |
|
Nagging thought: what if the process is being killed by docker due to an out-of-memory error, and this stagehand stack trace just happens to be the wait that scenario manifests? i.e. maybe ember-cli-typescript is the wrong place to look. Good to rule it out, I guess. |
|
Another avenue of investigation: |
|
Note: the container is supposed to have 4Gb of memory. I find it hard to believe our build would consume that without a rampant memory leak. |
|
On the “when did this start?” trail, we should look at the following PRs:
And see if we can produce and ember-build-tests failure on the commit immediately prior to each. |
|
Another line of investigation: Checking how much memory the build actually uses. |
|
Here’s my script, monitoring memory usage over time for cd ui
CLEAR_BROCCOLI_PERSISTENT_FILTER_CACHE=true yarn build:test >/dev/null 2>/dev/null &
while true
do
date
ps | grep node | egrep -o '[0-9]{3,}' | xargs -n 1 -I "{}" grep ^VmPeak /proc/{}/status
sleep 1
done |
|
Assuming I’ve done my data collecting and munging correctly (highly unlikely), we use a lot of memory during that build:
|
|
Here’s another using
|
|
If these memory usage stats are to be believe, then @izaaklauer wisely notes:
|
|
It’s the end of the week for me, and at this point in the investigation, I’m feeling fairly confident we’re dealing with a memory leak or runaway memory consumption by either a bug in one of our build dependencies or perhaps a bug caused by the confluence of our build dependencies. Next step: |
|
Possible next step: |
|
@gregone found a good guide to taking heap snapshots in Node: |
|
Other things to try:
|
|
#2287 removes almost a minute from the build. We'd need to make sure it all still works for the prod build |
|
Tried adding the script from above to a debug branch and got this (from current
So it’s using all the available real memory. My |
|
Summary of what I found while experimenting with #2299 & #2297
I updated #2297 to use |
|
#2355 also addresses this. |
Describe the bug
The ember build frontend tests seem to be failing often. I think this test specifically has failed for me about ~5 times today.
https://app.circleci.com/pipelines/github/hashicorp/waypoint/9896/workflows/0f6d5e84-1e59-47ef-9916-75f8f0d22a36/jobs/87707
Steps to Reproduce
The tests flake occasionally on PRs and commits to main.
Expected behavior
They pass.
Waypoint Platform Versions
N/A
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: