pantsd seg faults when trying to build large target #5162
Comments
|
The stack involving Thanks for the report! |
|
@youngderekm : As a workaround, you should be able to set |
|
I tried to restart the daemon with |
|
(fyi: can put triple backticks around code to improve readability) |
|
Interesting! Yes, that looks different. I don't repro with Python |
|
Failing in Is there anything unusual about the jvm platforms that are in use? |
|
I upgraded python (with About jvm platforms, I have this configuration and use |
|
I notice 2.7.10_2 in the stacks, so I must not have done the right thing to upgrade. I'll try again. |
|
I was able to upgrade python. I'm still having a number of problems. I won't keep pasting verbose callstacks unless that's helpful, but a few things I ran into are below: I often get a timeout watching for watch-project: I upped
Sometimes I get a permission error running I get this seg fault sometimes (pasting current thread only): I get these problems running |
|
seg fault/stacks from another compile attempt (this time with watchman_startup_timeout at 300): |
|
Is this machine memory constrained by any chance? Wonder if you're getting OOM killed. |
|
seems odd that the segfault is happening consistently across python versions. Stu's idea of looking at resource utilization with Activity Monitor et al during the repro seems like a good first step. are you also able to repro this against current master? what does the Current Thread and 3 other threads point to this same stack in the pinger in at least 2 of the stacktraces: which would make me want to look for abnormalities in but it's not clear to my how that would relate to the daemon. you never see these crashes with the daemon disabled? |
|
Hi, I tested again today. Sorry for the delay. I am now using 1.4.0.dev24. I've restarted the machine since I last test and I'm running with fewer applications running (quit Eclipse, etc. so I have more RAM free). It's working better this time. I've done a few compiles without crashes. It's possible like you suggested it was a resource exhaustion issue. I see that watchman in particular is taking a large amount of RAM, 2.90 GB at the moment. Looking at watchman logs, it looks like watchman is watching directories like .pants.d and .cache. Could that be a reason that it's using so much RAM? Is it possible to only watch source files? To answer your question about CPU cores, I'll keep testing with pantsd enabled so see if I get any more crashes. |
|
ah, great. but it still crashes somewhat frequently? watchman taking up 2.90GB sounds super abnormal to me. on our internal monorepo (very large), it clocks in at 240MB for me. I've also just bumped the watchman version in master to a build that should be more efficient fwiw. I also wouldn't expect |
|
@youngderekm : Hey! We've been able to begin an internal beta of the daemon as of approximately the |
|
Hi - I'm testing again with pants 1.4.0rc0 and I'm having better success. I've done some builds with it and haven't seen a crash or other weird behavior yet (just used for half a day again though). watchman is using 759MB now, which is much better than it was, but still higher than I would like (developers are constrained on RAM already). Is there any way to dump out what watchman is watching? or get any other internal stats from it? |
|
I'm not aware of a way to dump stats from watchman, unfortunately - but 800MB still seems high to me. I can tell you that:
|
|
Ok, it sounds like this report is not a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm trying to test using pantsd. I'm running 1.4.0.dev22. I enable with
enable_pantsd: True. If I use./pants compileon a reasonably small target it works as expected. If I compile our "world" target that depends on most other targets in our build, pantsd crashes. I see a segmentation fault in the pantsd.log file.pantd.log file output below. Please let me know if I should gather any additional information:
The text was updated successfully, but these errors were encountered: