v1.30.4 : Pants Filter operation seems stuck #12997
Comments
MrRexZ
changed the title
|
Are the PANTS_CONFIG_FILES=pants.ci.toml ./pants ... |
|
@MrRexZ : I presume that this issue also reproduces with the default logging ( |
|
@jsirois Thanks for the fast reply! :) Yes, I think they are reasonable values (maybe they can be further improved to speed things up, but I don't think it'll fix the issue I'm having). I've never changed the heap size (as I had never set the max size in my code), so I assume the system will use as much as it's allocated |
|
@stuhood Yes, I first encountered it with default logging. I'll try using lower log levels to see if the situation improves.
Thanks a lot for the info! Going off on tangent for a bit, our team has been starting to use Pants, and we have been using Pants 1.3x as Pants 2.x doesn't support JVM languages, and occasionally in our developers' machines, we're sometimes getting pants processes being stuck on some pants operations (which got resolved when we terminate Pants processes, and re-run). Good to hear that there are some improvements in 2.x, but we would be super grateful if we can have those improvements on Pants version we can run that'll allow us to run JVM-based language (Java and Scala), such as a backport or to have them working in Pants V2 :) Thank you! |
|
Ok, thanks. The next step will unfortunately likely be attaching a debugger and getting a stacktrace: you'll need Are you able to open an SSH session into it while the process is hung? If so, things are much easier: you can determine the pid of the hung process, and then run: If you are unable to attach to the container, you should add the following script to your CI environment: #!/bin/bash
set -eux
sleep $1
pid=$(pgrep -f "$2" | head -n 1)
echo -e attach $pid'\nset pagination off\nthread apply all bt' | sudo gdb...and then call it from your CI script immediately above calling # Launch in the background, sleep 360 seconds, and then backtrace the first process named `pantsd`.
./sleep_then_backtrace 360 pantsd & |
|
I've just had the time to get the logs and I have done some changes since there (I replaced the I've managed to obtain the stacktrace according to your suggestion above, let me know if the log attached in this post is helpful. |
|
Thanks a lot! That trace is exactly what we needed. It looks like you found a new issue: I've posted #13127 with the fix, and will get it cherrypicked to |
|
Pants 1.30.5rc1 is now released containing this fix: thanks for your patience! https://pypi.org/project/pantsbuild.pants/1.30.5rc1/ |
Describe the bug
Hi there,
I'm using Pants (version 1.30.4) both on my CI machines and local machines to run some automated tests in Scala.
I've been noticing issues with Pants operations sometime being stuck when running via CI.
In my script, I'm making use of
pants filtercommand to decide what part of input target to be passed into our test, and it's always during the./pants filterstep that the Pants operation stops working.Does anyone have any pointers on how to further debug this issue so that I can figure out what's the root cause?
Thank you! :)
Please see the attached here for relevant logs.
Here are part of the snippets from the logs (in the end, i had to manually cancel this CI job, and from the timestamp, there's a huge time gap which indicates that the
pants filterare stuck):The part it's stuck is always different on every run. In this case, it's
workunit_store: Starting: Reading dtl-python-sdk/datalogue/models/transformations, but in other cases, it's reading other files.So I don't think it's the files being the actual problem, but possibly due to the actions that happened during the reading the files.
I've tried running the command locally (where most of the other operations run are different from the ones usually run in CI, of course), and it works for the pants filter command.
Here's my
pants.tomlconfig:Pants version
1.30.04
OS
Linux
The text was updated successfully, but these errors were encountered: