waiting on log randomly doesn't work #767
Comments
|
@RediJedi Could you please try the following pattern: Does this help ? |
|
Thanks for the prompt reply! Unfortunately it's more of the same: |
|
You mean that is sporadically fails ? Or does it always fail ? Could you please run it with |
|
This is a VERY large file. Here is the output: Log outputIt seems as though the output is different! This is strange since the build runs the same each time. I can execute the build manually ten times and a smattering will fail, guaranteed. However, it seems as though instead of printing |
|
Nope! Spoke too soon. Those are expected. Here is yet another failure, but without the -X out: This was after I added the other string into an or in the regex. So neither work. |
|
Thanks, looks interesting. How do you access the Docker daemon ? Via TCP or via Unix socket ? I ask because unfortunately there tend to be concurrency issues within the unix socket library we are using (jnr-unixsocket) which are not yet fully resolved. If you are using Unix sockets, could you try it with TCP, too ? Another experiment could be to disable the log output to the console as then there is then only one thread fetching the logs (the waiter thread). |
|
If you feel fancy (and you are using Unix sockets which I suspect), could you please try out dmp |
|
Unfortunately no, we're using TCP to access the daemon. The entire system is (basically) Jenkins running on EC2 using Docker to spin up Jenkins slaves. These slaves run the builds which, in this case, use the same daemon to spin up the containers configured in the mvn pom. When you say disable the log output, do you mean at the app level? Currently logging dumps to sys out. What will the waiter thread read if we're not writing the logs? |
|
@RediJedi thanks for the info. It reads the log internally (not showing on the console). So when you log out to stdout, you fetch the logs twice: One for the output, one for the wait-thread. So my idea was to check whether it's a threading issue between those two threads. Is there any chance that I can reproduce it? I will try it later today locally but until now I wasn't able to re-produce the error. |
|
I don't think this will be of any help, but I've hit this issue as well. Specifically over a year ago, with an old version of the plugin and against an old version of Docker (1.7.1). I was ultimately able to use an url/http based wait instead and ended up forgetting about this, but I just hit it again with a slightly newer version of the plugin (0.15.16). Both of these were on an ancient RHEL6 box, using the TCP socket. The log wait did work fine on a Win 7 and 10 box. |
|
We are also having this problem. Any news on resolving it? Are there any known workarounds? We are seeing the problem with plugin version 0.21.0 on Centos with docker 17.05. When we run the same project on mac with Boot2docker with 17.04 we have no issues. In docker logs we can see the log message we are waiting for, but it is not picked up in the plugin and it times out. This happens randomly on our images. |
|
@robzet @Vulcannis I really tried hard to reproduce, but I still can't. The best thing would be to have a reproducible, minimal test case (even it happens only from time to time). In order to narrow down the issue further and to collect some empiric data, could please answer the following questions within a comment (even if the information is already given)?
In order to check the connection mode, its best to check |
|
@rhuss What we have tried so far is upgrading maven plugin and downgrading docker with no success. We are now moving to tcp waits. Docker version: 17.05.0-ce / 17.04.0-ce Thank you! |
|
@rhuss I'm not surprised, on our dev boxes we've never had an issue, it's just our CI server that always chokes. Here's it's info (ancient!): Docker version: 1.7.1, build 786b29d |
|
I'm having the exact same issue, on Mac and Linux, it seem very random maybe 1/4th of the time the build fails because of this. Using versions 0.22.1 and 0.23.0 of the maven docker plugin. Configure under, You can clearly see the third log line is the matching one. |
|
I managed to reproduce something similar to what is reported here but I'm not 100% sure it's the same scenario. To reproduce the issue the expected log line needs to be delayed for more that the configured timeout. Here the timeout is 5000ms and each generated line has a 1000ms sleep following it: The sample project is available here. @RediJedi @Vulcannis @robzet @aaronjwhiteside are you still experiencing this? Could you provide a full It would help to verify if by the time the expected line is logged the container has already been running for more then the specified timeout. |
|
I have the same issue with a mysql image. The logs with -X didn't show anything the last info I got from log checker was that pattern was OK it's stuck. |
|
Still seeing this issue every now and then, the log line is produced well before the timeout is reached, so I don't think that is the issue. It's hard to reproduce on demand and I'm hesitant about turning debug on in our CI environment. |
|
For me the issue is only reproducible on my MacOS. On Ubuntu it works OK. So I don't think this is a plugin problem. |
|
@aaronjwhiteside @danieljipa thanks for the feedback! @danieljipa can you reproduce the issue on MacOS every time? Do you have a small sample project I can use to try to find the issue? |
|
@isavin It's something like 3 out of 4 times it fails. It hangs and not even the exec:postStart command is executed. I will try to create a small sample for this. |
|
As an update. It seems to fail only with 0.24.0. With 0.23.0 it always work.
Minimal reproductible sample https://github.com/danjee/dockermavenplugin |
|
@danjee great! thank you! I'll have a look when I have time and let you know. |
|
Ty @isavin |
|
So this issue is still there, at least in 0.27.2 isn't it? |
|
0.28 also seems to still have this issue. I do use unix sockets and it fails randomly (about 3/4 times), although the log is always there. |
|
Next try: There's an update of the unix socket library in release 0.29.0, so it might be worth to check this out whether it fixes this issue. |
|
I'm still having this issue, with version 0.29.0. |
|
I am still having this issue when using 0.31.0. Interestingly, for a Spring boot project, it works fine when I use postgres:9.5.4 container with junit 4 but if I upgrade either the db to 10+ or junit to ver.5, then I get the same message as in the comment: " ghost commented on Feb 8, 2018" |
|
it turned out that the issue of timeout is triggered by a \n character in the tag. THe plugin simply is not able to create a container. \n got there because of autoformatting... |
|
@rhuss I am facing the same issue with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
The wait on logging will often fail even though the logging statement is plainly visible in the logs. This is most often observed when running in Jenkins. There doesn't appear to be any pattern as running the build manually afterwards will usually fix it, but not always. Sometimes we have to rerun it two or three times in a row until it decides to see the logging.
Info
mvn -v) : 3.3.9We've tried various permutations from simply looking for "Started" to the regex noted above. The issue is never resolved. As often as not the following failed build output can be observed:
The text was updated successfully, but these errors were encountered: