PDF Generation exceeded the timeout of 60 seconds

[ThemePoint]

Hello,

i am running Snappy in an Rabbit-MQ Consumer (PHP 7.2.19)(Docker-Container) to generate PDF-Tickets.
Snappy is here included into an symfony 4.2 framework but the symfony-bundle is do not used.
The server and containers are running on Ubuntu 18.04.
Since our systems process thousands of Consumer-Messages a day we have the following problem:

The snappy runs many hours without problems. But sometimes (at irregular intervals) snappy throw an ProcessTimedOutException with the message
Failed to generate invoice pdf: The process \"\/www\/management\/vendor\/h4cc\/wkhtmltopdf-amd64\/bin\/wkhtmltopdf-amd64 --lowquality --image-dpi 150 --image-quality 70 --margin-bottom '75mm' --margin-top '50mm' --enable-external-links --enable-internal-links --load-error-handling 'ignore' --load-media-error-handling 'ignore' --enable-local-file-access --disable-plugins --footer-html '\/tmp\/knp_snappy5eff19d0233bc0.91517889.html' --header-html '\/tmp\/knp_snappy5eff19d0233f01.67188931.html' --header-spacing '10' '\/tmp\/knp_snappy5eff19d0233375.94981478.html' '\/pdfs/tickets/T-754024-1.pdf'\" exceeded the timeout of 60 seconds.
When this error occurs the system can not create more pdf files until we restart the docker container. After this point snappy every time will throw the same exception.
When we restart the container the system runs as like before.
Set the timeout to 200 also do not help.

Does somebody have any idea about this behavior?
We can reproduce this behaviour in several systems on different servers.

[alexpozzi]

Hello,
It's difficult to help on this issue as it can be caused by several different reasons.
Have you tried having a look at #292 and #169 issues?

I'm afraid it's not related to Snappy itself but more to symfony/process, wkhtmltopdf binary or PHP/OS configuration.
Do you see wkhtmltopdf processes popping in your container when the issue occurs?

[ThemePoint]

I added a logger to the snappy process. The last crash shown us the following error:

The exit status code '134' says something went wrong:\nstderr: \"Loading pages (1\/6)\n[> ] 0%\r[======> ] 10%\r[==============================> ] 50%\r[============================================================] 100%\rCounting pages (2\/6) \n[============================================================] Object 1 of 1\rResolving links (4\/6) \n[============================================================] Object 1 of 1\rLoading headers and footers (5\/6) \n[===> ] 5%\r[======> ] 10%\r[======> ] 10%\r[==================> ] 30%\r*** buffer overflow detected ***: \/usr\/local\/bin\/wkhtmltopdf terminated\nAborted (core dumped)

Interesting is that the previous PDF which was generated includes more data than the one when the error occurred. Think the problem is not the HTML.
Can this problem related to the operating system?

[fbourigault]

@ThemePoint is your consumer running through the symfony binary?

I experience the same issues an running without symfony binary seems to solve the issue.

[alexpozzi]

Can this problem related to the operating system?

Difficult to say without being able to debug it directly, but I'm pretty sure it's not Snappy related.
I'm closing the issue as, I'm afraid, there is nothing I can do here.

[ThemePoint]

After a long time we found a solution for this problem.
It looks like the symfony messanger will create a cache on doctrine interactions. After a big count of interactions this cache will be so big that it makes a problem with snappy/wkhtmltopdf process.
In this case there must be set a limit of messages after which the messanger will restart the consumers.

[alexpozzi]

Thanks to let us know!

[low-res]

I just found out, that you can set the timeout in snappy.
$snappy->setTimeout(300);
If it's not set, the default timeout of Symfony\Component\Process is used which is 60 seconds

[itwondersteam]

After a long time we found a solution for this problem. It looks like the symfony messanger will create a cache on doctrine interactions. After a big count of interactions this cache will be so big that it makes a problem with snappy/wkhtmltopdf process. In this case there must be set a limit of messages after which the messanger will restart the consumers.

is there anyway to fix/work around this issue? currently experience the same. @alexpozzi

[sibihaseb]

same issue

[alexpozzi]

@itwondersteam It's very unlikely to be Snappy related so I'm afraid I can't do much for that.
I'm willing to look into that if you'll provide me code that reproduces the issue.

[zhuman90]

same issue

[lecneri]

Hello I was facing similar issue in my dev env, I use php embedded server to execute, and it's single threaded. When wkhtmltopdf tries to create pdf it makes a second request to the php server, requiring images, css, etc. but this request hangs because the 1st thread is locked waiting wkhtmltopdf finish the pdf creation, generating thus a deadlock.
To solve this issue I created this env var
export PHP_CLI_SERVER_WORKERS=2
then run the server again. Hope this helps you.
If that's the case, you may update the docs with this solution

[DocMok]

Hello I was facing similar issue in my dev env, I use php embedded server to execute, and it's single threaded. When wkhtmltopdf tries to create pdf it makes a second request to the php server, requiring images, css, etc. but this request hangs because the 1st thread is locked waiting wkhtmltopdf finish the pdf creation, generating thus a deadlock. To solve this issue I created this env var export PHP_CLI_SERVER_WORKERS=2 then run the server again. Hope this helps you. If that's the case, you may update the docs with this solution

thanks a lot, you save my time