Python 3.6 container image: First fig.to_image call is very slow (~ 5 minutes)

[stephprobst]

I'm trying to use kaleido on a Python 3.6 container provided by Redhat.

Container repo: https://github.com/sclorg/s2i-python-container/tree/master/3.6
Image: https://catalog.redhat.com/software/containers/rhscl/python-36-rhel7/599328ef5a13466ac6ac36aa

The minimum code sample I use for testing is:

import plotly.graph_objects as go
fig = go.Figure(data=[go.Scatter(y=[1, 3, 2])])
img_bytes = fig.to_image(format="png", width=200, height=200, scale=1)

I create a new, empty container, connect to it and run the code above. The initial call to the to_image method takes ~ 5 minutes to complete, independent of the assigned resources. All subsequent calls are very fast.

It doesn't matter if the container started a minute ago or half an hour ago. The initial call always takes 5 minutes.

Is this behavior by design? Can the initial start-up procedure really take that long or am I missing something?

Thank you.

[jonmmease]

Hi @Djeramon, it is expected that the first call takes longer that subsequent calls because the Kaleido executable is launched as a subprocess on the first call. But, I haven't seen the first call take more than ~5 seconds, so 5-minutes is definitely not expected.

Thanks for letting us know!

[jonmmease]

@Djeramon Could you add instructions for pulling this image? Is it available without a RedHat subscription?

It would be easier to look into if you could reproduce it using a Centos image available from docker hub. https://hub.docker.com/_/centos.

Thanks!

[Makman2]

Since there is not much traction here, I have to second this. Something makes it start up extremely slowly.

Fedora 31, inside venv (latest plotly and kaleido), using write_image API.

When I hit ctrl+c early, I get following traceback (might be of help):

/home/mak/dev/respice/venv/bin/python /home/mak/dev/respice/examples/thesis/buckmotor.py
Traceback (most recent call last):
  File "/home/mak/dev/respice/examples/thesis/buckmotor.py", line 74, in <module>
    fig.write_image('temp.png')
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/plotly/basedatatypes.py", line 3806, in write_image
    return pio.write_image(self, *args, **kwargs)
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/plotly/io/_kaleido.py", line 252, in write_image
    engine=engine,
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/plotly/io/_kaleido.py", line 132, in to_image
    fig_dict, format=format, width=width, height=height, scale=scale
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/kaleido/scopes/plotly.py", line 104, in transform
    figure, format=format, width=width, height=height, scale=scale
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/kaleido/scopes/base.py", line 280, in _perform_transform
    self._ensure_kaleido()
  File "/home/mak/dev/respice/venv/lib64/python3.7/site-packages/kaleido/scopes/base.py", line 182, in _ensure_kaleido
    startup_response_string = self._proc.stdout.readline().decode('utf-8')
KeyboardInterrupt

Process finished with exit code 1

[jonmmease]

Hi @Makman2, does it eventually finish for you after a long time?

If someone could find a way to reproduce this in a public docker container, that would be really helpful.

[jonmmease]

Another thought for debugging:

• Run the write_image command in an ipython session
• After it hangs for a while, stop it with Control-C
• Run the following and report any output that's printed

import plotly.io as pio
scope = pio.kaleido.scope
print(scope._std_error.getvalue().decode())

Thanks!

[Makman2]

@jonmmease thanks for the quick response! Yes it does finish after a long time. After I have stumbled upon this issue I actually let it run through (also in the magnitude of 5mins).

I haven't yet a docker image ready for reproduction, but inserted your snippet into my scripts, and get this (original traceback included from ctrl+c):

INTERRUPTED AT:
Traceback (most recent call last):
  File "replotlify.py", line 167, in showpatch
    fig.write_image(args.output)
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/plotly/basedatatypes.py", line 3806, in write_image
    return pio.write_image(self, *args, **kwargs)
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/plotly/io/_kaleido.py", line 252, in write_image
    engine=engine,
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/plotly/io/_kaleido.py", line 132, in to_image
    fig_dict, format=format, width=width, height=height, scale=scale
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/kaleido/scopes/plotly.py", line 104, in transform
    figure, format=format, width=width, height=height, scale=scale
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/kaleido/scopes/base.py", line 280, in _perform_transform
    self._ensure_kaleido()
  File "/home/mak/dev/doshisha-research/master-thesis/venv/lib64/python3.7/site-packages/kaleido/scopes/base.py", line 182, in _ensure_kaleido
    startup_response_string = self._proc.stdout.readline().decode('utf-8')
KeyboardInterrupt


PLOTLY DEBUG INFO:
[0108/204014.216783:WARNING:resource_bundle.cc(435)] locale_file_path.empty() for locale 
[0108/204014.222121:WARNING:resource_bundle.cc(435)] locale_file_path.empty() for locale 
[0108/204014.222140:WARNING:resource_bundle.cc(435)] locale_file_path.empty() for locale 
[0108/204014.236544:WARNING:resource_bundle.cc(435)] locale_file_path.empty() for locale 

[jonmmease]

OK, thanks for including that debug info. No issues there, so I guess it really is just running super slow.

A reproducible docker image would be super helpful, thanks!

[Makman2]

I was trying to recreate this in docker as well as on my other PC and failed (same environment: Fedora 31, same PyCharm project with venv).
It runs faster, about as long as the normal figure.show() function does take.
I tried to recreate it on the fedora:31, python:3.9 and python:3.7 (default on Fedora 31) images with no success.

There is one major difference though: My research computer (where it takes long) lies behind a proxy server and university networking. Not sure if this might be related, but as some sort of strapped down Chromium is used in Kaleido, it may be? Just an idea.

[jonmmease]

Hi @Makman2, thanks for working on reproducing the issue. If you're using fig.to_image/fig.write_image/plotly.io.to_image/plotly.io.write_image then there shouldn't be any external network interaction going on, so I don't think a proxy/firewall setup would cause this.

Does the computer have any other security software like SELinux running on it? An SELinux issue was investigated some in #37.

[Makman2]

Yes SELinux is enabled by default on Fedora.

[Makman2]

Since I am not able to reproduce right now, I still have access to the failing machine (though only another 2-3 months), is there maybe something I could further investigate from my end? Like running certain debug statements or looking through certain logs?

[jonmmease]

Thanks for offering @Makman2, unfortunately I don't have any other ideas right now. Could you temporarily disable SELinux to see if that makes a difference?

[Makman2]

I tried it, and no difference. Using setenforce 0 to put SELinux into permissive mode.

But since the idea with the proxy did stuck in my head, out of joke I just disabled my whole internet connection and re-run a simple plotly script that saves an image. AND: It got fast immediately!
So apparently there is some outbound connection happening, and maybe due to the proxy, something starts up until a long timeout is reached after ~5 minutes.

Until now I have run all my scripts out of PyCharm, which doesn't configure any proxy variables at all.
From the standard command line, I have configured HTTP_PROXY and friends (by default without credentials, but this does not seem to matter). The scripts run well and fast in such an environment.

Conclusion: If you are behind a proxy, you must configure proxy related environment variables (with or without credentials does not matter it appears, kaleido just needs the proxy address).

I guess to reproduce this, one needs to setup a restrictive proxy that regulates outbound connections, i.e. every simple query must go through the proxy, a bypass is not allowed (so you have to make sure that if the proxy is off, you can't access the internet without it). There appear to be plenty of proxy server implementations available that can be run quickly.

[jonmmease]

Thanks for the update @Makman2. When used from plotly.py, the only thing kaleido would be checking the internet for is MathJax. Could you try disabling MathJax when running from PyCharm (without the Proxy config) and see if that makes a difference?

import plotly.io as pio
pio.kaleido.scope.mathjax = None

Thanks!

[jjnurminen]

I was suffering from the same issue under Windows 10/Anaconda. Disabling MathJax as advised above fixed it.

[jonmmease]

Thanks for the feedback @jjnurminen, that's good to know.

@Makman2, when you have a chance I'd be interested to know if disabling MathJax fixes the issue for you as well.

We might want to consider vendoring MathJax in the PyPI package, and depending on it in the conda package.

[Makman2]

Sorry it took so long. Yes, it does. Disabling MathJax makes it fast again. 😄

[jonmmease]

Version 0.2.0 of kaleido now bundles MathJax, so by default it will work in offline environments without attempting to access the internet.

Unfortunately, plotly.py currently overrides the local MathJax path with the CDN location, but this will be fixed in plotly.py version 5 (plotly/plotly.py#3094).

Until plotly.py 5.0 is released, you should still set the MathJax path to None if you are running into this issue, but after 5.0 it won't be necessary anymore.

[Makman2]

Awesome, glad we finally could resolve this, especially that we found the root of the problem :) Thank you.

[somso2e]

Setting pio.kaleido.scope.mathjax = None doesn't work for me.
Since it's apparently an issue with the internet connection, I should mention, similar to others, my internet runs through a proxy. But I even tried turning it off and it still doesn't work.

I don't know what debug output is necessary but,

import plotly
import kaleido
scope = plotly.io.kaleido.scope
print(scope._std_error.getvalue().decode())
print(plotly.__version__)
print(kaleido.__version__)

5.11.0
0.2.1

The output @jonmmease asked for, just outputs nothing.

[sudilav]

I'm trying to utilize this library and at first it seemed great. Up until it never complete the write_image. The fact that you know about this, haven't fixed it, don't reproduce in some kind of pipeline, christ. You have written some great code only to use a backend system that bites your users and developers, making ultimately, this package completely useless to me.

[buckettt]

Had this issue also using kaleido and it persisted even when setting mathjax to None. My internet also goes through a proxy.

Switching to using orca seems to have resolved my issue.

fig.to_image(format="png", engine="orca")

Plotly version 5.9.0