Problems using pyppeteer and Chromium

[dafner]

Hi,

I am testing this library and found that after a few minutes of use (the amount varies and is not always fixed) the application stops receiving new messages or sending messages.
When this happens, I stop the application with Ctrl-C and the following messages appear on the screen indicating a problem with the network connection:

Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.',)>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved

Currently I am testing this library with pyppeteer and Chromium as a backend in order to test the asynchronous sending and receiving of messages.

I understand that this problem is not specific to this library and in the documentation mentions the application of a patch miyakogi/pyppeteer#160 that should be applied to pyppeteer.

In the readme also mentioned that a patched version of Chromium must be downloaded from the following link: https://github.com/alfred82santa/whalesong/blob/master/using_chromium before use the backend. However, the link is not available.

I will thank anyone who can clarify how to apply the patch and where I can download the corrected version of Chromium, since the instrucctions indicated in miyakogi/pyppeteer#160 are not very clear or give me the necessary instructions to be able to use this library with pyppeteer and Chromiun

Thanks very much

[alfred82santa]

using_chronium link must point to https://github.com/alfred82santa/whalesong/#using-chronium sorry...

On the other hand, your problem looks like you are using wrong websocket library version... it must be 6.0.0. Newer versions perform a ping/pong routine. But Chronium does not implement it on DevTool websocket. So, library closes socket.

[dafner]

Thanks for your prompt answer.

Now I testing my test application with firefox because the chromium backend are not very stable.

I don't know if anybody had the similar problem using chromium. I'm testing my test program using 2 cell phones and trying to send commands at the same time in order to verify the perfomance. While chromium works, the perfomance is much better than using Firefox.

I would like it, if possible; that someone share with me their experience with Chromiun working with some application.

By the way, I folowed all the step in order to install pyppeteer and whalesong for chromiun backend
Also, I installed pyppdf (https://github.com/kiwi0fruit/pyppdf/tree/master/pyppdf) package that download a chrome patched version
.local/share/pyppeteer/local-chromium/609904 and I linked to .local/share/pyppeteer/local-chromium/588429 which is the version that is installed using the command pyppeteer-install

In the following link (miyakogi/pyppeteer#160) suggest to use the following code:

def patch_pyppeteer():
import pyppeteer.connection
original_method = pyppeteer.connection.websockets.client.connect

def new_method(*args, **kwargs):
    kwargs['ping_interval'] = None
    kwargs['ping_timeout'] = None
    return original_method(*args, **kwargs)

pyppeteer.connection.websockets.client.connect = new_method

patch_pyppeteer()

And I modified the /usr/local/lib/python3.6/dist-packages/pyppeteer/connection.py in order to add those two params:

    self._ws = websockets.client.connect(
        self._url, max_size=None,
        ping_interval=None, ping_timeout=None,
        loop=self._loop)
    self._recv_fut = self._loop.create_task(self._recv_loop())
    self._closeCallback: Optional[Callable[[], None]] = None

Pero estos parámetros se reconocen en la última versión de websockets pero no en la versión 6:

File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/init.py", line 95, in start
await self._driver.start_driver()
File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver.py", line 49, in start_driver
await self._fut_start
File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver.py", line 53, in start_driver
await self._fut_start
File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver_chromium.py", line 72, in _internal_start_driver
**self.options)
File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 330, in launch
return await Launcher(options, **kwargs).launch()
File "/usr/local/lib/python3.6/dist-packages/pyppeteer/launcher.py", line 179, in launch
connectionDelay,
File "/usr/local/lib/python3.6/dist-packages/pyppeteer/connection.py", line 46, in init
loop=self._loop)
File "/usr/local/lib/python3.6/dist-packages/websockets/client.py", line 385, in init
factory, host, port, **kwds)
TypeError: create_connection() got an unexpected keyword argument 'ping_interval'

After all of these test; my test program based on minibot-chromium.py are not stable and stop to received and send messages after a random time. It's necesary to press the Ctrl-C keys to abort the application and them the following messages is displayed in the screen:

future: <Task finished coro=<WhalesongDriver._execute_command() done, defined at /home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver_chromium.py:107> exception=NetworkError('Protocol error Runtime.evaluate: Target closed.',)>
Traceback (most recent call last):
File "/home/administrador/workspace/whatsappx/webot/apptest/whalesong/driver_chromium.py", line 109, in _execute_command
f'(function() {{window.manager.executeCommand("{result_id}", "{command}", {dumps(params)})}})()'
File "/usr/local/lib/python3.6/dist-packages/pyppeteer/page.py", line 1186, in evaluate
return await frame.evaluate(pageFunction, *args, force_expr=force_expr)
File "/usr/local/lib/python3.6/dist-packages/pyppeteer/frame_manager.py", line 309, in evaluate
pageFunction, *args, force_expr=force_expr)
File "/usr/local/lib/python3.6/dist-packages/pyppeteer/execution_context.py", line 54, in evaluate
pageFunction, *args, force_expr=force_expr)
File "/usr/local/lib/python3.6/dist-packages/pyppeteer/execution_context.py", line 88, in evaluateHandle
_rewriteError(e)
File "/usr/local/lib/python3.6/dist-packages/pyppeteer/execution_context.py", line 238, in _rewriteError
raise error
File "/usr/local/lib/python3.6/dist-packages/pyppeteer/execution_context.py", line 85, in evaluateHandle
'userGesture': True,
pyppeteer.errors.NetworkError: Protocol error Runtime.evaluate: Target closed.

Partial list of the my installed packages

drwxr-sr-x 2 root staff 4096 Jul 9 10:45 nameparser-1.0.4.dist-info/
drwxr-sr-x 5 root staff 4096 Jul 8 19:05 openapi_client/
drwxr-sr-x 2 root staff 4096 Jul 8 19:05 openapi_client-1.0.0.dist-info/
drwxr-sr-x 21 root staff 4096 Apr 9 18:02 openpyxl/
drwxr-sr-x 2 root staff 4096 Apr 9 18:02 openpyxl-2.6.2-py3.6.egg-info/
drwxr-sr-x 5 root staff 4096 Jun 11 21:11 pip/
drwxr-sr-x 2 root staff 4096 Jun 11 21:11 pip-19.1.1.dist-info/
drwxr-sr-x 2 root staff 4096 Jun 11 18:18 pycache/
drwxr-sr-x 3 root staff 4096 Feb 28 14:36 pycard/
drwxr-sr-x 3 root staff 4096 Jul 8 17:09 pycares/
drwxr-sr-x 2 root staff 4096 Jul 8 17:09 pycares-3.0.0.dist-info/
drwxr-sr-x 3 root staff 4096 Jul 5 16:35 pydub/
drwxr-sr-x 2 root staff 4096 Jul 5 16:35 pydub-0.23.1.dist-info/
drwxr-sr-x 3 root staff 4096 Jul 16 14:23 pyee/
drwxr-sr-x 2 root staff 4096 Jul 16 14:23 pyee-5.0.0.dist-info/
drwxr-sr-x 3 root staff 4096 Jul 9 13:05 pypeln/
drwxr-sr-x 2 root staff 4096 Jul 9 13:05 pypeln-0.1.9.dist-info/
drwxr-sr-x 4 root staff 4096 Jul 17 00:08 pyppdf/
drwxr-sr-x 2 root staff 4096 Jul 17 00:08 pyppdf-0.0.10.dist-info/
drwxr-sr-x 3 root staff 4096 Jul 18 10:45 pyppeteer/
drwxr-sr-x 2 root staff 4096 Jul 17 12:16 pyppeteer-0.0.25.dist-info/
drwxr-sr-x 3 root staff 4096 Sep 6 2018 pytesseract/
drwxr-sr-x 2 root staff 4096 Sep 6 2018 pytesseract-0.2.4-py3.6.egg-info/
drwxr-sr-x 3 root staff 4096 Jul 5 18:51 redis/
drwxr-sr-x 2 root staff 4096 Jul 5 18:51 redis-3.2.1.dist-info/
drwxr-sr-x 3 root staff 4096 Jul 5 18:51 redis_collections/
drwxr-sr-x 2 root staff 4096 Jul 5 18:51 redis_collections-0.7.0.dist-info/
-rw-r--r-- 1 root staff 55 May 31 2018 setuptools.pth
drwxr-sr-x 3 root staff 4096 Jul 8 19:05 test/
drwxr-sr-x 5 root staff 4096 Apr 16 16:18 uvloop/
drwxr-sr-x 2 root staff 4096 Apr 16 16:18 uvloop-0.12.2.dist-info/
drwxr-sr-x 2 root staff 4096 Jun 11 18:18 virtualenv-16.6.0.dist-info/
-rw-r--r-- 1 root staff 104065 Jun 11 18:18 virtualenv.py
drwxr-sr-x 3 root staff 4096 Jun 11 18:18 virtualenv_support/
drwxr-sr-x 3 root staff 4096 Jul 1 18:32 vobject/
drwxr-sr-x 2 root staff 4096 Jul 1 18:32 vobject-0.9.6.1.dist-info/
drwxr-sr-x 6 root staff 4096 Jul 18 10:11 websockets/
drwxr-sr-x 2 root staff 4096 Jul 18 10:11 websockets-6.0.dist-info/

[alfred82santa]

You don't need to set that params if you use websocket 6.0