Default crawlee[cli] template project broken on Windows

[solenum]

This could just be a misunderstanding on my part, first time using crawlee. I followed the guide on the homepage and used pipx to create a project

pipx run 'crawlee[cli]' create my-crawler

This produces a project that fails to run after I activate the venv in powershell and attempt to run the project

cd my-crawler
.\.venv\Scripts\activate
python -m my-crawler

[asyncio] ERROR Task exception was never retrieved
future: <Task finished name='Task-12' coro=<Connection.run() done, defined at C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py:272> exception=NotImplementedError()>
      Traceback (most recent call last):
        File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py", line 279, in run
          await self._transport.connect()
        File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 133, in connect
          raise exc
        File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 120, in connect
          self._proc = await asyncio.create_subprocess_exec(
        File "C:\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
          transport, protocol = await loop.subprocess_exec(
        File "C:\Python310\lib\asyncio\base_events.py", line 1667, in subprocess_exec
          transport = await self._make_subprocess_transport(
        File "C:\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
          raise NotImplementedError
      NotImplementedError

Annoyingly this took me ages to figure out. It seems to be related to this line in the _main_.py that gets generated, as commenting it out resolves the problem:

    if platform.system() == 'Windows':
        # This mitigates a warning raised by curl-cffi. If you do not need to use curl-impersonate, you may remove this.
        asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
        # __comment out ^ to fix__

Is this just an issue with my configuration? I've updated pip and all packages to the latest versions, I guess my problems resolved with the above but I couldn't find a relevant issue anywhere and wasted a lot of my workday trying to figure this out so I figured I'd create this issue to hopefully prevent others from getting stuck on the same problem. (I'm not a Windows person - entirely possible this is something I'm just not understanding)

[solenum]

Full powershell terminal output

(.venv) PS C:\Users\redacted\Documents\Git\Work\redacted\redacted> python -m redacted
[crawlee.crawlers._playwright._playwright_crawler] INFO  Current request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished             │ 0        │
│ requests_failed               │ 0        │
│ retry_histogram               │ [0]      │
│ request_avg_failed_duration   │ None     │
│ request_avg_finished_duration │ None     │
│ requests_finished_per_minute  │ 0        │
│ requests_failed_per_minute    │ 0        │
│ request_total_duration        │ 0.0      │
│ requests_total                │ 0        │
│ crawler_runtime               │ 0.017117 │
└───────────────────────────────┴──────────┘
[asyncio] ERROR Task exception was never retrieved
future: <Task finished name='Task-12' coro=<Connection.run() done, defined at C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py:272> exception=NotImplementedError()>
      Traceback (most recent call last):
        File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py", line 279, in run
          await self._transport.connect()
        File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 133, in connect
          raise exc
        File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 120, in connect
          self._proc = await asyncio.create_subprocess_exec(
        File "C:\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
          transport, protocol = await loop.subprocess_exec(
        File "C:\Python310\lib\asyncio\base_events.py", line 1667, in subprocess_exec
          transport = await self._make_subprocess_transport(
        File "C:\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
          raise NotImplementedError
      NotImplementedError
Traceback (most recent call last):
  File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\redacted\__main__.py", line 12, in <module>
    asyncio.run(main())
  File "C:\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Python310\lib\asyncio\base_events.py", line 646, in run_until_complete
    return future.result()
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\redacted\main.py", line 22, in main
    await crawler.run(
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\crawlee\crawlers\_basic\_basic_crawler.py", line 606, in run
    await run_task
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\crawlee\crawlers\_basic\_basic_crawler.py", line 657, in _run_crawler
    await exit_stack.enter_async_context(context)  # type: ignore[arg-type]
  File "C:\Python310\lib\contextlib.py", line 619, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\crawlee\browsers\_browser_pool.py", line 202, in __aenter__
    await asyncio.wait_for(plugin.__aenter__(), timeout)
  File "C:\Python310\lib\asyncio\tasks.py", line 445, in wait_for
    return fut.result()
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\crawlee\browsers\_playwright_browser_plugin.py", line 141, in __aenter__
    self._playwright = await self._playwright_context_manager.__aenter__()
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\async_api\_context_manager.py", line 46, in __aenter__
    playwright = AsyncPlaywright(next(iter(done)).result())
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_connection.py", line 279, in run
    await self._transport.connect()
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 133, in connect
    raise exc
  File "C:\Users\redacted\Documents\Git\Work\redacted\redacted\.venv\lib\site-packages\playwright\_impl\_transport.py", line 120, in connect
    self._proc = await asyncio.create_subprocess_exec(
  File "C:\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "C:\Python310\lib\asyncio\base_events.py", line 1667, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "C:\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
    raise NotImplementedError
NotImplementedError

[janbuchar]

Hi @solenum, I'm not a Windows person, either 🙂 Can you share more details about your Windows installation (version, architecture)? Also, you're running this on windows directly, not using WSL or anything like that, right?

[solenum]

Yeah directly on windows (via a powershell terminal specifically), not using WSL or anything like that here. Windows 11 Pro 23H2, OS build 22631.5335

Python 3.10.5
pip 20.0.4
pipx 1.7.1

[Mantisus]

Playwright does not support working with WindowsSelectorEventLoopPolicy - https://playwright.dev/python/docs/library#incompatible-with-selectoreventloop-of-asyncio-on-windows

Thanks for catching that.

[janbuchar]

Thanks @Mantisus, makes sense. What do we make of this? Do we disable the code that sets WindowsSelectorEventLoopPolicy when the playwright crawler is selected?

[Mantisus]

Yeah, that's the best option for us.

Judging by the comment, it's set for curl-cffi. Perhaps you remember the problem with curl-cffi when using ProactorEventLoop?

[janbuchar]

Yeah, that's the best option for us.

Judging by the comment, it's set for curl-cffi. Perhaps you remember the problem with curl-cffi when using ProactorEventLoop?

Yeah, it printed out some random warning 🤷 Maybe it's not a problem anymore. But I wouldn't give it too much attention. The Playwright+curl-cffi combo should be pretty rare now that we have a playwright http client.