Cleanup of PlaywrightCrawler keeps zombie processes

[ROYOSTI]

I use PlaywrightCrawler with headless=True.
The package that I use is: crawlee[playwright]==0.6.3

In my code I have my own batching system in place.
But I noticed that my memory slowly starts to increase on each batch.
After some investigation I saw that ps -fC headless_shell gave me a lot headless_shell with CMD: <defunct> (zombie processes).
I had hoped that this issue was related to this fix: Remove tmp folder for PlaywrightCrawler in non-headless mode and it automatically would fix the problem. But unfortunately the issue still exists.
When I look up the parent PID I just see the entry point of my code: python al_crawler/crawler.py run_mongo_crawler --crawler_name=blabla

Below you can see my code for the batching system:

    # Create key values stores for batches
    scheduled_batches = await prepare_requests_from_mongo(crawler_name)
    processed_batches = await KeyValueStore.open(
        name=f'{crawler_name}-processed_batches'
    )

    # Create crawler
    crawler = await create_playwright_crawler(crawler_name)

    # Iterate over the batches
    async for key_info in scheduled_batches.iterate_keys():
        urls: List[str] = await scheduled_batches.get_value(key_info.key)
        requests = [
            Request.from_url(
                url,
                user_data={
                    'page_tags': [PageTag.HOME.value],
                    'chosen_page_tag': PageTag.HOME.value,
                    'label': PageTag.HOME.value,
                },
            )
            for url in urls
        ]
        LOGGER.info(f'Processing batch {key_info.key}')
        await crawler.run(requests)
        await scheduled_batches.set_value(key_info.key, None)
        await processed_batches.set_value(key_info.key, urls)

I assume a possible fix would be to create on each batch a new crawler? But I would like to reuse the same crawler.

[janbuchar]

Hello @ROYOSTI and thank you for your interest in Crawlee!

I use PlaywrightCrawler with headless=True. The package that I use is: crawlee[playwright]==0.6.3

What operating system do you use?

After some investigation I saw that ps -fC headless_shell gave me a lot headless_shell with CMD: <defunct> (zombie processes).

Does the space usage of those zombies add up? Does the number of zombies grow in time?

I assume a possible fix would be to create on each batch a new crawler? But I would like to reuse the same crawler.

If you could test that and report your findings, that could be helpful.

[Pijukatel]

Based on the description it looks similar to this: microsoft/playwright#34521

[ROYOSTI]

Sorry for my late answer @janbuchar! I had some days off and didn't look at Github.

What operating system do you use?

I run my code on an AWS EC2 instance (Amazon Linux AMI 2023.0.20250224 x86_64 ECS HVM EBS).

Does the space usage of those zombies add up? Does the number of zombies grow in time?

When starting the crawl I have around 200-250 headless_shell running.
After each batch is complete there are between 25-50 headless_shell zombies that stay.
So after each batch I see that pgrep -c headless_shell is growing cumulative with those zombies.
When printing top | grep 'MiB Mem' I see the free memory going down over time.

I tried running my code with creating a new crawler for each batch, but the result stays the same

  # Create key values stores for batches
    scheduled_batches = await prepare_requests_from_mongo(crawler_name, query_file)
    processed_batches = await KeyValueStore.open(
        name=f'{crawler_name}-processed_batches'
    )

    # Iterate over the batches
    async for key_info in scheduled_batches.iterate_keys():
        urls: List[str] = await scheduled_batches.get_value(key_info.key)
        requests = [
            Request.from_url(
                url,
                user_data={
                    'page_tags': [PageTag.HOME.value],
                    'chosen_page_tag': PageTag.HOME.value,
                    'label': PageTag.HOME.value,
                },
            )
            for url in urls
        ]
        LOGGER.info(f'Processing batch {key_info.key}')

        # Create crawler
        crawler = await create_playwright_crawler(crawler_name)
        await crawler.run(requests)

        await scheduled_batches.set_value(key_info.key, None)
        await processed_batches.set_value(key_info.key, urls)

The Playwright issue that @Pijukatel shared seems very similar and could be related.
Because in my Docker file I start from FROM mcr.microsoft.com/playwright/python:v1.50.0-jammy

[Pijukatel]

I could not reproduce the issue when creating new actor from template. When running on Apify platform I see that the amount of processes is not increasing (I tried couple of minutes). Could you double check that you have some sort of init running in your container. For example in my log you can see docker-init process running. And in that issue that I mentioned earlier there was dumb-init. So you should have something like that running also in your container

[ROYOSTI]

When I logged the processes I did not have a docker-init.
I ran my code with the following:

docker run --rm -t $docker_args \
    -v /mnt/storage:/app/storage \
    --user appuser \
    --security-opt seccomp=/var/lib/jenkins/helpers/docker/seccomp_profile.json \
    -e MONGO_HOST=${MONGO_HOST} \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e CRAWLEE_MEMORY_MBYTES=${CRAWLEE_MEMORY_MBYTES} \
    ${IMAGE_NAME} $prog_args

When adding --init everything works as expected! Thanks for pointing that out.
Why is it important to have that parameter?

[Pijukatel]

Why is it important to have that parameter?

From playwright documentation:
https://playwright.dev/docs/ci#docker

Using --init Docker flag or dumb-init is recommended to avoid special treatment for processes with PID=1. This is a common reason for zombie processes.

More detailed summary from:
https://github.com/Yelp/dumb-init?tab=readme-ov-file#why-you-need-an-init-system

Orphaned zombie processes aren't properly reaped.

A process becomes a zombie when it exits, and remains a zombie until its parent calls some variation of the wait() system call on it. It remains in the process table as a "defunct" process. Typically, a parent process will call wait() immediately and avoid long-living zombies.

If a parent exits before its child, the child is "orphaned", and is re-parented under PID 1. The init system is thus responsible for wait()-ing on orphaned zombie processes.

Of course, most processes won't wait() on random processes that happen to become attached to them, so containers often end with dozens of zombies rooted at PID 1