feat: enable migration testing

[szmarczak]

1. I noticed that _pauseOnMigration does not stop the autoscaled pool, keeping the event loop alive. Is this expected?
   Google Meet summary: probably, let's consult with Ondra
   Resolution: it should probably not block the event loop, but there's bunch of other things that are blocking the event loop, such as snapshotter so probably not worth the change
2. Do we want to E2E test this? Done ✅

Flaky tests encountered (should be a separate issue?):

https://github.com/apify/crawlee/actions/runs/3218900877/jobs/5263614933#step:9:70

FAIL test/core/session_pool/session_pool.test.ts (8.869 s)
  ● SessionPool - testing session pool › should initialize with default values for first time

    ENOENT: no such file or directory, stat '/home/runner/work/crawlee/crawlee/storage/datasets/default/000000001.json.lock'

https://github.com/apify/crawlee/actions/runs/3218900877/jobs/5263615026#step:9:70

FAIL test/core/crawlers/puppeteer_crawler.test.ts (40.046 s)
  ● PuppeteerCrawler › should work

    ENOENT: no such file or directory, stat '/home/runner/work/crawlee/crawlee/storage/datasets/default/000000001.json.lock'

https://github.com/apify/crawlee/actions/runs/3263113405/jobs/5361354845#step:9:82

FAIL test/core/autoscaling/snapshotter.test.ts (8.556 s)
  ● Snapshotter › correctly marks CPU overloaded using OS metrics

    expect(jest.fn()).toBeCalledTimes(expected)

    Expected number of calls: 5
    Received number of calls: 6

      168 |         expect(loopSnapshots[3].isOverloaded).toBe(false);
      169 |         expect(loopSnapshots[4].isOverloaded).toBe(true);
    > 170 |         expect(cpusMock).toBeCalledTimes(5);
          |                          ^
      171 |
      172 |         cpusMock.mockRestore();
      173 |         spy.mockRestore();

      at Object.<anonymous> (test/core/autoscaling/snapshotter.test.ts:170:26)

Closes #1531

[B4nan]

I just realised, this solution will force users to press ctrl + c twice to shut down the crawler, right? We wanted to allow testing the migrations, not force all users to go through this process. Can we use a different signal/keyboard shortcut, to keep the ctrl + c working as it was?

[szmarczak]

Unfortunately, no. Windows supports only SIGINT and SIGBREAK but POSIX supports only SIGINT and SIGTERM. I guess we could listen on both SIGBREAK and SIGTERM, but people would have to press different keyboard shortcuts. Also, SIGTERM by default doesn't have a keyboard shortcut.

[B4nan]

but people would have to press different keyboard shortcuts

I want them to use different shortcut. This shouldnt be the default, rather an opt-in. Migrations are rather apify specific thing, so is their testing.

I was thinking about something similar to watch mode in some test runners like vitest, where it listens to your input on prompt (while printing the logs), so you can press anything and map any behavior this way. I hope it could be done easily, if not, I'd at least hide this behind a config toggle and keep it disabled by default.

[metalwarrior665]

This would support #1531, right? That seems generally useful but maybe people did not yet hit a real need for it to be asking us :)

[B4nan]

Yes, so you would be fine with doing this by default? Maybe we should add the same mechanism, with a 10s exit automatically triggered, right now it will wait for the second ctrl+c so you always need to do it twice to actually shut it down. At least that's how I understood it.

[szmarczak]

After you press CTRL+C it will gracefully shut down (migration state). If you press CTRL+C again then it will force shutdown.

[szmarczak]

This shouldnt be the default, rather an opt-in. Migrations are rather apify specific thing, so is their testing.

We would have to show them how to do this (stty command), I never had the need to add another interrupt signal keyboard shortcuts.

[B4nan]

We would have to show them how to do this (stty command), I never had the need to add another interrupt signal keyboard shortcuts.

It woudn't be an interrupt signal, it would be a shortcut to trigger the pause code - it would be a different event, not sigint.

[szmarczak]

Yep, typo. All signals are interrupt signals. SIGINT means interrupt from keyboard.

[szmarczak]

Actually SIGTERM cannot be sent via keyboard :( (can be sent via kill though but that would be too complex)

Reverted back to handling just SIGINT