[Feature]: High concurrency memory usage

[goloveychuk]

🚀 Feature Request

When running playwright with high concurrency (100-200 workers) with remote browser pool, client begin to be a bottleneck. Main one is memory usage. With one worker 200mb (optimistically), total memory usage is 20GB for 100 workers.
I see two ways to solve this.

1. N*M concurrency model.
   Idea is that we spawn lower number of workers (let's say 10), and make each run 10 parallel "test workers", each creating one ws connection, etc.
   The assumption is that cpu usage not proportional to memory usage, and probably most of the work workers do is communication with playwright server (remote browser pool). I.e. await page.xxx().
   There could be some heavy work like gzip compression/decompression of tracing reports, image comparison, etc. Some of that could be offloaded to worker_threads.
   But generally, with client/server playwright arch, most of the heavy work should be done on server side, so I think NxM idea is fitted into this.

2. Distributed model
   Main playwright process now become to be orchestrator.
   Workers are created on server side, playwright sends all js (bundle?) and snapshots needed to execute test to worker.
   There are many issues with it. How to collect all js needed? Dynamic imports? fs.readFileSync? Different nodejs version? Different remote machine arch, etc.
   Variation of this: add to client and server third actor - worker server.

Wdyt?

Example

No response

Motivation

Make playwright with remote browsers work on high concurrency without sharding.

[dgozman]

@goloveychuk Thank you for the issue! Unfortunately, I don't think this will be worked on in the near future, unless there are some low hanging fruits in optimizing client-side performance. It would be very helpful to figure out where the memory is going - perhaps you could use memory profiling tools and see what's going on?

[goloveychuk]

I'll profile memory