Fix LocalExecutor memory spike by applying gc.freeze

[wjddn279]

related: #58143

Body

As discussed (not confirmed yet), this resolves the issue of sudden memory usage spikes in worker processes when using LocalExecutor. Memory increases due to unnecessary copying of read-only shared memory through COW caused by gc. By applying gc.freeze and moving existing objects to the permanent generation, we prevent COW from occurring.

When using fork mode, we create many worker processes at once to minimize gc.freeze and unfreeze calls. When using spawn mode, we maintain the existing approach to ensure stability.

Benchmark

memory usage

Comparison of per-process memory usage in LocalExecutor before and after applying this PR. Measured in the same environment running 500 tasks per minute for 12 hours.

AS-IS	TO-BE
smem_as-is.txt	smem_to-be.txt

gc.freeze / unfreeze performance (elapsed time)

We measured the elapsed time of gc.freeze and gc.unfreeze for each scheduler loop iteration. Most operations took microseconds, confirming virtually no impact. The actual operation is a very lightweight process that simply marks objects in the current generation as permanent generation without any memory copying.https://github.com/python/cpython/pull/3705/files

[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] loc=scheduler_job_runner.py:1375
2025-11-15T05:13:36.911715Z [info     ] freeze: 18.83 μs               [airflow.jobs.scheduler_job_runner.SchedulerJobRunner] loc=scheduler_job_runner.py:1366
2025-11-15T05:13:36.912238Z [info     ] freeze_cpu: 10.08 μs           [airflow.jobs.scheduler_job_runner.SchedulerJobRunner] loc=scheduler_job_runner.py:1367
2025-11-15T05:13:36.912517Z [info     ] unfreeze: 10.92 μs             [airflow.jobs.scheduler_job_runner.SchedulerJobRunner] loc=scheduler_job_runner.py:1374
2025-11-15T05:13:36.913268Z [info     ] unfreeze_cpu: 6.87 μs  

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

LGTM. but there is the comment on the immediate ramping up of remaining workers, and definitely we need more than one pair of eyes to take a look.

[potiuk]

I also marked it as backportable to 3-1-test. That would be a fantastic bugfix for 3.1.4 Local Executor memory usage.

[jscheffl]

 __________________
< Wow! I like COWs >
 ------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

[wjddn279]

@ashb I’ve incorporated the feedback!

[potiuk]

@ashb - any comment? I would love to merge this one today in preparation for upcoming 3.1.4

[potiuk]

I merged for 3.1.4, We can always iterate in the future and the results of tests sounds very plausible . Thanks @wjddn279 and we can I think work on other memory optimisations after that.

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test

[wjddn279]

#protm self nominating
The memory usage improvement achieved by this is at a satisfactory level. #59033 (comment)