FEAT-#7368: Add a new environment variable for using dynamic partitioning

[Retribution98]

The use of Dynamic-partitioning depends on various factors such as data size, number of CPUs, operations performed,
and it is up to the user to determine whether Dynamic-partitioning will give a boost in his case or not.

Performance results for abs:

32 CPUS

112 CPUS

• first commit message and PR title follow format outlined here

  NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

• passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
• [x passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
• signed commit with git commit -s
• Resolves Add a new environment variable for using dynamic partitioning #7368
• tests added and passing
• module layout described at docs/development/architecture.rst is up-to-date

[anmyachev]

If we plan to look for a heuristic that can switch implementations automatically, then we could add a few words about this (and a link to the issue).

[anmyachev]

It would be good to keep the previous default behavior.

Suggested change

	if not DynamicPartitioning.get():
	if np.prod(partitions.shape) <= 1.5 * CpuCount.get() and not DynamicPartitioning.get():

[Retribution98]

Do you think we should do that? Shouldn't the user be given more freedom to decide when to activate this option?
The user can activate this locally, only for the required operations.

[anmyachev]

The user now has the ability to force the use of another code branch at his own choice, this is already more flexibility than before. And since this condition worked quite well before and considering that slowdowns are possible when using this new variable, I would replace the default behavior more carefully.

[anmyachev]

Confused with similar code in rebalance_partitions function. Ok, leave it as is.