[Data] - Preserve existing handlers for logging - don't use dictConfig

[goutamvenkat-anyscale]

Why are these changes needed?

So the way the logger works is log() -> buffer -> flush() -> target (file/console/remote_uri)
On shutdown (triggered by logging.config.dictConfig), the MemoryHandler is closed which sets the output target to None
When the target is set to None, the logs in the buffer have nowhere to be flushed, so they're retained in memory, hence it accumulates to no end.

Solution:
Temporarily detach the handlers for loggers (preserve attributes like level, target, filter and formatters) and restore them after calling dictConfig()

Based off of https://github.com/ray-project/ray/pull/48958/files#diff-d5e32f872cf7b123f0899714728ac7fc0916a5609844a7d1bf7317d98094778c (Ray Core - Ray Data log fix)

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run pre-commit jobs to lint the changes in this PR. (pre-commit setup)
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gemini-code-assist]

Code Review

This pull request effectively addresses the issue of configuring logging at module import time by deferring it until a ray.data operation is active. The use of a thread-safe, double-checked locking pattern in configure_logging ensures that initialization is performed lazily and idempotently, which is a robust approach. The changes correctly remove the eager configuration from ray/data/__init__.py and introduce the lazy call in register_dataset_logger. I have one suggestion to further improve the implementation by moving environment variable access out of the critical section to minimize lock contention.

[goutamvenkat-anyscale]

/gemini review

[gemini-code-assist]

Code Review

This pull request addresses a critical issue where log handlers were being closed during reconfiguration, potentially causing log messages to be lost. The solution of saving and restoring handler states is a good approach. My main feedback is that the current implementation overlooks the root logger, which could still be affected by the original issue. I've added a specific comment with a suggestion to include the root logger in the preservation logic.

[omatthew98]

Some of the code is a little unintuitive so maybe try to capture some more info in comments, but overall lgtm.

[alexeykudinkin]

Please add a comment to explain how there could be a logger w/ no handlers