[log files readability] Disable any tqdm/tqdm_ray by default (set `enable_progress_bar: False` by default), as progress bar leads to unreadable log files without new lines and disable any coloring / text styles in Ray logging format when sinking to files

[vadimkantorov]

Even from ray.experimental.tqdm_ray import tqdm leads to hard to process logs with some ANSI control sequences (maybe text colors, bold/dim or maybe \rs as well) and no newlines when simply redirected to file

Is it possible to make tqdm or DummyProgressBar produce simply regular lines, without plenty of spaces or \rs or fancy control sequences (or any <?, ?it/s any other ANSI control sequence text color/text style stuff - even simple ${it} / ${total} every interval steps would do. We'd mostly like to work with logs sinked to files, so fancy coloring actually hurts) so that the log can be viewable in vim?

How can one make Trinity logs look normal with new lines? (might boil down to some ray logging configuration - https://docs.ray.io/en/latest/ray-observability/user-guides/configure-logging.html which looks extremely complicated)?

How do I correctly set up sinking logs to files?

Thank you!

[pan-x-c]

You can set buffer.explorer_input.taskset.enable_progress_bar to false to disable the progress bar.

[vadimkantorov]

For progress bar: should DummyProgressBar be made to just print iteration / total every interval iterations?

---

I guess maybe the question is how to set up logging to files in ray so that no ANSI control sequences get pushed into the files? (And proper newlines are added)

[pan-x-c]

Sorry, I misunderstood the question. In our normal use, the progress bar is the main reason that makes the log confusing and difficult to read. No matter how you handle it, as long as the progress bar is enabled, it will disrupt the Ray log. Therefore, , we usually use the above method to directly turn off the progress bar in this case.

[pan-x-c]

Use export RAY_COLOR_PREFIX=0 && trinity run --config your_config_file could avoid those no-ANSI control sequences for coloring.

[vadimkantorov]

Thank you! For both mode, is there a way to have ray output to separate files directly each rank's log?

E.g. separate log for every ray worklet/actor? Like so, it'd be easier to understand if any component got hung or stuck anywhere

[vadimkantorov]

as long as the progress bar is enabled, it will disrupt the Ray log

I was proposing to make DummyProgressBar extremely simple and just print() ieration counts...

Basically, need some help to set up logging and its config to get separate per-rank logging files (for various debugging)... Ray makes it extremely complex and complicated: quite a long document: https://docs.ray.io/en/latest/ray-observability/user-guides/configure-logging.html

[vadimkantorov]

No matter how you handle it, as long as the progress bar is enabled, it will disrupt the Ray log.

I suggest it should then be disabled by default... tqdm or even ray's tqdm make more suffering than niceties...

[pan-x-c]

Thanks for the feedback, we will include log readability improvements in our development plan.

Currently, there is no direct way to separate log for each ray actor within Trinity-RFT. When we really need to view the log of an Actor, we usually use the Ray dashboard, but the problem at this time becomes how to find the desired Actor in the Ray dashboard.

[vadimkantorov]

there is no direct way to separate log for each ray actor within Trinity-RFT

I think, this is critical to implement for any debugging.

Also, ray's log dedup and trying to cram everything into a single log / stdout only makes the problems concealed and debugging complicated

Figuring out how to do basic logging to separate files in Ray is hard :( hoping Trinity to provide a walk-through or native support for this

---

A problem with ray dashboard is that it's gone if machine crashes or if a cloud spot machine gets pre-empted. So IMO the good old per-rank / per-process log files are needed (which can simply be stored directly to a networked disk)

[pan-x-c]

Thanks for the suggestion. We will change the default behavior of the progress bar in the next PR. Log color switching and ray-actor level logging will be added in the coming weeks

[vadimkantorov]

I also found log_to_driver=False in the Ray's docs. This might help to disable collecting all logs to one file

[vadimkantorov]

Another useful thing would be somehow adding GPU rank / process to the prefixes in addition to PID. Rank's range are stable across runs as opposed to PIDs

[pan-x-c]

Thanks for the suggestion. Currently, most logs in Trinity-RFT are printed by the logger object created inside the actor (e.g., Explorer, Trainer, WorkflowRunner). Our current plan is to extend the logger to print the logs to the checkpoint_job_dir/logs/actor_name file.

[vadimkantorov]

It should also be good for debugging the various distributed/RPC-related delays/hangs in ranks to also be able to have the most-stupid per-rank logfile. Especially if there are many nodes and ranks per node.