Log some sort of heartbeats to wandb (both from trainer and explorer): both at startup and in loops

[vadimkantorov]

We've experiences a problem with wandb logging while running a Trinity run:

• [Bug]: Only system stats synced, no app metrics, but the run seems online wandb/wandb#10404

Seemingly we just ran the same Trinity config on a new machine set, but this for some reason led to no Trinity metrics being uploaded to Wandb: we waited for 20 hours, according to Ray Dashboard, Trinity was indeed having done some initial validation, and then some training steps - and nothing got uploaded to wandb except system metrics

I'm proposing to start logging some sort of heartbeat to wandb (and maybe allowing in config to force commit=True), so that there are some custom bits logged to Wandb almost immediately.

And if nothing gets logged/uploaded, it can immediately be assumed/confirmed problem with Wandb. Currently, if eval_on_startup: true, anything would get logged only in _finish_explore_step(...) and _finish_eval_step(...) which can take a while

Also, wandb has had a bug where step= was forcing it to ignore commit=True (not sure if this bug is still present):

• [feature request] Function to trigger a sync manually without calling wandb.finish (also possibly a way to perform a sync synchronously / wait for completion of sync) wandb/wandb#3282 (comment)

[vadimkantorov]

Currently we got some metrics uploaded to wandb from trainer instance - but nothing from explorer instance, after an hour of running (eval_on_startup: false, small batch size)

It would be good to have some heartbeats to wandb from both trainer and explorer instances: to indicate absence of hangs, or maybe even display some rough sense of current state in wandb via these heartbeats

---

My current problem is that Explorer appears to be hung - and nothing logged to wandb. Maybe Explorer completed what it should be doing, but Trainer waits for more results, or something like this. More introspection is needed for understanding what's doing/waiting every component...

[hiyuchang]

It seems the explorer cannot finish the exploring task; thus I suggest checking the Workflow to see if there is any problem.

Also, you may use the feature in pull#217 to see separate logs for explorer/trainer/..., which helps in finding the problems.

[vadimkantorov]

to see separate logs for explorer/trainer/..., which helps in finding the problems.

Yep, I'm in the process upgrading to the new trinity to check the new logs

[vadimkantorov]

It seems the explorer cannot finish the exploring task;

In this particular case - maybe, but I think it'll also be nice to log some heartbeats to wandb (including at startup) as well - in addition to logs

Currently we observed that both trainer and explorer have 100% GPU util, but no metrics uploaded to wandb ever (except system metrics). I'm worried that wandb simply tries to buffer more metrics before upload? Should commit=True be used?

[hiyuchang]

You may try using monitor_type: tensorboard or monitor_type: mlflow to see whether logs are correct.

For wandb, commit=True may cause issues if the current step is smaller than the previous uploaded log step.

[vadimkantorov]

For wandb, commit=True may cause issues if the current step is smaller than the previous uploaded log step.

Oh wow :(

In any case, might be good to send some initial heartbeats (e.g. wandb.log({'hearbeat' : iteration_num}, step=iteration_num, commit=True)) to wandb as well in both trainer and explorer... Otherwise it's simply not clear - whether we need to wait for longer for metrics to be uploaded? Or if the system is stuck? Or if wandb is simply not uploading any custom metrics?