TensorBoard very slow on GCS

[teamdandelion]

From @bw4sz on #80:

Loading (O(200MB)) of logs via TensorBoard on GCS takes more than ten minutes, but if they copy the event files locally, it loads in seconds.

I am guessing we are running into an issue where on GCS we are loading the events without readahead chunks, so we are doing round trip requests incredibly frequently, maybe as often as for every individual event. This makes for horrible performance.

[potiuk]

Related to that (and more important) - we have also found that using tensorboard to read directly from GCS using default reload intervals incurs incredible costs.

There is a cost related wth count of API calls made to access GCS and it seems that every reload directly from GCP causes every single file to be accessed separately via the GCS api . when you have bigger (but not outrageous) number of log files (few thousands is enough) It can very quickly get out of control - and with the default reload frequency of tensorboard it might get your GCS costs unreasonably high.

In our case it was about 4 USD/day while having just around 3000 files in the logs !!!). We got about 4 millions of API requests generated in 4 days (!) just by running one tensorboard instance. We are changing strategy now to sync the GCS data locally (using gsutil rsync) and read it from there because of that.

[potiuk]

Just a comment - gsutil rsync has another problem (it deletes and recreates event files) - I opened a new issue for it: #349

[MtDersvan]

Are there any updates on this issue?
After switching to GCS for storing checkpoints and events, it is now nearly unusable to refresh logs.
When launching tensorboard, it will fetch logs from a selected directory, but it won't update it or fetch anything after. In our case even leaving tensorboard for hours won't refresh stats unless fully restarted.
Neither specifying GCS bucket directly:

tensorboard --logdir=${GCS_LOGS_BUCKET} --reload_interval=2

, or using gcsfuse to synchronize local directory with a bucket and then point tensorboard to local directory, works, and in both cases it is necessary to restart tensorboard to refresh logs.
Any fix or workaround would be much appreciated.

[carlthome]

Status?