Support fsspec for reading remote directories

[f4hy]

A common gotcha on my team is using hdfs or other remote storage systems as the location to store the tensorboard logs. If tensorflow>2.0 is installed this works just fine, but if tf is not installed at all then writing the output works only for local or s3 locations.

When using tensorboard to write logs installing tensorflow just to get remote directory access is a pretty heavy dependency to add. Would the team be open to replacing the remote file access with something like fsspec (what pandas uses for remote file support) instead of gfile? If so I would be happy to take on the project.

[stephanwlee]

Hi @f4hy, sorry for a late reply.

I would like to understand your set up a little bit more. TensorBoard today does not let you write any summaries without TensorFlow. Are you asking for a possibility where you can emit logs without dependency on TensorFlow? If so, have you ever looked into tensorboardX (not our project)? tensorboardX does not support hdfs but if that's what you are using today, I would ask you to file an issue against them instead.

Above, you were explicitly asking about the writer parts of TensorBoard but are you at all hindered by the read path? Without TensorFlow installed, both rustboard and our TF shims do not support hdfs.

• tensorboard/tensorboard/data/server/cli/dynamic_logdir.rs

  Lines 29 to 32 in b6e8baf

  	pub enum DynLogdir {
  	Disk(DiskLogdir),
  	Gcs(gcs::Logdir),
  	}

• tensorboard/tensorboard/compat/tensorflow_stub/io/gfile.py

  Lines 404 to 406 in b6e8baf

  	register_filesystem("", LocalFileSystem())
  	if S3_ENABLED:
  	register_filesystem("s3", S3FileSystem())

[f4hy]

I am writing to tensorboard through pytorch, https://pytorch.org/docs/stable/tensorboard.html
This works perfectly fine to write summaries without tensorflow installed. However without tensorflow it goes thorugh

tensorboard/tensorboard/compat/tensorflow_stub/io/gfile.py

Line 15 in 9bb887b

"""A limited reimplementation of the TensorFlow FileIO API.

to emulate the tensorflow filesystem api. This works just fine for the LocalFilesystem and S3Filesystem defined in that file.

My proposal would be to remove these implementations of LocalFilesystem and S3filesystem in the compat/tensorflow_stub/io/gfile.py and instead use the package fsspec to implement a generic filesystem API which would support local/s3/gcs/hdfs/etc. If tensorflow was installed, it continue to use the tensorflow api, but when using the compat.tensorflow_stub implemented in tensorboard we can support more filesystems.

[d4l3k]

It would be really nice to have tensorboard compat fallback to using fsspec. There's a lot more file systems available through fsspec and is used by a number of major projects such as pytorch lightning https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/utilities/cloud_io.py#L41

Here's where pytorch currently integrates in with tensorboard https://github.com/pytorch/pytorch/blob/master/torch/utils/tensorboard/writer.py#L9

@stephanwlee if there's interested in adding support I'd be happy to send a PR

It shouldn't be too hard to add optional fsspec support that only runs if you have the dependency installed