Incomplete exit caused by FileNotFoundError in tensorboard

[KumoLiu]

When file missing in tensorboard, the process can not exit correctly.

2024-02-02 08:56:07,591 - INFO - Epoch: 1/800, Iter: 9/16 -- train_loss: 0.6617 
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/event_file_writer.py", line 233, in run
    self._record_writer.write(data)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/summary/writer/record_writer.py", line 40, in write
    self._writer.write(header + header_crc + data + footer_crc)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 766, in write
    self.fs.append(self.filename, file_content, self.binary_mode)
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 160, in append
    self._write(filename, file_content, "ab" if binary_mode else "a")
  File "/usr/local/lib/python3.10/dist-packages/tensorboard/compat/tensorflow_stub/io/gfile.py", line 164, in _write
    with io.open(filename, mode, encoding=encoding) as f:
FileNotFoundError: [Errno 2] No such file or directory: b'/workspace/Code/eval/events.out.tfevents.1706864132.yunliu-ms-7d31.825351.1'
2024-02-02 08:56:13,231 - INFO - Epoch: 1/800, Iter: 10/16 -- train_loss: 0.6130 

[KumoLiu]

tensorboard has fixed the issue since 2.12.0, will update the version of the tensorboard to fix the issue.
tensorflow/tensorboard#6168