Replace tf_record_iterator with a non-deprecated way to read TFRecords

[nfelt]

TensorBoard currently relies on tf.pywrap_tensorflow.PyRecordReader_New() to read tfrecords files (aka event files). Not only is this bad because it's not strictly speaking a public API symbol, it's also being hidden in TF 2.0.

There is a reimplementation of record reading logic in pure python in our tensorflow_stub here:
https://github.com/tensorflow/tensorboard/blob/3f055ba9c057ad2fbacfbd2c61b6e27471a51396/tensorboard/compat/tensorflow_stub/pywrap_tensorflow.py

I haven't benchmarked this, but I expect that it's considerably slower than the C++ version from TensorFlow. If so and if we can't get the performance to be comparable to the C++ version, then when TF is available we should prefer the C++ code. To use that code, however, we'll need to expose it under a real public python API.

What exists today is:

• tf.train.summary_iterator - deprecated, will go away entirely in TF 2.0

• tf.TFRecordReader - deprecated queue-based reader being replaced by tf.data

• tf.io.tf_record_iterator - deprecated in favor of tf.data.TFRecordDataset for tf.data, and not enough for our needs

• tf.data.TFRecordDataset - tf.data-compatible Dataset, but ultimately still an iterator, and not enough for our needs

• tf.io.TFRecordWriter - writer side (so not immediately useful, but an example of what does still exist)

An iterator API isn't powerful enough for us because we need the ability to read to the end of a file and then wait for new data, preserving our read offset. As far as I can tell, all the above iterator approaches will abort iteration when they reach the end of the file, so reading new data would require re-reading all the previously consumed data. Also, I believe we currently rely on being able to handle cases where a record isn't fully flushed (e.g. the file appears to end in a corrupted record, but it's really just that the data isn't all there yet, and the appropriate response is to wait and retry), which the iterator APIs also don't allow.

So it's likely that we would need to basically create a new TFRecordReader. I think it would make the most sense to live at tf.io.TFRecordReader alongside the writer; we'd just have to clearly message that most users will actually want the TFRecordDataset (and that this reader is somewhat different from the old tf.TFRecordReader).

[nfelt]

For reference: PR #2096 changed our current imports from tf.compat.v1 to tensorflow.python, which is still available in TF 2.0 but not part of the public API (though pywrap_tensorflow was never covered by API guarantees in TF 1.x anyway).

[nfelt]

#3185 relaxes our use of PyRecordReader so that we only use it when necessary, and going forward, in new TF releases we will instead use tf.compat.v1.io.tf_record_iterator which was recently updated in tensorflow/tensorflow@54f1de4 to no longer abandon the read offset after reaching EOF (which resolves the issue described above that prevented us from using it, since we couldn't use it to poll the end of a file for newly appended data).

The remaining work is as follows:

1. Add tests to TF's tf_record_test.py to check for read offset preservation on error or EOF, to avoid any regressions in the functionality of tf_record_iterator that we require.

2. Ensure that we don't inadvertently fall back to using the tensorflow_stub/pywrap_tensorflow.py no-TF implementation of PyRecordReader_New. Right now our tensorboard.compat._pywrap_tensorflow compatibility mechanism for no-TF support will get triggered once tensorflow.python.pywrap_tensorflow disappears entirely, which would result in using the no-TF stub reader code even when TF is available, which is undesirable. We should prevent this by just removing that compatibility mechanism entirely - I think it's used only from event_file_loader.py (projector_plugin.py imports it but doesn't use it; I think it's an accidental leftover there).

3. Migrate away from tf_record_iterator to something that is not deprecated, in the long term. At that point this becomes a particularly thorny subtask of Reduce usage of tf.compat.v1 APIs #1718 about avoiding dependencies on tf.compat.v1.

[nfelt]

We've now done remaining work items 1 (tensorflow/tensorflow@db26bd5) and 2 (#3185).

I'll leave this issue open to track item 3 but rename it to capture the adjustment in scope.