SQLite Tensorflow Data Export Docs

[derenrich]

I see that tensorboard now supports reading logs from sqlite but I'm not seeing any documentation on how to produce that format of data. Is it not possible/suggested yet? the closest i could find was this test case https://github.com/tensorflow/tensorflow/blob/0978e060af5a7c7ad22ead745ac781d6c86e1985/tensorflow/contrib/summary/summary_ops_graph_test.py#L240

[nfelt]

Thanks for asking! The support for reading event data from a SQLite DB is still experimental, so we haven't yet written proper documentation for it. Also, we aren't yet guaranteeing any stability in the DB schema/format, so I wouldn't rely on it as your only data format for this data.

However, if you'd like to give it a try, you can use tf.contrib.summary.create_db_writer() as shown in the test code you found, which has a little documentation here: https://www.tensorflow.org/api_docs/python/tf/contrib/summary/create_db_writer

There's also a very small amount of relevant test logic in TensorBoard's CorePluginTest here:
https://github.com/tensorflow/tensorboard/blob/1.9.0/tensorboard/plugins/core/core_plugin_test.py#L260

[derenrich]

Thanks for the pointers. I got it working nicely. I also found https://www.tensorflow.org/api_docs/python/tf/contrib/summary generically helpful though the formatting there is broken.

Is there an ETA for a stabilization of the sqlite format? I'm working on surrounding tooling where we're considering persisting these files (and maybe read/write them outside of tensorflow/board) and I'd rather not persist the proto log files personally

[nfelt]

Great, glad to hear it. Unfortunately at this point I can't give a specific ETA for stability of the sqlite schema, other than to say that as a lower bound it's probably still at least a couple months away.

The proto log files are actually much more stable - is there a particular reason you'd rather not persist them? If you were able to persist those as a backup, we expect to have tooling to convert those logs into whatever ends up being the stable sqlite format, so that would give you a way to use sqlite now and still be forwards-compatible with future changes.

[derenrich]

I have a desire to consume the logs programmatically outside of tensorboard (in addition to tensorboard) and the SQL format is just easier for me to deal with. Yeah I might fall back to the logs because of what you say

[nfelt]

Okay. FYI, there's tf.train.summary_iterator() which you can use to parse the event log file format.

[Elites2017]

I used the MobileNet_v1_1.0_224 tensorflow model for object detection. Now, I have my custom frozen graph (.pb file) that I need to convert to tflite extension so that I can use my model for mobile devices.

When I tried to convert it with the toco tool --output_node_names=MobileNetV1/Predictions/Reshape_1 I got the ValueError: Invalid tensors 'input' were found. How can I see theinput and output node names inthis tensorboard graph obtained after training?

[nfelt]

@Elites2017 it doesn't look like your problem is related to this GitHub issue, which is about SQLite data export documentation. Please ask on StackOverflow or open a new GitHub issue if you think your issue is a bug in TensorBoard.

[Elites2017]

@Elites2017 it doesn't look like your problem is related to this GitHub issue, which is about SQLite data export documentation. Please ask on StackOverflow or open a new GitHub issue if you think your issue is a bug in TensorBoard.

Ok, thank you. I don't think it's a bug. I just haven't found documentation on how to get the input and output node names from tensorboard graph.

[nfelt]

Closing this since the experimental DB mode has been removed (both the writing side, which was only in tf.contrib.summary and didn't outlast TF 1.x, and the TensorBoard reading side, which was removed in #3539). We're still looking into ways to improve data access (#3465) but we've decided sqlite as an on-disk format wasn't really the right way to address this.