HParams plugin ModuleNotFoundError v1.13.1

[moritzmeister]

Installing Tensorflow/board v1.13.1 via pip, it seems like the pip package is missing the HParams plugin.

Importing it from tensorboard.plugins import hparams
I get a ModuleNotFoundError: No module named 'tensorboard.plugins.hparams'

But the module is in the release branch of 1.13.0 here on GitHub? Am I missing something?
@wchargin can shed some light? why is the HParams plugin not included in the build?

• TensorBoard version:
  v1.13.0
• OS Platform and version
  Ubuntu 16.04
• Python version
  v3.6.8

help(tensorboard.plugins)

Help on package tensorboard.plugins in tensorboard:

NAME
    tensorboard.plugins

PACKAGE CONTENTS
    audio (package)
    base_plugin
    beholder (package)
    core (package)
    custom_scalar (package)
    debugger (package)
    distribution (package)
    graph (package)
    histogram (package)
    image (package)
    interactive_inference (package)
    pr_curve (package)
    profile (package)
    projector (package)
    scalar (package)
    text (package)

[Harshini-Gadige]

Can you try with tb-nightly and let us know if you are getting the same error?

[wchargin]

Hi @moritzmeister—you’re correct that hparams is not included in the
1.13 series. A version of the code is in the repository, but #1882 had
not merged by the time that we released TensorBoard 1.13.0, so the
plugin was not enabled or included in the Pip package. This is also why
it’s not mentioned in the release notes (RELEASE.md).

The hparams plugin will be in TensorBoard 1.14, and is in our nightly
builds (pip uninstall tensorboard && pip install -U tb-nightly).

[moritzmeister]

Thanks for the clarification @wchargin :)
When will 1.14 be released? Only with TF2.0 later this year or earlier?

[wchargin]

1.14 will (likely) be released earlier than TF 2.0, and will be
compatible with the 1.x series.

[moritzmeister]

Thanks again!
I got one last question though. I saw the ml_demo you created, and I am wondering about the comment in the beginning:

The tag names emitted for Keras metrics changed from "acc" (in 1.x)
to "accuracy" (in 2.x), so this demo does not work properly in
TensorFlow 1.x (even with tf.enable_eager_execution()).

How are the metrics written by the Keras TensorBoard callback mapped to the Metrics in the HParams View?
Are the tags that you define in the experiment Proto the same as Keras uses in the summary it writes? If these tags create the connection, how can I check the tags of the metrics that Keras is writing? (So I can adjust for TF1.x)

metrics = [ api_pb2.MetricInfo( name=api_pb2.MetricName(group="validation", tag="epoch_accuracy"), display_name="accuracy (val.)", ),

Thank you so much!

[wchargin]

How are the metrics written by the Keras TensorBoard callback mapped
to the Metrics in the HParams View? Are the tags that you define in
the experiment Proto the same as Keras uses in the summary it writes?

Yes; metrics are identified by the tag of the associated scalar summary
(not, for instance, than the Keras-specific metric “name”).

how can I check the tags of the metrics that Keras is writing?

The result of model.fit has a history attribute, which is a dict.
I can’t find any specification of what this is supposed to contain, but
empirically it appears to approximately map the tag names to their lists
of scalar values. With

result = model.fit(...)
print(list(result.history))

the output is ['loss', 'val_accuracy', 'val_loss', 'accuracy'] in
current TF 2.0 nightly, and ['acc', 'loss', 'val_acc', 'val_loss'] in
the current TF 1.x nightly, whence you can discern what you want. But do
note that I say “approximately”, because in TF 2.0 val_accuracy is not
actually a tag name at all; instead, there is one train run and one
validation run; each has a scalar summary with tag accuracy.

Hope this helps. Sorry that I don’t have a clear-cut answer.

[wchargin]

Update: I’ve asked the Keras team; we’ll bump tf.keras.__version__ at
the TF 2.0 release. Currently, that version is 2.2.4; as of TF 2.0, it
will likely be 2.3.

[moritzmeister]

Hey @wchargin! Thanks one more time.

I've been trying out your resent additions to the Python API, because I really like what you are doing there and I think it's very useful. But there is one thing that I might have misunderstood, you said in an earlier answer:

1.14 will (likely) be released earlier than TF 2.0, and will be
compatible with the 1.x series.

However, you are using this tf.summary.experimental.write_raw_pb(raw_pb, step=0)
method which is not in the TF1.x series. And also not in the TF 2.0.0a0 preview:

module 'tensorflow._api.v2.compat.v2.summary.experimental' has no attribute 'write_raw_pb' Traceback (most recent call last): File "/srv/hops/anaconda/anaconda/envs/hparams/lib/python3.6/site-packages/maggy/tensorboard.py", line 57, in write_hparams_proto hp.hparams_config(hparams=HPARAMS, metrics=METRICS) File "/srv/hops/anaconda/anaconda/envs/hparams/lib/python3.6/site-packages/tensorboard/plugins/hparams/summary_v2.py", line 131, in hparams_config return _write_summary("hparams_config", pb) File "/srv/hops/anaconda/anaconda/envs/hparams/lib/python3.6/site-packages/tensorboard/plugins/hparams/summary_v2.py", line 225, in _write_summary return tf.summary.experimental.write_raw_pb(raw_pb, step=0) AttributeError: module 'tensorflow._api.v2.compat.v2.summary.experimental' has no attribute 'write_raw_pb'

I suppose it is only in the nightly of TF2.0. Do I see this right and how is the TensorBoard 1.14 then going to work with the TF1.x series?