Tensorboard is not working

[wsnoble]

When I train a model with tensorboard turned on, I get a numpy error about mismatched dimensions:

casanovo --mode=train --config config.yaml --peak_path=target.train.100k.mgf --peak_path_val=target.train.100k.mgf
...
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

The full stderr output, as well as the input file and the config file, are attached. This same command works fine if I turn off tensorboard output in the config file.

stderr.txt
config.txt
target.train.100k.mgf.txt

[bittremieux]

Can you try to change metric_value in this line to metric_value.cpu() and see if it works then?

[wsnoble]

I tried this:

                   metric_value = metrics.get(key, np.nan)
                    if not np.isnan(metric_value):
                        self.tb_summarywriter.add_scalar(
                            descr, metric_value.cpu(), metrics["step"]
                        )

The behavior remains unchanged.

[cfmelend]

Was able to replicate this bug and prevent it from occurring with the same workaround that @wsnoble mentions in the original post (ie. remove the tb_summarywriter config parameter/set it to None).

[bittremieux]

Apparently it works on dev, closing.