Tensorboard failed to visualize embedding

[randxie]

What happened:
The tensorboard UI failed to render embedding. It will get frozen with "computing PCA" and then the page becomes unresponsive.

What did you expect to happen:
Here's a normal tensorboard output.

What steps did you take:
I tried to upgrade the tensorboard version in the pod but no luck.

[neuromage]

Are you relying on a specific new version of Tensorboard? Can you reproduce the 'freeze' on a local machine using TB 1.11? This is the version we currently deploy in Pipelines:

pipelines/backend/src/crd/controller/viewer/reconciler/reconciler.go

Line 168 in 578e823

c.Image = "tensorflow/tensorflow:1.11.0"

/cc @jingzhang36

[randxie]

Hi @neuromage , thanks for pointing me to the image version. I downgraded my tensorflow version to 1.11 and was able to reproduce the error. Looks like such visualization requires TF 1.13.

[neuromage]

Ah ok makes sense. I think we can upgrade our TB version.
@jingzhang36 do you foresee any issues with this? If not, I'll go ahead and send a PR out in the morning.

[jingzhang36]

@neuromage Thanks for looking into this! Please go ahead with the version bump.

[randxie]

Btw, is there a way we can upgrade the tensorboard version in an existing cluster?

[jessiezcc]

@randxie, can you pls close this issue since it seems the issue is being resolved.