Investigate flaky tensorflow tests

[merelcht]

Description

Follow up on #363 and #377

"Thinking about this, wonder if it makes sense to use something like https://github.com/box/flaky? It's used fairly widely, but it's not that actively-maintained (not that need new functionality). I'm just not sure if we even want the tests slowed down rerunning the test X times."

Context

In #363 we added some logic to skip the tensorflow tests on windows, because they started to fail randomly. Re-running would occasionally "fix" it.

In #377, a change was added to run tensorflow tests separately from the other tests, because when run as part of the full test suite they failed. Running locally and on Gitpod was always successful.