Storage_server json inhibits ert from plotting after error encountered

[andreas-el]

Whenever Ert encounters an error during plotting, you cannot plot storage content anymore.
Ert will just hang.

---

Reproduce by:
First run poly-example to have something to plot.

Add some error, for example adding raise ValueError() at the bottom of observations_for_key in plot_api.py

                all_observations = pd.concat(
                    [all_observations, pd.DataFrame(data_struct)]
                )
        raise ValueError()
        return all_observations.T

Run ert and have the error raised when trying to plot.
Revert code snippet, and observe that you cannot plot content anymore.
Ert will just hang.

I've diffed the output and found this;

diff --color -Nau lol/poly_example/storage/storage_server.json test-data/poly_example/storage/storage_server.json
--- lol/poly_example/storage/storage_server.json	1970-01-01 01:00:00
+++ test-data/poly_example/storage/storage_server.json	2024-08-13 13:49:49
@@ -0,0 +1,8 @@
+{
+    "urls": [
+        "http://127.0.0.1:51834",
+        "http://AC-WR4GDXT41X:51834",
+        "http://AC-WR4GDXT41X:51834"
+    ],
+    "authtoken": "<token>"
+}
\ No newline at end of file

Removing that json file seems to unclog plotting again.

[sondreso]

I believe what's happening here is that the first exception takes down ERT, and it's not able to clean up the files properly (leaving the storage_server.json behind). The next time ERT starts it tries to pick up the session from this file instead of starting a new (the presence of the file indicates a running session that you can connect to), but it isn't valid anymore.

[JHolba]

the json file does not seem to be used any more. removing all reads/checks of it does not impact our tests. also tested plotter with snake oil case.

[JHolba]

this issue seems to be caused by ert not closing base_services on mac when ert goes down unexpectedly. dark storage will continue running and responding to base_url/healthcheck causing ert to think that it already has a running dark storage when starting up.