Limit re-try interval/count when connection to StorageD fails

[kevinphippsstfc]

This morning there seems to be a problem on the Diamond preprod IDS server (dlsids-pre) that looks like the IDS cannot connect to StorageD but it just keeps trying and trying in quick succession and it is producing lots of log output.

Something needs adding to either increase the interval before the IDS tries again and/or to only try a number of times before giving up.

The bottom of the log file contains the following:

2023-03-06 09:57:01,761 INFO [Thread-717452521] RestorerThreadManager - Removing thread for preparedId 6c2f9ba3-a86a-4242-a12f-b97d30cc2700 from list 2023-03-06 09:57:01,761 DEBUG [Thread-717452518] RestorerThreadManager - Found 0/2 files requested already on main storage 2023-03-06 09:57:01,761 INFO [Thread-717452525] ArchiveSDStorageV2 - Requesting files from StorageD: 1 2023-03-06 09:57:01,761 DEBUG [Thread-717452521] RestorerThread - RestorerThread finishing for preparedId 6c2f9ba3-a86a-4242-a12f-b97d30cc2700 2023-03-06 09:57:01,761 DEBUG [Thread-717452518] CompletedRestoresManager - No completed file found to delete for preparedId 7817dbe9-0832-4284-9d96-cf78e0758477 2023-03-06 09:57:01,761 ERROR [Thread-717452523] RestorerThread - IOException for preparedId ID 7817dbe9-0832-4284-9d96-cf78e0758477. Recreating restorer thread. java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na] at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[na:na] at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[na:na] at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[na:na] at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:na] at java.base/java.net.Socket.connect(Socket.java:609) ~[na:na] at org.icatproject.ids.storage.ArchiveSDStorageV2.restore(ArchiveSDStorageV2.java:167) ~[ArchiveSDStorageV2.class:na] at org.icatproject.ids.thread.RestorerThread.run(RestorerThread.java:54) ~[RestorerThread.class:na] 2023-03-06 09:57:01,761 DEBUG [Thread-717452525] ArchiveSDStorageV2 - Connecting to fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,761 DEBUG [Thread-717452517] RestorerThreadManager - Found 0/892 files requested already on main storage 2023-03-06 09:57:01,761 DEBUG [Thread-717452518] ArchiveSDStorageV2 - Using property plugin.archive.uri: //fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,761 DEBUG [Thread-717452523] RestorerThreadManager - Found 0/2 files requested already on main storage 2023-03-06 09:57:01,761 DEBUG [Thread-717452517] CompletedRestoresManager - No completed file found to delete for preparedId 3fb309e6-690a-41f2-8ed8-adeba1a26d62 2023-03-06 09:57:01,761 DEBUG [Thread-717452518] ArchiveSDStorageV2 - Using property plugin.archive.connectionTimeout: 10000 2023-03-06 09:57:01,761 ERROR [Thread-717452522] RestorerThread - IOException for preparedId ID 6c2f9ba3-a86a-4242-a12f-b97d30cc2700. Recreating restorer thread. java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na] at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[na:na] at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[na:na] at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[na:na] at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:na] at java.base/java.net.Socket.connect(Socket.java:609) ~[na:na] at org.icatproject.ids.storage.ArchiveSDStorageV2.restore(ArchiveSDStorageV2.java:167) ~[ArchiveSDStorageV2.class:na] at org.icatproject.ids.thread.RestorerThread.run(RestorerThread.java:54) ~[RestorerThread.class:na] 2023-03-06 09:57:01,761 DEBUG [Thread-717452523] CompletedRestoresManager - No completed file found to delete for preparedId 7817dbe9-0832-4284-9d96-cf78e0758477 2023-03-06 09:57:01,761 DEBUG [Thread-717452517] ArchiveSDStorageV2 - Using property plugin.archive.uri: //fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,761 ERROR [Thread-717452525] ArchiveSDStorageV2 - IOException restoring from StorageD: class java.net.ConnectException Connection refused (Connection refused) 2023-03-06 09:57:01,761 DEBUG [Thread-717452518] ArchiveSDStorageV2 - Using property plugin.archive.readTimeout: 28800000 2023-03-06 09:57:01,761 DEBUG [Thread-717452522] RestorerThreadManager - Found 0/1 files requested already on main storage 2023-03-06 09:57:01,761 DEBUG [Thread-717452523] ArchiveSDStorageV2 - Using property plugin.archive.uri: //fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,761 DEBUG [Thread-717452517] ArchiveSDStorageV2 - Using property plugin.archive.connectionTimeout: 10000 2023-03-06 09:57:01,761 INFO [Thread-717452518] ArchiveSDStorageV2 - ArchiveSDStorageV2 initialized 2023-03-06 09:57:01,761 DEBUG [Thread-717452522] CompletedRestoresManager - No completed file found to delete for preparedId 6c2f9ba3-a86a-4242-a12f-b97d30cc2700 2023-03-06 09:57:01,761 DEBUG [Thread-717452523] ArchiveSDStorageV2 - Using property plugin.archive.connectionTimeout: 10000 2023-03-06 09:57:01,761 DEBUG [Thread-717452517] ArchiveSDStorageV2 - Using property plugin.archive.readTimeout: 28800000 2023-03-06 09:57:01,761 DEBUG [Thread-717452518] RestorerThread - Successfully instantiated org.icatproject.ids.storage.ArchiveSDStorageV2 for 7817dbe9-0832-4284-9d96-cf78e0758477 2023-03-06 09:57:01,761 DEBUG [Thread-717452522] ArchiveSDStorageV2 - Using property plugin.archive.uri: //fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,761 DEBUG [Thread-717452523] ArchiveSDStorageV2 - Using property plugin.archive.readTimeout: 28800000 2023-03-06 09:57:01,762 INFO [Thread-717452517] ArchiveSDStorageV2 - ArchiveSDStorageV2 initialized 2023-03-06 09:57:01,762 DEBUG [Thread-717452522] ArchiveSDStorageV2 - Using property plugin.archive.connectionTimeout: 10000 2023-03-06 09:57:01,762 INFO [Thread-717452523] ArchiveSDStorageV2 - ArchiveSDStorageV2 initialized 2023-03-06 09:57:01,762 INFO [Thread-717452518] RestorerThreadManager - Removing thread for preparedId 7817dbe9-0832-4284-9d96-cf78e0758477 from list 2023-03-06 09:57:01,762 DEBUG [Thread-717452517] RestorerThread - Successfully instantiated org.icatproject.ids.storage.ArchiveSDStorageV2 for 3fb309e6-690a-41f2-8ed8-adeba1a26d62 2023-03-06 09:57:01,762 DEBUG [Thread-717452522] ArchiveSDStorageV2 - Using property plugin.archive.readTimeout: 28800000 2023-03-06 09:57:01,762 INFO [Thread-717452526] ArchiveSDStorageV2 - Requesting files from StorageD: 2 2023-03-06 09:57:01,762 DEBUG [Thread-717452523] RestorerThread - Successfully instantiated org.icatproject.ids.storage.ArchiveSDStorageV2 for 7817dbe9-0832-4284-9d96-cf78e0758477 2023-03-06 09:57:01,762 DEBUG [Thread-717452518] RestorerThread - RestorerThread finishing for preparedId 7817dbe9-0832-4284-9d96-cf78e0758477 2023-03-06 09:57:01,762 INFO [Thread-717452522] ArchiveSDStorageV2 - ArchiveSDStorageV2 initialized 2023-03-06 09:57:01,762 DEBUG [Thread-717452526] ArchiveSDStorageV2 - Connecting to fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,762 INFO [Thread-717452517] RestorerThreadManager - Removing thread for preparedId 3fb309e6-690a-41f2-8ed8-adeba1a26d62 from list 2023-03-06 09:57:01,762 DEBUG [Thread-717452522] RestorerThread - Successfully instantiated org.icatproject.ids.storage.ArchiveSDStorageV2 for 6c2f9ba3-a86a-4242-a12f-b97d30cc2700 2023-03-06 09:57:01,762 INFO [Thread-717452527] ArchiveSDStorageV2 - Requesting files from StorageD: 892 2023-03-06 09:57:01,762 INFO [Thread-717452523] RestorerThreadManager - Removing thread for preparedId 7817dbe9-0832-4284-9d96-cf78e0758477 from list 2023-03-06 09:57:01,762 INFO [Thread-717452528] ArchiveSDStorageV2 - Requesting files from StorageD: 2 2023-03-06 09:57:01,762 DEBUG [Thread-717452517] RestorerThread - RestorerThread finishing for preparedId 3fb309e6-690a-41f2-8ed8-adeba1a26d62 2023-03-06 09:57:01,762 ERROR [Thread-717452526] ArchiveSDStorageV2 - IOException restoring from StorageD: class java.net.ConnectException Connection refused (Connection refused) 2023-03-06 09:57:01,762 INFO [Thread-717452522] RestorerThreadManager - Removing thread for preparedId 6c2f9ba3-a86a-4242-a12f-b97d30cc2700 from list 2023-03-06 09:57:01,762 DEBUG [Thread-717452523] RestorerThread - RestorerThread finishing for preparedId 7817dbe9-0832-4284-9d96-cf78e0758477 2023-03-06 09:57:01,762 INFO [Thread-717452529] ArchiveSDStorageV2 - Requesting files from StorageD: 1 2023-03-06 09:57:01,762 DEBUG [Thread-717452528] ArchiveSDStorageV2 - Connecting to fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,762 DEBUG [Thread-717452527] ArchiveSDStorageV2 - Connecting to fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,763 DEBUG [Thread-717452522] RestorerThread - RestorerThread finishing for preparedId 6c2f9ba3-a86a-4242-a12f-b97d30cc2700 2023-03-06 09:57:01,763 DEBUG [Thread-717452529] ArchiveSDStorageV2 - Connecting to fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,763 ERROR [Thread-717452527] ArchiveSDStorageV2 - IOException restoring from StorageD: class java.net.ConnectException Connection refused (Connection refused) 2023-03-06 09:57:01,763 ERROR [Thread-717452528] ArchiveSDStorageV2 - IOException restoring from StorageD: class java.net.ConnectException Connection refused (Connection refused) 2023-03-06 09:57:01,763 ERROR [Thread-717452529] ArchiveSDStorageV2 - IOException restoring from StorageD: class java.net.ConnectException Connection refused (Connection refused) 2023-03-06 09:57:01,764 ERROR [Thread-717452519] RestorerThread - IOException for preparedId ID e96c45c6-e40a-4c58-9272-fa7530338fe7. Recreating restorer thread. java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na] at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[na:na] at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[na:na] at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[na:na] at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:na] at java.base/java.net.Socket.connect(Socket.java:609) ~[na:na] at org.icatproject.ids.storage.ArchiveSDStorageV2.restore(ArchiveSDStorageV2.java:167) ~[ArchiveSDStorageV2.class:na] at org.icatproject.ids.thread.RestorerThread.run(RestorerThread.java:54) ~[RestorerThread.class:na] 2023-03-06 09:57:01,764 DEBUG [Thread-717452519] RestorerThreadManager - Found 0/1 files requested already on main storage 2023-03-06 09:57:01,764 DEBUG [Thread-717452519] CompletedRestoresManager - No completed file found to delete for preparedId e96c45c6-e40a-4c58-9272-fa7530338fe7 2023-03-06 09:57:01,764 DEBUG [Thread-717452519] ArchiveSDStorageV2 - Using property plugin.archive.uri: //fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,764 DEBUG [Thread-717452519] ArchiveSDStorageV2 - Using property plugin.archive.connectionTimeout: 10000 2023-03-06 09:57:01,764 DEBUG [Thread-717452519] ArchiveSDStorageV2 - Using property plugin.archive.readTimeout: 28800000 2023-03-06 09:57:01,764 INFO [Thread-717452519] ArchiveSDStorageV2 - ArchiveSDStorageV2 initialized 2023-03-06 09:57:01,764 DEBUG [Thread-717452519] RestorerThread - Successfully instantiated org.icatproject.ids.storage.ArchiveSDStorageV2 for e96c45c6-e40a-4c58-9272-fa7530338fe7 2023-03-06 09:57:01,764 INFO [Thread-717452519] RestorerThreadManager - Removing thread for preparedId e96c45c6-e40a-4c58-9272-fa7530338fe7 from list 2023-03-06 09:57:01,764 DEBUG [Thread-717452519] RestorerThread - RestorerThread finishing for preparedId e96c45c6-e40a-4c58-9272-fa7530338fe7 2023-03-06 09:57:01,764 INFO [Thread-717452530] ArchiveSDStorageV2 - Requesting files from StorageD: 1 2023-03-06 09:57:01,764 DEBUG [Thread-717452530] ArchiveSDStorageV2 - Connecting to fdsstoraged24.fds.rl.ac.uk:7456 2023-03-06 09:57:01,765 ERROR [Thread-717452530] ArchiveSDStorageV2 - IOException restoring from StorageD: class java.net.ConnectException Connection refused (Connection refused)