Tidier dies silently

[kevinphippsstfc]

This was found during a recent incident where the disk partition for the DLS HTTP IDS filled up because the Tidier was no longer deleting files to keep the cache within the configured size limits.

The partition filling up was the initial problem that was found. Upon inspection of the ids.log files it was then found that there had been no logging from the Tidier for over a month. As only the last 30 days of log files are kept, it was not possible to tell from the ids.log when the Tidier last ran. However, the server.log file had this entry in it:

[2023-08-07T18:32:38.629+0100] [Payara 6.2023.6] [SEVERE] [] [] [tid: _ThreadID=250 _ThreadName=Timer-1] [timeMillis: 1691429558629] [levelValue: 1000] [[
  Exception in thread "Timer-1" ]]

[2023-08-07T18:32:39.514+0100] [Payara 6.2023.6] [SEVERE] [] [] [tid: _ThreadID=250 _ThreadName=Timer-1] [timeMillis: 1691429559514] [levelValue: 1000] [[
  java.lang.IllegalArgumentException: Comparison method violates its general contract!
        at java.base/java.util.TimSort.mergeLo(TimSort.java:781)
        at java.base/java.util.TimSort.mergeAt(TimSort.java:518)
        at java.base/java.util.TimSort.mergeCollapse(TimSort.java:448)
        at java.base/java.util.TimSort.sort(TimSort.java:245)
        at java.base/java.util.Arrays.sort(Arrays.java:1515)
        at java.base/java.util.ArrayList.sort(ArrayList.java:1750)
        at java.base/java.util.Collections.sort(Collections.java:179)
        at uk.ac.stfc.storaged.TreeSizeVisitor.getDates(TreeSizeVisitor.java:41)
        at uk.ac.stfc.storaged.MainSDStorage.getDatafilesToArchive(MainSDStorage.java:85)
        at org.icatproject.ids.Tidier$Action.run(Tidier.java:89)
        at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
        at java.base/java.util.TimerThread.run(Timer.java:506)
]]

which I think indicates when the Tidier thread died and what caused it.

From some googling I think it points to the following issue:
https://stackoverflow.com/questions/58418782/sorting-list-of-files-in-android-throwing-comparison-method-violates-its-genera
which can happen when line 41 of the TreeSizeVisitor Collections.sort(dates, dateComparator); is run and for some reason the modified date of one or more of the files that it is trying to order gets modified again.

This presumably does not happen very often as this is the first time I have come across it and the line is in the dls-ids-plugin component which was used when DLS were running the ids.server component and is still being used in ids.r2dfoo, but nonetheless it could happen again so we should fix it.