Add directory monitor for Latex citations (#11245)

* Add directory monitor

* Ignore parsing nested files

* Add listener for LaTex directory, update LatexCitationsTab

* Update CHANGELOG.md

* Decision record

* Refine ADR-030

* Update module-info.java

* handle non-existing directories

* Java Doc

* Java Doc

* Move directoryMonitor to Globals

---------

Co-authored-by: Oliver Kopp <kopp.dev@gmail.com>

CHANGELOG.md

@@ -18,6 +18,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv
 - We added tooltip on main table cells that shows cell content or cell content and entry preview if set in preferences. [10925](https://github.com/JabRef/jabref/issues/10925)
 - We added the ability to add a keyword/crossref when typing the separator character (e.g., comma) in the keywords/crossref fields. [#11178](https://github.com/JabRef/jabref/issues/11178)
 - We added an exporter and improved the importer for Endnote XML format. [#11137](https://github.com/JabRef/jabref/issues/11137)
+- We added support for automatically update LaTeX citations when a LaTeX file is created, removed, or modified. [#10585](https://github.com/JabRef/jabref/issues/10585)
 
 ### Changed
 

build.gradle

@@ -296,6 +296,8 @@ dependencies {
     // YAML formatting
     implementation 'org.yaml:snakeyaml:2.2'
 
+    implementation 'commons-io:commons-io:2.16.1'
+
     testImplementation 'io.github.classgraph:classgraph:4.8.170'
     testImplementation 'org.junit.jupiter:junit-jupiter:5.10.2'
     testImplementation 'org.junit.platform:junit-platform-launcher:1.10.2'

docs/decisions/0028-http-return-bibtex-string.md

@@ -2,8 +2,6 @@
 nav_order: 28
 parent: Decision Records
 ---
-<!-- we need to disable MD025, because we use the different heading "ADR Template" in the homepage (see above) than it is foreseen in the template -->
-<!-- markdownlint-disable-next-line MD025 -->
 # Return BibTeX string and CSL Item JSON in the API
 
 ## Context and Problem Statement

docs/decisions/0029-cff-export-multiple-entries.md

@@ -1,10 +1,7 @@
 ---
-nav_order: 28
+nav_order: 29
 parent: Decision Records
 ---
-
-<!-- we need to disable MD025, because we use the different heading "ADR Template" in the homepage (see above) than it is foreseen in the template -->
-<!-- markdownlint-disable-next-line MD025 -->
 # Exporting multiple entries to CFF
 
 ## Context and Problem Statement

docs/decisions/0030-use-apache-commons-io-for-directory-monitoring.md

@@ -0,0 +1,48 @@
+---
+nav_order: 30
+parent: Decision Records
+---
+# Use Apache Commons IO for directory monitoring
+
+## Context and Problem Statement
+
+In JabRef, there is a need to add a directory monitor that will listen for changes in a specified directory.
+
+Currently, the monitor is used to automatically update the [LaTeX Citations](https://docs.jabref.org/advanced/entryeditor/latex-citations) when a LaTeX file in the LaTeX directory is created, removed, or modified ([#10585](https://github.com/JabRef/jabref/issues/10585)).
+Additionally, this monitor will be used to create a dynamic group that mirrors the file system structure ([#10930](https://github.com/JabRef/jabref/issues/10930)).
+
+## Considered Options
+
+* Use [java.nio.file.WatchService](https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/nio/file/WatchService.html)
+* Use [io.methvin.watcher.DirectoryWatcher](https://github.com/gmethvin/directory-watcher)
+* Use [org.apache.commons.io.monitor](https://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/monitor/package-summary.html)
+
+## Decision Outcome
+
+Chosen option: "Use [org.apache.commons.io.monitor](https://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/monitor/package-summary.html)", because comes out best (see below).
+
+## Pros and Cons of the Options
+
+### java.nio.file.WatchService
+
+* Good, because it is a standard Java API for watching directories.
+* Good, because it does not need polling, it is event-based for most operating systems.
+* Bad, because:
+  1. Does not detect files coming together with a new folder (JDK issue: [JDK-8162948](https://bugs.openjdk.org/browse/JDK-8162948)).
+  2. Deleting a subdirectory does not detect deleted files in that directory.
+  3. Access denied when trying to delete the recursively watched directory on Windows (JDK issue: [JDK-6972833](https://bugs.openjdk.org/browse/JDK-6972833)).
+  4. Implemented on macOS by the generic `PollingWatchService`. (JDK issue: [JDK-8293067](https://bugs.openjdk.org/browse/JDK-8293067))
+
+### io.methvin.watcher.DirectoryWatcher
+
+* Good, because it implemented on top of the `java.nio.file.WatchService`, which is a standard Java API for watching directories.
+* Good, because it resolves some of the issues of the `java.nio.file.WatchService`.
+  * Uses`ExtendedWatchEventModifier.FILE_TREE` on Windows, which resolves issues (1, 3) of the `java.nio.file.WatchService`.
+  * On macOS have native implementation based on the Carbon File System Events API, this resolves issue (4) of the `java.nio.file.WatchService`.
+* Bad, because issue (2) of the `java.nio.file.WatchService` is not resolved.
+
+### org.apache.commons.io.monitor
+
+* Good, because there are no observed issues.
+* Good, because can handle huge amount of files without overflowing.
+* Bad, because it uses a polling mechanism at fixed intervals, which can waste CPU cycles if no change occurs.

external-libraries.md

@@ -294,6 +294,12 @@ Project: Apache Commons Digester
 URL:     https://commons.apache.org/proper/commons-digester/
 ```
 
+```yaml
+Id:      commons-io:commons-io
+Project: Apache Commons IO
+URL:     https://commons.apache.org/proper/commons-io/
+```
+
 ```yaml
 Id:      de.rototor.jeuclid:jeuclid-core
 Project: JEuclid
@@ -537,7 +543,7 @@ Id:      com.ibm.icu:*
 Project: International Components for Unicode
 URL:     https://icu.unicode.org/
 License: Unicode License (https://www.unicode.org/copyright.html)
-Note:    Our own fork https://github.com/JabRef/icu. Upstream PR: https://github.com/unicode-org/icu/pull/2127
+Note:    Our own fork https://github.com/JabRef/icu. [Upstream PR](https://github.com/unicode-org/icu/pull/2127)
 Path:    lib/icu4j.jar
 SourcePath: lib/ic4j-src.jar
 ```
@@ -579,49 +585,49 @@ License: LGPL-2.1-or-later
 
 ```yaml
 Id:      org.openjfx:javafx-base
-Project  JavaFX
+Project: JavaFX
 URL:     https://openjfx.io/
 License: GPL-2.0 WITH Classpath-exception-2.0
 ```
 
 ```yaml
 Id:      org.openjfx:javafx-controls
-Project  JavaFX
+Project: JavaFX
 URL:     https://openjfx.io/
 License: GPL-2.0 WITH Classpath-exception-2.0
 ```
 
 ```yaml
 Id:      org.openjfx:javafx-fxml
-Project  JavaFX
+Project: JavaFX
 URL:     https://openjfx.io/
 License: GPL-2.0 WITH Classpath-exception-2.0
 ```
 
 ```yaml
 Id:      org.openjfx:javafx-graphics
-Project  JavaFX
+Project: JavaFX
 URL:     https://openjfx.io/
 License: GPL-2.0 WITH Classpath-exception-2.0
 ```
 
 ```yaml
 Id:      org.openjfx:javafx-media
-Project  JavaFX
+Project: JavaFX
 URL:     https://openjfx.io/
 License: GPL-2.0 WITH Classpath-exception-2.0
 ```
 
 ```yaml
 Id:      org.openjfx:javafx-swing
-Project  JavaFX
+Project: JavaFX
 URL:     https://openjfx.io/
 License: GPL-2.0 WITH Classpath-exception-2.0
 ```
 
 ```yaml
 Id:      org.openjfx:javafx-web
-Project  JavaFX
+Project: JavaFX
 URL:     https://openjfx.io/
 License: GPL-2.0 WITH Classpath-exception-2.0
 ```
@@ -765,6 +771,7 @@ commons-cli:commons-cli:1.6.0
 commons-codec:commons-codec:1.16.0
 commons-collections:commons-collections:3.2.2
 commons-digester:commons-digester:2.1
+commons-io:commons-io:2.16.1
 commons-logging:commons-logging:1.2
 commons-validator:commons-validator:1.7
 de.rototor.jeuclid:jeuclid-core:3.1.11

src/main/java/module-info.java

@@ -54,6 +54,9 @@
     requires java.prefs;
     requires com.fasterxml.aalto;
 
+    // YAML
+    requires org.yaml.snakeyaml;
+
     // Annotations (@PostConstruct)
     requires jakarta.annotation;
     requires jakarta.inject;
@@ -88,13 +91,14 @@
     uses org.mariadb.jdbc.credential.CredentialPlugin;
 
     // Apache Commons and other (similar) helper libraries
+    requires com.google.common;
+    requires io.github.javadiffutils;
+    requires java.string.similarity;
     requires org.apache.commons.cli;
     requires org.apache.commons.csv;
+    requires org.apache.commons.io;
     requires org.apache.commons.lang3;
     requires org.apache.commons.text;
-    requires com.google.common;
-    requires io.github.javadiffutils;
-    requires java.string.similarity;
 
     requires com.github.tomtung.latex2unicode;
     requires fastparse;
@@ -146,5 +150,4 @@
     requires de.saxsys.mvvmfx.validation;
     requires dd.plist;
     requires mslinks;
-    requires org.yaml.snakeyaml;
 }

src/main/java/org/jabref/gui/Globals.java

@@ -5,6 +5,7 @@
 import org.jabref.gui.remote.CLIMessageHandler;
 import org.jabref.gui.telemetry.Telemetry;
 import org.jabref.gui.undo.CountingUndoManager;
+import org.jabref.gui.util.DefaultDirectoryMonitor;
 import org.jabref.gui.util.DefaultFileUpdateMonitor;
 import org.jabref.gui.util.DefaultTaskExecutor;
 import org.jabref.gui.util.TaskExecutor;
@@ -14,6 +15,7 @@
 import org.jabref.logic.remote.server.RemoteListenerServerManager;
 import org.jabref.logic.util.BuildInfo;
 import org.jabref.model.entry.BibEntryTypesManager;
+import org.jabref.model.util.DirectoryMonitor;
 import org.jabref.model.util.FileUpdateMonitor;
 import org.jabref.preferences.PreferencesService;
 
@@ -65,6 +67,7 @@ public class Globals {
     private static KeyBindingRepository keyBindingRepository;
 
     private static DefaultFileUpdateMonitor fileUpdateMonitor;
+    private static DefaultDirectoryMonitor directoryMonitor;
 
     private Globals() {
     }
@@ -92,6 +95,13 @@ public static synchronized FileUpdateMonitor getFileUpdateMonitor() {
         return fileUpdateMonitor;
     }
 
+    public static DirectoryMonitor getDirectoryMonitor() {
+        if (directoryMonitor == null) {
+            directoryMonitor = new DefaultDirectoryMonitor();
+        }
+        return directoryMonitor;
+    }
+
     // Background tasks
     public static void startBackgroundTasks() {
         // TODO Currently deactivated due to incompatibilities in XML
@@ -109,6 +119,11 @@ public static void shutdownThreadPools() {
         if (fileUpdateMonitor != null) {
             fileUpdateMonitor.shutdown();
         }
+
+        if (directoryMonitor != null) {
+            directoryMonitor.shutdown();
+        }
+
         JabRefExecutorService.INSTANCE.shutdownEverything();
     }
 

src/main/java/org/jabref/gui/LibraryTab.java

@@ -84,6 +84,7 @@
 import org.jabref.model.entry.field.Field;
 import org.jabref.model.entry.field.FieldFactory;
 import org.jabref.model.entry.field.StandardField;
+import org.jabref.model.util.DirectoryMonitorManager;
 import org.jabref.model.util.FileUpdateMonitor;
 import org.jabref.preferences.PreferencesService;
 
@@ -127,9 +128,9 @@ private enum PanelMode { MAIN_TABLE, MAIN_TABLE_AND_ENTRY_EDITOR }
 
     // Indicates whether the tab is loading data using a dataloading task
     // The constructors take care to the right true/false assignment during start.
-    private SimpleBooleanProperty loading = new SimpleBooleanProperty(false);
+    private final SimpleBooleanProperty loading = new SimpleBooleanProperty(false);
 
-    // initally, the dialog is loading, not saving
+    // initially, the dialog is loading, not saving
     private boolean saving = false;
 
     private PersonNameSuggestionProvider searchAutoCompleter;
@@ -151,6 +152,7 @@ private enum PanelMode { MAIN_TABLE, MAIN_TABLE_AND_ENTRY_EDITOR }
 
     private final IndexingTaskManager indexingTaskManager;
     private final TaskExecutor taskExecutor;
+    private final DirectoryMonitorManager directoryMonitorManager;
 
     private LibraryTab(BibDatabaseContext bibDatabaseContext,
                       LibraryTabContainer tabContainer,
@@ -171,6 +173,7 @@ private LibraryTab(BibDatabaseContext bibDatabaseContext,
         this.entryTypesManager = entryTypesManager;
         this.indexingTaskManager = new IndexingTaskManager(taskExecutor);
         this.taskExecutor = taskExecutor;
+        this.directoryMonitorManager = new DirectoryMonitorManager(Globals.getDirectoryMonitor());
 
         bibDatabaseContext.getDatabase().registerListener(this);
         bibDatabaseContext.getMetaData().registerListener(this);
@@ -832,6 +835,11 @@ private void onClosed(Event event) {
         } catch (RuntimeException e) {
             LOGGER.error("Problem when closing change monitor", e);
         }
+        try {
+            directoryMonitorManager.unregister();
+        } catch (RuntimeException e) {
+            LOGGER.error("Problem when closing directory monitor", e);
+        }
         try {
             PdfIndexerManager.shutdownIndexer(bibDatabaseContext);
         } catch (RuntimeException e) {
@@ -864,6 +872,10 @@ public BibDatabaseContext getBibDatabaseContext() {
         return this.bibDatabaseContext;
     }
 
+    public DirectoryMonitorManager getDirectoryMonitorManager() {
+        return directoryMonitorManager;
+    }
+
     public boolean isSaving() {
         return saving;
     }

src/main/java/org/jabref/gui/StateManager.java

@@ -190,7 +190,7 @@ public ObservableList<Task<?>> getBackgroundTasks() {
     }
 
     public void addBackgroundTask(BackgroundTask<?> backgroundTask, Task<?> task) {
-        this.backgroundTasks.add(0, new Pair<>(backgroundTask, task));
+        this.backgroundTasks.addFirst(new Pair<>(backgroundTask, task));
     }
 
     public EasyBinding<Boolean> getAnyTaskRunning() {

src/main/java/org/jabref/gui/entryeditor/EntryEditor.java

@@ -55,6 +55,7 @@
 import org.jabref.model.entry.BibEntry;
 import org.jabref.model.entry.BibEntryTypesManager;
 import org.jabref.model.entry.field.Field;
+import org.jabref.model.util.DirectoryMonitorManager;
 import org.jabref.model.util.FileUpdateMonitor;
 import org.jabref.preferences.PreferencesService;
 
@@ -82,6 +83,7 @@ public class EntryEditor extends BorderPane {
     private final BibDatabaseContext databaseContext;
     private final EntryEditorPreferences entryEditorPreferences;
     private final ExternalFilesEntryLinker fileLinker;
+    private final DirectoryMonitorManager directoryMonitorManager;
 
     private Subscription typeSubscription;
 
@@ -121,6 +123,7 @@ public class EntryEditor extends BorderPane {
     public EntryEditor(LibraryTab libraryTab) {
         this.libraryTab = libraryTab;
         this.databaseContext = libraryTab.getBibDatabaseContext();
+        this.directoryMonitorManager = libraryTab.getDirectoryMonitorManager();
 
         ViewLoader.view(this)
                   .root(this)
@@ -307,7 +310,7 @@ private List<EntryEditorTab> createTabs() {
                 bibEntryTypesManager,
                 keyBindingRepository);
         entryEditorTabs.add(sourceTab);
-        entryEditorTabs.add(new LatexCitationsTab(databaseContext, preferencesService, taskExecutor, dialogService));
+        entryEditorTabs.add(new LatexCitationsTab(databaseContext, preferencesService, dialogService, directoryMonitorManager));
         entryEditorTabs.add(new FulltextSearchResultsTab(stateManager, preferencesService, dialogService, taskExecutor));
 
         return entryEditorTabs;