Synchronize file links issues when .bib is in the same directory as the PDFs

[felker]

• I have tested the latest master version from http://builds.jabref.org/master/ and the problem persists

JabRef version 4.1 on Mac OS X 10.13.2 x86_64 with Java 1.8.0_151

I am a recent adopter of JabRef after I finally had enough with Mendeley, which had become completely incapable of opening my library. So, I exported what I could to .bib and arduously cleaned up the entries in JabRef.

The final step in this process was to ensure that the database was in 1-1 correspondence with my Published/ directory which contained all the PDFs. I was hoping to synchroize the majority of the files automatically, and fix the names of the remaining unliked PDFs with names not matching the JabRef entry BibTeX keys However, running the file link tools in JabRef has not worked for me. In addition to failing to link the eligible files in v 4.1, JabRef will open the "Undefined file type" prompt for file type "PDF" repeatedly, even after "Define PDF" is completed.

Steps to reproduce:

1. Quality > Synchronize file links
2. Select "Automatically set file links. Do not overwrite existing links."
3. Select "Check existing file links"

Log File

Opening: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib
Problem executing command
java.util.concurrent.ExecutionException: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib.sav.lock
  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
  at org.jabref.JabRefExecutorService.executeAndWait(JabRefExecutorService.java:60)
  at org.jabref.gui.externalfiles.SynchronizeFileField.run(SynchronizeFileField.java:124)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at spin.Invocation.evaluate(Invocation.java:175)
  at spin.off.SpinOffEvaluator$1.run(SpinOffEvaluator.java:108)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib.sav.lock
  at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88)
  at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
  at java.util.Iterator.forEachRemaining(Iterator.java:115)
  at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
  at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
  at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
  at org.jabref.logic.util.io.CiteKeyBasedFileFinder.findFilesByExtension(CiteKeyBasedFileFinder.java:95)
  at org.jabref.logic.util.io.CiteKeyBasedFileFinder.findAssociatedFiles(CiteKeyBasedFileFinder.java:49)
  at org.jabref.gui.externalfiles.AutoSetFileLinksUtil.findAssociatedNotLinkedFiles(AutoSetFileLinksUtil.java:51)
  at org.jabref.gui.externalfiles.AutoSetLinks.lambda$autoSetLinks$2(AutoSetLinks.java:89)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  ... 1 more
Caused by: java.nio.file.NoSuchFileException: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib.sav.lock
  at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
  at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
  at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
  at java.nio.file.Files.readAttributes(Files.java:1737)
  at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
  at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
  at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
  at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:95)
  ... 17 more 

Note, on the development version JabRef_macos_4_2-dev--snapshot--2018-01-10--master--5cbeb2baa.dmg, the Log File is simply:

Opening: /Users/kfelker/Documents/Research/References/Published/jabref_db.bib
java.io.IOException: Problem in finding files
java.io.IOException: Problem in finding files
java.io.IOException: Problem in finding files
java.io.IOException: Problem in finding files
...

[LinusDietz]

Yeap, that was something we worked on quite recently. (see c93ae63) I guess the problem is that I can't link all files accordingly and the exception handling (maybe only the displayed message) is wrong.

[felker]

I see. So is there a way to fix the "Define PDF" issue, independent of the exception handling?

[tobiasdiez]

Short version for @felker: The problem occurs because your bib file is in the same folder as the PDFs. If you (temporarily) move your bib file to a parent directory and set the PDF file directory in File > Database properties to the sub-directory, then the file link util should work as expected.

Short version for developers: Files.find and Files.walk suck.

Long version: The reason for the problem is that some files are changed between the creation of the stream (Files.find) and the access in collect. As of now the java API does not provide a way to catch and handle exceptions that occur while walking the tree. This is pretty suboptimal since IOExceptions can happen all the time for a variety of reasons (here it that the lock file is deleted, but it could also be security problems / ... ). There is an open feature request but this is inactive since 4 years.... On the bright side of things, there even was a patch proposed. So it should not be too hard to use this patch and create our own walk and find methods in FileUtil.

[felker]

Thanks for the reply @tobiasdiez . This fixed both the Log errors and the Define PDF issue.

However, I am not sure that the "Synchronize file links" or "Automatically set file links" abilities are working correctly. For example, I have an entry with bibtexkey 'AuerPaletou1994' and no linked file. There is matching file at /Users/kfelker/Documents/Research/References/Published/Radiation/AuerPaletou1994.pdf. This should be automatically linked by JabRef-- it does recursively search subfolders by default, right? However, the following steps:

1. Quality > Synchronize file links
2. Select "Automatically set file links. Allow overwriting existing links."
3. Select "Check existing file links"

fails to add the link, and the result says "Finished synchronizing file links. Entries changed: 336" which is my entire library, but most do not end up file links. I am guessing that is due to "Check existing file links"?

My relevant settings are:

• Preferences > File > External file links
  • Main file directory = /Users/kfelker/Documents/Research/References/Published
  • Autolink files with names starting with the BibTeX key

In addition, I produced a new Log error (but unfortunately I did not record the steps that caused it)

Opening: /Users/kfelker/Desktop/bib_database/jabref_db.bib
Uncaught exception occurred in Thread[JavaFX Application Thread,5,main]
java.lang.IndexOutOfBoundsException: [-2147483648, 9) is not a valid range within [0, 9)
	at org.reactfx.util.Lists.checkRange(Lists.java:128)
	at org.reactfx.util.Lists.checkRange(Lists.java:123)
	at org.reactfx.collection.MemoizationListImpl.forget(MemoizationList.java:184)
	at org.fxmisc.flowless.CellListManager.cropTo(CellListManager.java:82)
	at org.fxmisc.flowless.CellPositioner.cropTo(CellPositioner.java:29)
	at org.fxmisc.flowless.Navigator.cropToNeighborhoodOf(Navigator.java:227)
	at org.fxmisc.flowless.Navigator.placeStartAtMayCrop(Navigator.java:200)
	at org.fxmisc.flowless.Navigator.visit(Navigator.java:129)
	at org.fxmisc.flowless.StartOffStart.accept(TargetPosition.java:85)
	at org.fxmisc.flowless.Navigator.layoutChildren(Navigator.java:78)
	at javafx.scene.Parent.layout(Parent.java:1087)
	at org.fxmisc.flowless.VirtualFlow.layoutChildren(VirtualFlow.java:257)
	at javafx.scene.Parent.layout(Parent.java:1087)
	at javafx.scene.Parent.layout(Parent.java:1093)
	at javafx.scene.Parent.layout(Parent.java:1093)
	at javafx.scene.Parent.layout(Parent.java:1093)
	at javafx.scene.Parent.layout(Parent.java:1093)
	at javafx.scene.Scene.doLayoutPass(Scene.java:552)
	at javafx.scene.Scene.preferredSize(Scene.java:1646)
	at javafx.scene.Scene.impl_preferredSize(Scene.java:1720)
	at javafx.stage.Window$9.invalidated(Window.java:846)
	at javafx.beans.property.BooleanPropertyBase.markInvalid(BooleanPropertyBase.java:109)
	at javafx.beans.property.BooleanPropertyBase.set(BooleanPropertyBase.java:144)
	at javafx.stage.Window.setShowing(Window.java:922)
	at javafx.stage.Window.show(Window.java:937)
	at com.sun.javafx.stage.EmbeddedWindow.show(EmbeddedWindow.java:58)
	at javafx.embed.swing.JFXPanel.lambda$addNotify$49(JFXPanel.java:799)
	at com.sun.javafx.application.PlatformImpl.lambda$null$172(PlatformImpl.java:295)
	at java.security.AccessController.doPrivileged(Native Method)
	at com.sun.javafx.application.PlatformImpl.lambda$runLater$173(PlatformImpl.java:294)
	at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)

[tobiasdiez]

Yes, this seems to be a bug with the "Check existing file links" option. You can safely uncheck this option or try out Quality > Automatically set file links.

@JabRef/developers can we just remove the "Synchronize file links" action? To me this looks like auto-find together with a file existence checker. For the first part, there exists already a menu item and the second part of the feature is covered by the integrity check.

[Siedlerchr]

👍 for removing the synchronize file links. Autostart file link Already covers all

[koppor]

The dialog renders as follows:

With "auto-find" you mean the following?

The "integrity check" is something like that. koppor#282 ?

Did anyone check what "Check links" is doing? --> I would move that functionality to a second dialog and not show it on Mac OS X (make it a Windows feature only).

The following functionality is not in the check integrity thingy:

I would say that we improve our check integrity so that only file-based things can be executed. Otherwise, it takes too long for a database...

[tobiasdiez]

No "auto-link" refers the "Automatically set file links (F7)" option, which does exactly the same thing as the the first part of the sync-file links.

Yes, there is no equivalent integrity check for the last dialog. But what is this dialog even asking?! I really don't understand it.

[koppor]

"PDF" vs. mimetype pdf (application/pdf)

See #98

That means, instead of "PDF" "application/pdf" (or similar) should be used. But that was an idea of JabRef 1.x or 2.x. Maybe, this check should be dropped... Maybe we should work on the format of the file field and remove the mimetype as currently, everything works well with the file extension only. - Or do I miss something in the context of compressed files?

[Siedlerchr]

Regarding the select files dialog, it will be reworked, see my wip pr

[felker]

@tobiasdiez is correct in what I was referring to as "auto-link". As indicated in #3629 , there seems to be a problem with "Automatically set file links (F7)", so I will wait to add the links to my database.

[koppor]

DevCall decision: Remove "check links" at a separate PR. - Then, we know, which code it is and what it did. This could be a start for integrating the functionality in "check integrity".

[koppor]

@tobiasdiez We should enable to have .bib in the same directory as .pdf files. E.g., a shared drive.

DevCall decision: Reimplement Files.walk, ... -- Maybe even imperatively (AKA "traditionally") to make the code more understandable.

[felker]

Repeating my comment in #3629 here in the hopes that it is useful for others:

The "Automatically set files link" functionality seems to work well on macOS High Sierra with JabRef 3.8.2 and JDK build 1.8.0_162-b12.

My current workflow for managing my BibTex database involves switching from JabRef 3.8.2 to "Automatically set file links", to JabRef 4.1 or 4.2 to add new entries via DOI since #2879 was never patched in 3.8.2.

[charryzzz]

@tobiasdiez I have the same problem using Jabref v 4.3.1 in ubuntu 16.04.

both the bib file and PDFs are in the “main file direction”

While it works without "Check existing file links" or using the version jabref v4.0

[Siedlerchr]

Synchronize File links was removed in favor of Automatically set file links in dev-5.0.
So I am closing this issue as no longer relevant

[PierreMarchand20]

Ok, but I would like to point out that "Automatically set file links" is also not working even with dev-5.0 on macOS.

Besides, I feel like the "Automatically set file links" feature checks every file when I just want to add another file and link it (meaning that all the other files have already their links). But I could be mistaking since it crashes when I try this feature anyway.

[Siedlerchr]

@PierreMarchand20 Could you please add some details on where it crashes and if you something in the log?

[PierreMarchand20]

Well, it just hangs, so there is no log to give unfortunately. I just open my database, take a random file where I delete the link. Then I click on Quality>Automatically set file links and it freezes

[PierreMarchand20]

Ok I tried with a new database with only one entry, and this time it does not hang (so that, I really feel like it is looking at all the files in my database where I just want to update the selected entry). But it actually does not find the file (and I checked the options and it should find it).

[github-actions]

This issue has been inactive for half a year. Since JabRef is constantly evolving this issue may not be relevant any longer and it will be closed in two weeks if no further activity occurs.

As part of an effort to ensure that the JabRef team is focusing on important and valid issues, we would like to ask if you could update the issue if it still persists. This could be in the following form:

• If there has been a longer discussion, add a short summary of the most important points as a new comment (if not yet existing).
• Provide further steps or information on how to reproduce this issue.
• Upvote the initial post if you like to see it implemented soon. Votes are not the only metric that we use to determine the requests that are implemented, however, they do factor into our decision-making process.
• If all information is provided and still up-to-date, then just add a short comment that the issue is still relevant.

Thank you for your contribution!