Loading large database takes way too long after commit 5.7--2022-08-01--8389254

[amaxi99]

JabRef version

Latest development branch build (please note build date below)

Operating system

Windows

Details on version and operating system

Windows 11 21H2

Checked with the latest development build

• I made a backup of my libraries before testing the latest development version.
• I have tested the latest development version and the problem persists

Steps to reproduce the behaviour

Opening a large database with 15.000 entries (22MB file size) takes about 3 minutes in current development versions. In versions before August 1 it took about 4 s to open the same file (until version JabRef 5.7--2022-08-01--8389254).

Appendix

...

Log File

Paste an excerpt of your log file here

[ThiloteE]

meta-issue: #8906

[Siedlerchr]

confirmed. Guess the root issue is this method which is executed for each entry

jabref/src/main/java/org/jabref/model/database/BibDatabase.java

Lines 602 to 611 in 0d82506

	private void updateVisibleFields() {
	visibleFields.clear();
	Set<Field> allFields = new TreeSet<>(Comparator.comparing(Field::getName));
	for (BibEntry e : getEntries()) {
	allFields.addAll(e.getFields());
	}
	visibleFields.addAll(allFields.stream().filter(field -> !FieldFactory.isInternalField(field))
	.filter(field -> StringUtil.isNotBlank(field.getName()))
	.collect(Collectors.toSet()));
	}

and is costly
It was introduced by the AutomaticFieldEditor

[HoussemNasri]

I profiled the app and I can confirm that updateVisibleFields is responsible for the performance downgrade. While working on the automatic field editor, I made the false assumption that loading a library would use BibDatabase#insertEntries once, which is why updateVisibleFields was called when inserting entries, but the actual implementation inserts an entry as soon as it is parsed.

The good news is that updateVisibleFields is not even required. The automatic field editor dialog was designed to automatically update its list of visible fields whenever a new field is added, but due to numerous bugs that popped up, we now only do so when the dialog is opened.

I suggest that we go back to the old implementation. We'll lose the ability to observe changes in visible fields, but since it affects the performance I think it is worthwhile. The fix can be as easy as copy-pasting the BibDatabase class state before updateVisibleFields was introduced.

https://github.com/JabRef/jabref/blob/be4a7c55c669dae4279bea6562a6c9795adada37/src/main/java/org/jabref/model/database/BibDatabase.java