Rewrite MedlineImporter with JAXB and add nbib fields #1479

tschechlovdev · 2016-06-06T11:01:49Z

Regarding:#1063
I've rewritten the MedlineImporter (to be more specific: the importDatabase method), so that it uses a JAXB parser now, instead of a SAX parser. I've also added the nbib fields to the importer. Tests for the changes will be coming.

Change in CHANGELOG.md described
Tests created for changes

boceckts · 2016-06-27T14:10:31Z

CHANGELOG.md

@@ -73,7 +73,7 @@ We refer to [GitHub issues](https://github.com/JabRef/jabref/issues) by using `#
 - Integrity check table can be sorted by clicking on column headings
 - Added \SOFTWARE\Jabref 'Path' registry entry for installation path inside the installer
 - Added an additional icon to distinguish DOI and URL links ([feature request #696](https://github.com/JabRef/jabref/issues/696))
- Added nbib fields to Medlineplain importer
+- Added nbib fields to Medlineplain importer and to MedlineImpporter


boceckts · 2016-06-27T17:19:51Z

Very good work 👍, only a few remarks from my side.

koppor · 2016-06-28T08:12:36Z

src/test/resources/net/sf/jabref/importer/fileformat/MedlineImporterTestSingleEntry.xml

@@ -1,7 +1,7 @@
-<PubmedArticleSet>


Why was this removed?

Just to test the case, that PubMedArticleSet doesn't appear in a file.

Why not adding a second test case? - Modified test files always cause a strange feeling

koppor · 2016-06-28T08:13:43Z

Please do not include the generated classes. Try to use https://github.com/jacobono/gradle-jaxb-plugin to generate them. Does that work?

matthiasgeiger · 2016-06-29T12:28:08Z

Can you please document the source of the XSD you used to create the JAXB classes. Thanks!

tschechlovdev · 2016-07-01T21:57:22Z

@koppor I didn't get the plugin work. But I've wrote an ant task, that I've found here. This works local, but for some reason travis ci says, that src/main/gen is a non existend directory.

@matthiasgeiger I wrote the source as a comment in the xsd file(s), is that ok ?

…o nBibXmlImporter

tschechlovdev · 2016-07-16T19:52:56Z

@matthiasgeiger now it should work. I'm not sure why, but somehow the first two lines
`

`

were causing trouble. My solution is to skip the first lines, till we reach the first line, that contains , , or .

koppor · 2016-07-17T11:00:01Z

XML parsers should do that work. Do not walk through XML by hand.

Please remove this method and try the solution presented at http://stackoverflow.com/a/22204172/873282

xsr.next(); from XMLStreamReader seems to be your friend.

koppor · 2016-07-17T21:31:47Z

src/main/java/net/sf/jabref/importer/fileformat/MedlineImporter.java

+        ArrayList<String> keywords = new ArrayList<>();
+        String result = "";
+        for (MeshHeading keyword : meshHeadingList.getMeshHeading()) {
+            result = keyword.getDescriptorName().getContent();


Declare result variable here, not out of the loop

koppor · 2016-07-17T21:45:07Z

Other than my small comments: looks good (codewise). Haven't tried the UI.

koppor · 2016-07-17T22:25:05Z

src/main/java/net/sf/jabref/importer/fileformat/MedlineImporter.java

+        } catch (JAXBException e1) {
+            LOGGER.debug("could not parse document", e1);
+            return ParserResult.fromErrorMessage(e1.getLocalizedMessage());
+        } catch (XMLStreamException e) {


combine these two exeptions by JAXBException | XMLStreamExeption e

…o nBibXmlImporter

tschechlovdev · 2016-07-19T20:09:03Z

I've adressed the comments. Are there any other remarks?

koppor · 2016-07-19T21:44:40Z

LGTM

tschechlovdev · 2016-07-21T09:39:07Z

So can this be merged?

* master: (268 commits) Added DateFormatter to LayoutEntry so that it actually works... (JabRef#1619) Implemented JabRef#1345 - cleanup ISSN (JabRef#1590) Always use https for help files (JabRef#1615) Resolves JabRef#1613 Use Jabref default icon for uninstaller for now (JabRef#1616) Added more fields and fixed some issues (JabRef#1617) Added LayoutFormatterPreferences (and related files) (JabRef#1608) [WIP] Create new fetcher infrastructure (JabRef#1594) Set user agent to fix 403 status error More fields added to FieldName (JabRef#1614) Added model.entry.FieldName that contains field name constants (JabRef#1602) Fixes imports Test CustomImporter (JabRef#1501) The field list gets the focus as soon as it is focused (JabRef#1541) When clicking on a tab, the first field now has the focus (JabRef#988) Add test in BibEntryWriterTest for type change Rewrite MedlineImporter with JAXB and add nbib fields (JabRef#1479) Some Globals.prefs injection in logic and model (JabRef#1593) Added filter to not show selected integrity checks (JabRef#1588) Replace getField with getFieldOptional in all of the tests and in som… (JabRef#1591) Move preferences (JabRef#1604) ... # Conflicts: # src/main/java/net/sf/jabref/importer/ImportMenuItem.java # src/main/java/net/sf/jabref/importer/OpenDatabaseAction.java

@comment

Move event (JabRef#1601) * Move event package to model Update dependencies: postgres 9.4.1208 -> 9.4.1209 and wiremock from 2.1.6 to 2.1.7 Added ISBN integrity checker (JabRef#1586) Added ISBN integrity checker * Extracted ISBN class Reenable errorprone (see http://errorprone.info/) Extend the OpenConsoleFeature (JabRef#1582) * Extend the OpenConsoleFeature by selection of custom terminal emulator. - Add radio selection to the AdvancedTab - Add new JabRefPreferences - Add file check and execution commands - Add localization keys * Fix localization key. * Move console selection to ExternalTab.java * Change localization entry. * Add command executor. * Fix placeholder replacement. * Fix codacy. * Update localization key. * Remove "Specify terminal emulator" option. Add GUI outputs. * Set default command for Windows. Fix localization entries. * Remove empty lines in language files. * Use lambda expressions insead of ActionListeners * Refactoring. * Update CHANGELOG.md. * Small refactorings. Move preferences (JabRef#1604) * Move preferences-related classes into separate package * Rename JabRefPreferencesFilterDialog -> PreferencesFilterDialog and move it to gui * Fix checkstyle warning Set user agent to fix 403 status error Replace getField with getFieldOptional in all of the tests and in som… (JabRef#1591) * Replace getField with getFieldOptional in all of the tests and in some more code * Some more conversions Added filter to not show selected integrity checks (JabRef#1588) * Added filter to not show selected integrity checks * Removed unused variable Some Globals.prefs injection in logic and model (JabRef#1593) * Some Globals.prefs injection in logic and model * Some more conversions and some fixes * More injections * Even more injections * Yes, even more injections * Indeed, even more injections * Probably the last injections for now * Removed unrequired dependency and fixed issue * Dropped support for selecting sub/super to equations * Added preference classes for LatexFieldFormatter and FieldContentParser * Removed some left over code * Added JournalAbbreviationPreferences * Encapsulated LatexFieldFormatterPreferences in SavePreferences * Changed getShortDescription to accept boolean argument * Removed Globals.prefs from tests and removed unused imports * Unused import * Unused import Rewrite MedlineImporter with JAXB and add nbib fields (JabRef#1479) Add test in BibEntryWriterTest for type change When clicking on a tab, the first field now has the focus (JabRef#988) * the first Field does now have focus when clicking on a tab in the entry editor * Make first field focused when selecting a tab in entry editor The field list gets the focus as soon as it is focused (JabRef#1541) Test CustomImporter (JabRef#1501) * Test CustomImporter Fixes imports Added model.entry.FieldName that contains field name constants (JabRef#1602) * Added model.entry.FieldName that contains field name constants * More constants * Renamed and added more constants * Some more fields and cleanups * Removed MedlineHandler left from merge conflicts More fields added to FieldName (JabRef#1614) * More fields added to FieldName * Some Medline fixes [WIP] Create new fetcher infrastructure (JabRef#1594) * Introduce new Fetcher interfaces * Refactor arXiv fulltext fetcher * Add query based arXiv fetcher * Reformat code * Add a few tests for the arxiv parser * Make new arXiv fetcher available * Fix small problems related to files * Fix tests * Rename interface methods * Add changelog entry * Mark old EntryFetcher interface as deprecated * Move fetcher to importer \ fetcher * Move HelpFile from gui.help to logic.help * Rename fetchers * Rename FulltextFinder * Optimize imports * Fix failing test * Ignore failing test Added LayoutFormatterPreferences (and related files) (JabRef#1608) * Added LayoutFormatterPreferences (and related files) * Rebased * Included JournalAbbreviationLoader in LayoutPreferences Added more fields and fixed some issues (JabRef#1617) Resolves JabRef#1613 Use Jabref default icon for uninstaller for now (JabRef#1616) Always use https for help files (JabRef#1615) Implemented JabRef#1345 - cleanup ISSN (JabRef#1590) * Implemented JabRef#1345 - cleanup ISSN * Fixed comments * Extracted ISSN class * Added tests for ISSN and ISBN Added DateFormatter to LayoutEntry so that it actually works... (JabRef#1619) Converted a few getField to getFieldOptional (JabRef#1625) * Converted a few getField to getFieldOptional Fixed JabRef#636 by using DOICheck and DOIStrip in export filters (JabRef#1624) Improved LaTeX to Unicode/HTML formatters to output more sensible values for unknown commands (JabRef#1622) Updated preview entries (JabRef#1606) * Updated preview entries, which return new entry Moved, removed, and used String constants (JabRef#1618) * Moved, removed, and used String constants * Some more fixes * Moved NEWLINE, made FILE_SEPARATOR public and used it * Moved NEWLINE and FILE_SEPARATOR to OS * Moved ENCODING_PREFIX and SIGNATURE * Corrected Globals in a few comments... * Apparently the localization tests find commented out text... More field names and a method (JabRef#1627) * Introduced FieldName in ArXiV * Some more field names * More field names Cleanup FindFile and asssociated tests (JabRef#1596) * Cleanup FindFile and rework it using Streams and nio methods- * Unignore test for trying on CI * Use explicit List and Set in findFiles and caller methods * Use Lazy Stream to find files changes should be tested manually Some enhancements and cleanups related to dates (JabRef#1575) * Some enhancements and cleanups related to dates * Fixed some time zone issues * Replaced SimpleDateFormat in ZipFileChooser and replaced arrays with Lists * Changed EasyDateFormat constructors * Fixed stupid mistake * Added CHANGELOG entry * Maybe tests are passing now? * Some server side print debugging... * As it should be * Tryng LocalDateTime * No time zone * Added test for Cookie * Fixed imports... * Added a third possible date format as it turns out that the server changed while developing this PR Builds are now stored via build-upload.jabref.org Consistent file name casing (and other localization improvements) (JabRef#1629) * AUX files * ZIP files * BIB files * JAR files * didn't * Couldn't what's * Consistent casing * AUX apparently is commonly used in French words... * Fixed the flawed quick-and-dirty find-and-replace failures define xjc input/ouput dir (subsequent builds will be faster) (JabRef#1628) Execute task only when input/output dir changed. Fixed a minor issue and refactored MergeEntries (JabRef#1634) * Fixed a minor issue and refactored MergeEntries * Fixed import * Added CHANGELOG entry Added LabelPatternPreferences (JabRef#1607) * Added LabelPatternPreferences * Removed static initializer More tests (JabRef#1635) * Added more tests for Cookie * Enabled some layout tests and added test for StringUtil.intValueOfWithNull * Updated a test * Split tests Updated Errorprone to 2.0.11 (JabRef#1636) * Updated Errorprone to 2.0.11 * Corrected test Keep @comment text in a bib file (JabRef#1638) * Kep @comment text in a bib file * Add test for @comment that contains regular entries Replaced some getField and fixed some bugs (JabRef#1631) * Replaced some getField and fixed some bugs * Fixed a few things * Added CHANGELOG entries * Improved equals implementation * Text book equals and hashCode Fixed JabRef#1639 (JabRef#1641) * Fixed JabRef#1639 * Removed old code Export OO/LO citations to new database (JabRef#1630) * Export OO/LO citations to new database * Fixed problem with duplicates * Added some comments * Fixed spelling in comment * Removed general Exception Unified some equals (JabRef#1640) * Unified some equals * Imported correct Objects... Fixed one more NPE which should have been fixed in JabRef#1631 (JabRef#1649) Finished method to hide visible fields and show hidden fields - Hide method done - Show method done - ToDo repaint hidden field - ToDo test class finished field repaint remove sysouts

tschechlovdev added the stupro label Jun 6, 2016

tschechlovdev force-pushed the nBibXmlImporter branch from 6c58c13 to 2a0b1b0 Compare June 6, 2016 11:07

koppor mentioned this pull request Jun 6, 2016

Import: only one entry imported from a pubmed-export-file #1267

Closed

tschechlovdev changed the title ~~[WIP] Rewrite MedlineImporter with Staxparser and add nbib fields~~ [WIP] Rewrite MedlineImporter with JAXB and add nbib fields Jun 13, 2016

tschechlovdev force-pushed the nBibXmlImporter branch from b9dacf2 to 2bd9612 Compare June 26, 2016 13:25

tschechlovdev added the stupro-ready-for-internal-review label Jun 26, 2016

boceckts reviewed Jun 27, 2016
View reviewed changes

boceckts added status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers and removed stupro-ready-for-internal-review labels Jun 27, 2016

boceckts changed the title ~~[WIP] Rewrite MedlineImporter with JAXB and add nbib fields~~ Rewrite MedlineImporter with JAXB and add nbib fields Jun 27, 2016

koppor reviewed Jun 28, 2016
View reviewed changes

koppor removed the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Jun 28, 2016

tschechlovdev force-pushed the nBibXmlImporter branch from 0b7554f to 9a36bd3 Compare July 1, 2016 20:44

tschechlovdev force-pushed the nBibXmlImporter branch from 3b25f13 to ba4e617 Compare July 3, 2016 18:54

tschechlovdev changed the title ~~Rewrite MedlineImporter with JAXB and add nbib fields~~ [WIP]Rewrite MedlineImporter with JAXB and add nbib fields Jul 3, 2016

rewrite with JAXB and add nbib fields

e96133b

tschechlovdev added 2 commits July 16, 2016 19:01

ignore first unnecessary lines

631106e

Merge branch 'nBibXmlImporter' of github.com:tschechlovdev/jabref int…

984e21c

…o nBibXmlImporter

koppor mentioned this pull request Jul 17, 2016

Plugin is very slow on some pages Justineo/github-hovercard#32

Closed

use XMLStreamReader to go to the root element

e3a0d3c

koppor reviewed Jul 17, 2016
View reviewed changes

tschechlovdev added 4 commits July 18, 2016 17:34

address the comments

f380852

address the comments

204cc2d

Merge branch 'nBibXmlImporter' of github.com:tschechlovdev/jabref int…

2aae23a

…o nBibXmlImporter

minor fixes

9c50193

tschechlovdev added the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Jul 18, 2016

tschechlovdev changed the title ~~[WIP]Rewrite MedlineImporter with JAXB and add nbib fields~~ Rewrite MedlineImporter with JAXB and add nbib fields Jul 18, 2016

koppor merged commit 6fb9251 into JabRef:master Jul 21, 2016

tschechlovdev deleted the nBibXmlImporter branch July 21, 2016 10:50

tobiasdiez mentioned this pull request Oct 3, 2016

Medline search throws Exception and hangs (Jabref 3.6) #2110

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite MedlineImporter with JAXB and add nbib fields #1479

Rewrite MedlineImporter with JAXB and add nbib fields #1479

tschechlovdev commented Jun 6, 2016 •

edited by koppor

Loading

boceckts Jun 27, 2016

boceckts commented Jun 27, 2016

koppor Jun 28, 2016

tschechlovdev Jun 28, 2016

koppor Jul 17, 2016

koppor commented Jun 28, 2016

matthiasgeiger commented Jun 29, 2016

tschechlovdev commented Jul 1, 2016

tschechlovdev commented Jul 16, 2016

koppor commented Jul 17, 2016

koppor Jul 17, 2016

koppor commented Jul 17, 2016

koppor Jul 17, 2016

tschechlovdev commented Jul 19, 2016

koppor commented Jul 19, 2016

tschechlovdev commented Jul 21, 2016

Rewrite MedlineImporter with JAXB and add nbib fields #1479

Rewrite MedlineImporter with JAXB and add nbib fields #1479

Conversation

tschechlovdev commented Jun 6, 2016 • edited by koppor Loading

boceckts Jun 27, 2016

Choose a reason for hiding this comment

boceckts commented Jun 27, 2016

koppor Jun 28, 2016

Choose a reason for hiding this comment

tschechlovdev Jun 28, 2016

Choose a reason for hiding this comment

koppor Jul 17, 2016

Choose a reason for hiding this comment

koppor commented Jun 28, 2016

matthiasgeiger commented Jun 29, 2016

tschechlovdev commented Jul 1, 2016

tschechlovdev commented Jul 16, 2016

koppor commented Jul 17, 2016

koppor Jul 17, 2016

Choose a reason for hiding this comment

koppor commented Jul 17, 2016

koppor Jul 17, 2016

Choose a reason for hiding this comment

tschechlovdev commented Jul 19, 2016

koppor commented Jul 19, 2016

tschechlovdev commented Jul 21, 2016

tschechlovdev commented Jun 6, 2016 •

edited by koppor

Loading