1888-opacLink #1906

TobiasNx · 2023-09-25T12:35:34Z

This PR does the following:

Refactoring the mapping heldBy so that the ISIL of the IZ is provided, that changes the order of heldBy.isil and heldBy.id
Adding a macro for adding a opacLink as sameAs
Adding mapping files (tsv) for opacLink tusing a submodule for prod and a local copy for test
Adding sameAs for all mapped hasItem-objects

@acka47 do you have a better property than opacLink?

Outdated:
~~First draft of opac link enrichment mechanism~~

~~@blackwinter: my first design runs into an error, could you have a look at it:~~

~Error-message~

``` [main] ERROR org.lobid.resources.AlmaMarc21XmlToLobidJsonMetafixTest - Errored when transforming org.metafacture.metafix.FixProcessException: Error while executing Fix expression (at file:/home/tobias/git/lobid-resources/src/main/resources/alma/fix/macros.fix, line 267): split_field("$i.@opacLink","{hbzid}") at org.metafacture.metafix.RecordTransformer.tryRun(RecordTransformer.java:249) at org.metafacture.metafix.RecordTransformer.lambda$transform$2(RecordTransformer.java:125) at java.lang.Iterable.forEach(Iterable.java:75) at org.metafacture.metafix.RecordTransformer.transform(RecordTransformer.java:124) at org.metafacture.metafix.RecordTransformer.lambda$null$14(RecordTransformer.java:183) at org.metafacture.metafix.RecordTransformer.lambda$null$22(RecordTransformer.java:226) at org.metafacture.metafix.RecordTransformer.lambda$null$1(RecordTransformer.java:125) at org.metafacture.metafix.RecordTransformer.tryRun(RecordTransformer.java:237) at org.metafacture.metafix.RecordTransformer.lambda$transform$2(RecordTransformer.java:125) at java.lang.Iterable.forEach(Iterable.java:75) at org.metafacture.metafix.RecordTransformer.transform(RecordTransformer.java:124) at org.metafacture.metafix.RecordTransformer.transform(RecordTransformer.java:116) at org.metafacture.metafix.FixMethod$10.apply(FixMethod.java:155) at org.metafacture.metafix.RecordTransformer.lambda$null$20(RecordTransformer.java:206) at org.metafacture.metafix.RecordTransformer.lambda$null$22(RecordTransformer.java:226) at org.metafacture.metafix.RecordTransformer.lambda$null$1(RecordTransformer.java:125) at org.metafacture.metafix.RecordTransformer.tryRun(RecordTransformer.java:237) at org.metafacture.metafix.RecordTransformer.lambda$transform$2(RecordTransformer.java:125) at java.lang.Iterable.forEach(Iterable.java:75) at org.metafacture.metafix.RecordTransformer.transform(RecordTransformer.java:124) at org.metafacture.metafix.RecordTransformer.lambda$null$17(RecordTransformer.java:197) at org.metafacture.metafix.RecordTransformer.lambda$null$22(RecordTransformer.java:226) at org.metafacture.metafix.RecordTransformer.lambda$null$1(RecordTransformer.java:125) at org.metafacture.metafix.RecordTransformer.tryRun(RecordTransformer.java:237) at org.metafacture.metafix.RecordTransformer.lambda$transform$2(RecordTransformer.java:125) at java.lang.Iterable.forEach(Iterable.java:75) at org.metafacture.metafix.RecordTransformer.transform(RecordTransformer.java:124) at org.metafacture.metafix.FixBind$1.lambda$execute$1(FixBind.java:40) at org.metafacture.metafix.Value.asList(Value.java:190) at org.metafacture.metafix.Value.asList(Value.java:184) at org.metafacture.metafix.FixBind$1.execute(FixBind.java:33) at org.metafacture.metafix.RecordTransformer.lambda$null$4(RecordTransformer.java:138) at org.metafacture.metafix.RecordTransformer.lambda$null$22(RecordTransformer.java:226) at org.metafacture.metafix.RecordTransformer.lambda$null$1(RecordTransformer.java:125) at org.metafacture.metafix.RecordTransformer.tryRun(RecordTransformer.java:237) at org.metafacture.metafix.RecordTransformer.lambda$transform$2(RecordTransformer.java:125) at java.lang.Iterable.forEach(Iterable.java:75) at org.metafacture.metafix.RecordTransformer.transform(RecordTransformer.java:124) at org.metafacture.metafix.RecordTransformer.transform(RecordTransformer.java:116) at org.metafacture.metafix.FixMethod$1.apply(FixMethod.java:60) at org.metafacture.metafix.RecordTransformer.lambda$null$20(RecordTransformer.java:206) at org.metafacture.metafix.RecordTransformer.lambda$null$22(RecordTransformer.java:226) at org.metafacture.metafix.RecordTransformer.lambda$null$1(RecordTransformer.java:125) at org.metafacture.metafix.RecordTransformer.tryRun(RecordTransformer.java:237) at org.metafacture.metafix.RecordTransformer.lambda$transform$2(RecordTransformer.java:125) at java.lang.Iterable.forEach(Iterable.java:75) at org.metafacture.metafix.RecordTransformer.transform(RecordTransformer.java:124) at org.metafacture.metafix.Metafix.endRecord(Metafix.java:222) at org.metafacture.monitoring.StreamBatchLogger.endRecord(StreamBatchLogger.java:129) at org.metafacture.biblio.marc21.MarcXmlHandler.endElement(MarcXmlHandler.java:132) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1781) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2966) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:601) at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:504) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213) at org.metafacture.xml.XmlDecoder.process(XmlDecoder.java:69) at org.metafacture.xml.XmlDecoder.process(XmlDecoder.java:43) at org.metafacture.io.FileOpener.process(FileOpener.java:158) at org.metafacture.io.FileOpener.process(FileOpener.java:41) at org.metafacture.files.DirReader.dir(DirReader.java:99) at org.metafacture.files.DirReader.process(DirReader.java:75) at org.lobid.resources.AlmaMarc21XmlToLobidJsonMetafixTest.transformFile(AlmaMarc21XmlToLobidJsonMetafixTest.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) Caused by: java.util.regex.PatternSyntaxException: Illegal repetition {hbzid} at java.util.regex.Pattern.error(Pattern.java:1969) at java.util.regex.Pattern.closure(Pattern.java:3171) at java.util.regex.Pattern.sequence(Pattern.java:2148) at java.util.regex.Pattern.expr(Pattern.java:2010) at java.util.regex.Pattern.compile(Pattern.java:1702) at java.util.regex.Pattern.(Pattern.java:1352) at java.util.regex.Pattern.compile(Pattern.java:1028) at org.metafacture.metafix.FixMethod$43.apply(FixMethod.java:595) at org.metafacture.metafix.RecordTransformer.lambda$null$20(RecordTransformer.java:206) at org.metafacture.metafix.RecordTransformer.lambda$null$22(RecordTransformer.java:226) at org.metafacture.metafix.RecordTransformer.lambda$null$1(RecordTransformer.java:125) at org.metafacture.metafix.RecordTransformer.tryRun(RecordTransformer.java:237) ... 103 more Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 9.088 sec <<< FAILURE! - in UnitTests ```

blackwinter · 2023-09-25T12:39:15Z

Caused by: java.util.regex.PatternSyntaxException: Illegal repetition

The pattern in split_field is interpreted as a regular expression. You need to escape the curly braces.

TobiasNx · 2023-09-25T13:59:29Z

@blackwinter thanks.
There seems to be a problem with copy_field (but also paste) a copied and manipulated field (here @iz) within a list bind.
The copy of the copied field is not manipulated (not turned into a ISIL), therefore the new lookup for the opac link does not work.

This seems to be a fix issue. The last commit highlights the problem: 4291810

blackwinter · 2023-09-25T14:10:11Z

You're calling the opacLink macro too early; @iz contains the MMS ID at that point, not the ISIL.

Also, all maps are referencing the hbzId.tsv.

TobiasNx · 2023-09-26T13:23:53Z

@jens thanks a lot.
I am getting closer. Could you help me with the opacLinks based on zdbId they are not mapped properly.

src/test/resources/alma-fix/990156060190206441.json

src/main/resources/alma/fix/macros.fix

Co-authored-by: Jens Wille <ww@blackwinter.de>

This provides changes that can be reused for the opacUrl macro

TobiasNx · 2023-09-27T11:54:15Z

https://primo.kobv.de/primo_library/libweb/action/dlSearch.do?vid=TRI_UB_I&institution=TRI_UB_I&search_scope=localude&bulkSize=10&lang=ger&indx=1&onCampus=false&query=any,contains,HT008733617

These links seem to be outdated.

blackwinter

Looks good, thanks!

src/main/resources/alma/fix/macros.fix

src/main/resources/alma/fix/item.fix

Co-authored-by: Jens Wille <ww@blackwinter.de>

TobiasNx · 2023-09-28T13:55:18Z

There is one last change @blackwinter

src/main/resources/alma/fix/macros.fix

dr0i

putting context.jsonld is not necessary I think? Please remove it. It brings conflicts, as discussed . There is hope so - the ordering of the context!

TobiasNx · 2023-09-28T15:09:01Z

putting context.jsonld is not necessary I think? Please remove it. It brings conflicts, as discussed . There is hope so - the ordering of the context!

Since there are so many commits I am not able to kick the changes for this specific file out. @blackwinter or @dr0i are you able to help.

blackwinter · 2023-09-28T15:32:02Z

I'm not exactly sure what conflicts you're talking about, but does 7c6fb57 help?

dr0i · 2023-09-29T06:35:20Z

thx @blackwinter!

dr0i · 2023-09-29T06:36:29Z

Will be deployed next Monday.

blackwinter · 2023-10-02T08:38:53Z

This didn't work as expected. E.g. 990366624790206441:

"hasItem" : [ {
  // ...
  "seeAlso" : [ "DE-Hag4HT020469610" ],
  "id" : "http://lobid.org/items/990366624790206441:DE-Hag4:5378132900006461#!"
}, {
  // ...
  "seeAlso" : [ "DE-575HT020469610" ],
  "id" : "http://lobid.org/items/990366624790206441:DE-575:53102917670006480#!"
} ],

I couldn't find any link that was added correctly.

dr0i · 2023-10-02T10:35:06Z

Right, that was because gitsubmodules (since #1902 ) didnt't update the isil2opac (hbz/lookup-tables@19c5676). I've restarted the indexing, should be deployed tomorrow.

blackwinter · 2023-10-02T10:40:12Z

Caused by missing file map: lookup() calls map.get() which tries FileMap.init() leading to an exception every time. This then leaves the looked up value untouched even though delete: "true" was specified.

TobiasNx added 2 commits September 25, 2023 14:15

Intoduce a variable for the ALMA IZ #1888

225091b

Draft opac link lookup #1888

ce0346b

TobiasNx assigned blackwinter Sep 25, 2023

blackwinter assigned TobiasNx and unassigned blackwinter Sep 25, 2023

Adjust opac link lookup #1888

c2e348b

Call the macro later in item.fix #1888

d0e3745

TobiasNx force-pushed the 1888-opacLink branch from 4291810 to d0e3745 Compare September 25, 2023 15:11

TobiasNx added 3 commits September 26, 2023 14:52

Merge remote-tracking branch 'origin/master' into 1888-opacLink

b32844a

Fix variables for opac link maps #1888

efd9ad1

Fix split_field sep_char #1888

c34749e

TobiasNx commented Sep 26, 2023

View reviewed changes

src/test/resources/alma-fix/990156060190206441.json Outdated Show resolved Hide resolved

TobiasNx added 3 commits September 27, 2023 08:16

Test if zdbId exists before mapping opacLinkZdbId #1888

94b2c6c

Only use hbzId opac link for non ZDB resources #1888

a6140b3

Introduce opac links for isbn and issn #1888

1a21c77

blackwinter requested changes Sep 27, 2023

View reviewed changes

src/main/resources/alma/fix/macros.fix Outdated Show resolved Hide resolved

src/main/resources/alma/fix/macros.fix Outdated Show resolved Hide resolved

TobiasNx and others added 6 commits September 27, 2023 12:11

Update src/main/resources/alma/fix/macros.fix

4f05f37

Co-authored-by: Jens Wille <ww@blackwinter.de>

Update src/main/resources/alma/fix/macros.fix

ed40503

Co-authored-by: Jens Wille <ww@blackwinter.de>

Add field variable to macro and change position of call_macro #1888

43e21f4

Prefer sublibrary opac over IZ opac #1888

cd3547b

Refacture heldBy mapping #1888

515d577

This provides changes that can be reused for the opacUrl macro

Add opac links to all non-physical resources #1888

27c7d49

TobiasNx mentioned this pull request Sep 27, 2023

Cache table "isil2opac_hbzid" #1911

Closed

TobiasNx marked this pull request as ready for review September 27, 2023 12:02

TobiasNx requested a review from blackwinter September 27, 2023 12:09

blackwinter approved these changes Sep 27, 2023

View reviewed changes

Change property name to seeAlso #1888

a415cd8

blackwinter requested changes Sep 28, 2023

View reviewed changes

src/main/resources/alma/fix/macros.fix Show resolved Hide resolved

src/main/resources/alma/fix/item.fix Outdated Show resolved Hide resolved

src/main/resources/alma/fix/item.fix Outdated Show resolved Hide resolved

dr0i removed their request for review September 28, 2023 13:30

Update src/main/resources/alma/fix/item.fix

5599f3d

Co-authored-by: Jens Wille <ww@blackwinter.de>

TobiasNx force-pushed the 1888-opacLink branch from 6e71917 to fadcd15 Compare September 28, 2023 13:34

Fix forgotten change in property name #1888

dca4580

TobiasNx force-pushed the 1888-opacLink branch from fadcd15 to dca4580 Compare September 28, 2023 13:39

TobiasNx added 2 commits September 28, 2023 15:45

Use local maps in test and submodul in prod #1888

20c01d4

Update opac maps #1888

adbda88

blackwinter approved these changes Sep 28, 2023

View reviewed changes

Add lookup for opac links with almaMmsId #1888

fcd7604

blackwinter reviewed Sep 28, 2023

View reviewed changes

src/main/resources/alma/fix/macros.fix Outdated Show resolved Hide resolved

Fix almaMmsId Opac Link #1888

3e4ea81

TobiasNx requested review from dr0i and removed request for acka47 September 28, 2023 14:23

dr0i requested changes Sep 28, 2023

View reviewed changes

Merge remote-tracking branch 'origin/master' into 1888-opacLink

d553266

Reset context.jsonld. (#1906)

7c6fb57

dr0i merged commit 03720ef into master Sep 29, 2023
1 check passed

dr0i deleted the 1888-opacLink branch September 29, 2023 06:35

blackwinter mentioned this pull request Oct 2, 2023

FileMap with missing file(s) should optionally behave as if empty. metafacture/metafacture-core#500

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1888-opacLink #1906

1888-opacLink #1906

TobiasNx commented Sep 25, 2023 •

edited

Loading

blackwinter commented Sep 25, 2023

TobiasNx commented Sep 25, 2023 •

edited

Loading

blackwinter commented Sep 25, 2023

TobiasNx commented Sep 26, 2023

TobiasNx commented Sep 27, 2023

blackwinter left a comment

TobiasNx commented Sep 28, 2023

dr0i left a comment

TobiasNx commented Sep 28, 2023

blackwinter commented Sep 28, 2023

dr0i commented Sep 29, 2023

dr0i commented Sep 29, 2023

blackwinter commented Oct 2, 2023

dr0i commented Oct 2, 2023

blackwinter commented Oct 2, 2023

1888-opacLink #1906

1888-opacLink #1906

Conversation

TobiasNx commented Sep 25, 2023 • edited Loading

blackwinter commented Sep 25, 2023

TobiasNx commented Sep 25, 2023 • edited Loading

blackwinter commented Sep 25, 2023

TobiasNx commented Sep 26, 2023

TobiasNx commented Sep 27, 2023

blackwinter left a comment

Choose a reason for hiding this comment

TobiasNx commented Sep 28, 2023

dr0i left a comment

Choose a reason for hiding this comment

TobiasNx commented Sep 28, 2023

blackwinter commented Sep 28, 2023

dr0i commented Sep 29, 2023

dr0i commented Sep 29, 2023

blackwinter commented Oct 2, 2023

dr0i commented Oct 2, 2023

blackwinter commented Oct 2, 2023

TobiasNx commented Sep 25, 2023 •

edited

Loading

TobiasNx commented Sep 25, 2023 •

edited

Loading