`decode-json` + `catch-object-exception` + `open-http` turns something into an error which is none

[TobiasNx]

https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-lines%0A%7C+catch-object-exception+%0A%7C+open-http%0A%7C+as-records%0A%7C+decode-json%0A%7C+list-fix-values%28%22type%5B%5D%22%29%0A%7C+print%0A%3B%0A&data=https%3A//lobid.org/resources/1188059580.json%0Ahttps%3A//lobid.org/resources/HT020467687.json

file content, first link provides no json but a Error message second does and is not broken:

https://lobid.org/resources/1188059580.json
https://lobid.org/resources/HT020467687.json

inputFile
| open-file
| as-lines
| catch-object-exception 
| open-http
| as-records
| decode-json
| list-fix-values("type[]")
| print
;

When running it on the command line the following exception with two erros returns:

$ '/home/tobias/Downloads/metafix-runner-1.1.2/bin/metafix-runner' '/home/tobias/git/metafacture_workflows/rpbArticleRelatiedTypeInLobid/lobidTypeTesterForRpb.flux'
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/tobias/Downloads/metafix-runner-1.1.2/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/tobias/Downloads/metafix-runner-1.1.2/lib/slf4j-simple-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
0    [main] ERROR .flowcontrol.ObjectExceptionCatcher  - 'com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'ERROR': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"ERROR: "Not found: 1188059580""; line: 1, column: 6]' while processing object: https://lobid.org/resources/1188059580.json
114  [main] ERROR .flowcontrol.ObjectExceptionCatcher  - 'com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'ERROR': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"ERROR: "Not found: 1188059580"{
  "@context" : "http://lobid.org/resources/context.jsonld",
  "id" : "http://lobid.org/resources/990366605490206441#!",
  "type" : [ "BibliographicResource", "Book" ],
  "medium" : [ {
    "label" : "Print",
    "id" : "http://rdaregistry.info/termList/RDAproductionMethod/1010"
  } ],
  "title" : "Weinfreunde Marx & Engels",
  "almaMmsId" : "990366605490206441",
  "hbzId" : "HT020467687",
  "deprecatedUri" : "http://lobid.org/resources/HT020467687#!",
  "isbn" : ["[truncated 14186 chars]; line: 1, column: 6]' while processing object: https://lobid.org/resources/HT020467687.json
0       |

If the exception catcher is outcommented the process only returns one error:

Exception in thread "main" org.metafacture.framework.MetafactureException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'ERROR': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"ERROR: "Not found: 1188059580""; line: 1, column: 6]
        at org.metafacture.json.JsonDecoder.processRecord(JsonDecoder.java:250)
        at org.metafacture.json.JsonDecoder.process(JsonDecoder.java:235)
        at org.metafacture.json.JsonDecoder.process(JsonDecoder.java:45)
        at org.metafacture.io.RecordReader.emitRecord(RecordReader.java:144)
        at org.metafacture.io.RecordReader.process(RecordReader.java:132)
        at org.metafacture.io.RecordReader.process(RecordReader.java:40)
        at org.metafacture.io.HttpOpener.process(HttpOpener.java:306)
        at org.metafacture.io.HttpOpener.process(HttpOpener.java:50)
        at org.metafacture.io.LineReader.process(LineReader.java:69)
        at org.metafacture.io.LineReader.process(LineReader.java:55)
        at org.metafacture.io.LineReader.process(LineReader.java:37)
        at org.metafacture.io.FileOpener.process(FileOpener.java:158)
        at org.metafacture.io.FileOpener.process(FileOpener.java:41)
        at org.metafacture.flux.parser.StringSender.process(StringSender.java:43)
        at org.metafacture.flux.parser.Flow.start(Flow.java:118)
        at org.metafacture.flux.parser.FluxProgramm.start(FluxProgramm.java:168)
        at org.metafacture.runner.Flux.main(Flux.java:87)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'ERROR': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"ERROR: "Not found: 1188059580""; line: 1, column: 6]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2477)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:760)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:3041)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2082)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:808)
        at org.metafacture.json.JsonDecoder.decode(JsonDecoder.java:294)
        at org.metafacture.json.JsonDecoder.processRecord(JsonDecoder.java:247)
        ... 16 more

When only using the second link, the transformation works.

[TobiasNx]

Ha, I moved the exception catcher in front of decode-json, then it works.

Could the exception catcher not properly end its job after open-http?

[dr0i]

(I had problems to understand your issue, but I will give it a try.)
Your provided links don't work. Seems you have missed %0A at the end of the link , as this works:

https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-lines%0A%7C+catch-object-exception+%0A%7C+open-http%0A%7C+as-records%0A%7C+decode-json%0A%7C+list-fix-values%28%22type%5B%5D%22%29%0A%7C+print%0A%3B%0A&data=https%3A//lobid.org/resources/1188059580.json%0Ahttps%3A//lobid.org/resources/HT020467687.json%0A

The result when query 1188059580.json is empty (as you've written) but no error message is given (at least in the playground it is not). There has to be some kind of error, though, because the input resource ("https://lobid.org/resources/1188059580.json") does not exist and the webserver answers with an exception "Not found: 1188059580" and this is structurally no JSON so cannot be parsed by a JSON parser.

The thing that's smelly IMO is your log message:

"Not found: 1188059580"{
"@context" : "http://lobid.org/resources/context.jsonld",

The output of the resource (starting with @context) does not belong to the first output. At the very least a line break is missing. This is done here .

[blackwinter]

The issue here is that the wrong input is logged by ObjectExceptionCatcher (catch-object-exception) as the culprit (concatenation of both inputs) due to RecordReader (as-records) not clearing its internal buffer (builder) in case of a downstream exception. When moving the catch statement further down, the reader is no longer involved.

The fix appears to be trivial and is submitted in #717.

Side note: The RecordReader is technically not necessary here since it doesn't actually "split the input into individual records". All it does is convert the Reader from HttpOpener into a String for JsonDecoder. Unfortunately, there is no dedicated module for this conversion (Reader -> String), only for the inverse direction (String -> Reader = StringReader).

[blackwinter]

Functional review: @TobiasNx
Code review: @dr0i

[TobiasNx]

Side note: The RecordReader is technically not necessary here since it doesn't actually "split the input into individual records". All it does is convert the Reader from HttpOpener into a String for JsonDecoder. Unfortunately, there is no dedicated module for this conversion (Reader -> String), only for the inverse direction (String -> Reader = StringReader).

I thought that as-lines and as-records are the Reader -> String Modules. I always think that as-batch would be a better name for as-records since it does not provide a line by line transformation but a multi-line string that maybe is separated into individual records by a record separator.

[blackwinter]

@TobiasNx: You reacted with a thumbs up to my previous comment. Does this mean you approve the fix? If so, please assign me.

[TobiasNx]

Not yet. But I will test today!

[TobiasNx]

+1 tested it! Also varied the input.