Pass data through Metamorph

[cboehme]

Metamorph should be able to pass entities and literals through even if they are not processed.

[dr0i]

There is metamorph's _else, but I guess you mean "something else" ;)

[cboehme]

Yes, I meant something that could also pass whole entities and not only literals. Currently, it is not possible to easily write a script that simply changes some literals or entities in a record while passing the remainder of a record untouched. This makes it quite difficult to write script to filter some data in a record without completely rewriting it.

[dr0i]

Seems related to an email from February 2015, which also provides more details.

[blackwinter]

Just an FYI in reference to PR #328 (not really sure where to discuss): You could achieve (almost) the same result with a filter (org.metafacture.metamorph.Filter) and _else in combination with an unnamed entity [CORRECTION: entity is not required, we're only using it for the (optional) if] (cf. hbz.limetrans.filter.LibraryMetadataFilter).

@Test
public void metamorph1_passthrough() {
    final Filter metamorph = new Filter(InlineMorph.in(this) //
            .with("<rules>")//
            .with("    <data source='_else'/>")//
            .with("</rules>")//
            .create());
    metamorph.setReceiver(receiver);

    metamorph.startRecord("1");
    metamorph.startEntity("clone");
    metamorph.literal("id", "0");
    metamorph.endEntity();
    metamorph.startEntity("clone");
    metamorph.literal("id", "1");
    metamorph.endEntity();
    metamorph.endRecord();

    final InOrder ordered = inOrder(receiver);
    ordered.verify(receiver).startRecord("1");
    ordered.verify(receiver).startEntity("clone");
    ordered.verify(receiver).literal("id", "0");
    ordered.verify(receiver).endEntity();
    ordered.verify(receiver).startEntity("clone");
    ordered.verify(receiver).literal("id", "1");
    ordered.verify(receiver).endEntity();
    ordered.verify(receiver).endRecord();
}

The only difference is that literals are emitted with their base name only, not with the full entity path (id instead of clone.id), which seems more in line with the usual (pre-1.1) mappings.

[hagbeck]

Sorry, but this isn't working anymore! I've pulled the branch and the resulting dist produces

<marc:controlfield tag="leader">01339nmm a2200024 c 4500</marc:controlfield>
<marc:controlfield tag="001">1635091</marc:controlfield>
<marc:controlfield tag="005">20180914</marc:controlfield>
<marc:controlfield tag="007">cr |||||||||||</marc:controlfield>
<marc:controlfield tag="008">171117s2017uuuu|||||| |o|||||||||||ger||</marc:controlfield>
<marc:controlfield tag="020 .a">9783662532607</marc:controlfield>
<marc:controlfield tag="0247.2">doi</marc:controlfield>
<marc:controlfield tag="0247.a">10.1007/978-3-662-53260-7</marc:controlfield>
<marc:controlfield tag="035 .a">(UNION_SEAL)HT019079275</marc:controlfield>
<marc:controlfield tag="035 .a">(DE-599)HBZHT019079275</marc:controlfield>
<marc:controlfield tag="040 .e">rda</marc:controlfield>
<marc:controlfield tag="24500a">Weißbuch Gelenkersatz</marc:controlfield>
<marc:controlfield tag="24500b">Versorgungssituation bei endoprothetischen Hüft- und Knieoperationen in Deutschland</marc:controlfield>
<marc:controlfield tag="24500c">herausgegeben von H.-H. Bleß, M. Kip</marc:controlfield>
<marc:controlfield tag="260 .a">Berlin, Heidelberg</marc:controlfield>
<marc:controlfield tag="260 .b">Springer Berlin Heidelberg</marc:controlfield>
<marc:controlfield tag="260 .c">2017</marc:controlfield>
<marc:controlfield tag="260 .b">Imprint: Springer</marc:controlfield>
<marc:controlfield tag="2641.a">Berlin, Heidelberg</marc:controlfield>
<marc:controlfield tag="2641.b">Springer Berlin Heidelberg</marc:controlfield>
<marc:controlfield tag="2641.c">2017</marc:controlfield>

The morph I've used

<?xml version="1.0" encoding="UTF-8"?>
<metamorph xmlns="http://www.culturegraph.org/metamorph"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.culturegraph.org/metamorph metamorph.xsd"
           version="1">

     <!-- Metadata -->
    <meta />

    <!-- Macro definitions -->
    <macros />

    <!-- Transformation rules-->
    <rules>
        <data source="_else"/>
    </rules>

    <maps />

</metamorph>

And the flux

"infile.xml"|
open-file|
decode-xml|
handle-marcxml|
morph(FLUX_DIR + "morph1.xml", *)|
encode-marcxml|
write( "output.xml");

[blackwinter]

@hagbeck: It's caused by this particular change in the branch:

-            writeRaw(String.format(CONTROLFIELD_OPEN_TEMPLATE, name));
+            writeRaw(String.format(CONTROLFIELD_OPEN_TEMPLATE, name.replaceFirst("\\W","")));

[dr0i]

@hagbeck for 100% backward compatibility a meta.version was introduced.
The default is to behave like with the original Metamorph - this is what your results reflect.

To use the Metamorph 1.1 you have to put a meta.version into morph with value "1.1". I opened a PR (hagbeck/metafacture-sandbox#4) so you can see how it's done.

[dr0i]

@blackwinter oh, thx, wasn't aware of this "Filter" ... Wonder if this is the better solution and wonder why this was not discussed here before. Will try it and see how it behaves, but it looks very promising!

[blackwinter]

The default is to behave like with the original Metamorph - this is what your results reflect.

The difference is tag="020␣␣.a" (old) vs. tag="020␣.a" (new).

[blackwinter]

Ok, I see, the comment "this isn't working anymore" refers to metafacture-sandbox, not the original behaviour. I still think it's an incompatibility, though. This change is not guarded by the version parameter.

[dr0i]

@blackwinter yes, it's indeed an issue. Wished to have put 7ca5615 in another branch so that it could be discussed there.

[blackwinter]

wonder why this was not discussed here before.

I guess that's because it's not selective and doesn't allow "enriching" the output. It only allows for converting from one format into another and/or filtering out unwanted records. The e-mail thread you mentioned above sounds like pass-through should be applicable to selected elements, not necessarily the stream as a whole.

Wished to have put 7ca5615 in another branch so that it could be discussed there.

Well, why did you make those changes in the first place? I assume they were intended to address a shortcoming of this particular pass-through implementation: namely, that it passes the full entity path instead of just the individual entity/literal names (see the difference between metamorph1_1() and metamorph1_passthrough()). I would contend that any implementation of this feature should get by without requiring modifications in downstream consumers.

Both these points seem to indicate that we should maybe approach this issue from a different angle: API-wise, isn't this rather a property of the data element instead of the Metamorph definition? It seems what we want is a data element that acts as a entity/data hybrid that would be able to "unflatten" the stream.

To illustrate (untested):

@Test
public void metamorph1_unflatten() {
    metamorph = InlineMorph.in(this) //
            .with("<rules>")//
            .with("    <entity name='flattened' flushWith='record'>")//
            .with("        <data source='_else'/>")//
            .with("    </entity>")//
            .with("    <entity name='unflattened' flushWith='record'>")//
            .with("        <data source='_else' unflatten='true'/>")//
            .with("    </entity>")//
            .with("</rules>")//
            .createConnectedTo(receiver);

    metamorph.startRecord("1");
    metamorph.startEntity("clone");
    metamorph.literal("id", "0");
    metamorph.endEntity();
    metamorph.startEntity("clone");
    metamorph.literal("id", "1");
    metamorph.endEntity();
    metamorph.endRecord();

    final InOrder ordered = inOrder(receiver);
    ordered.verify(receiver).startRecord("1");
    ordered.verify(receiver).startEntity("flattened");
    ordered.verify(receiver).literal("clone.id", "0");
    ordered.verify(receiver).literal("clone.id", "1");
    ordered.verify(receiver).endEntity();
    ordered.verify(receiver).startEntity("unflattened");
    ordered.verify(receiver).startEntity("clone");
    ordered.verify(receiver).literal("id", "0");
    ordered.verify(receiver).endEntity();
    ordered.verify(receiver).startEntity("clone");
    ordered.verify(receiver).literal("id", "1");
    ordered.verify(receiver).endEntity();
    ordered.verify(receiver).endEntity();
    ordered.verify(receiver).endRecord();
}

WDYT?

[dr0i]

Added to the wiki.