Add HTML input support #312

fsteeg · 2020-01-22T11:23:35Z

For HTML processing as in https://github.com/programmieraffe/oerhoernchen20 (with sources configured as in https://github.com/programmieraffe/oerhoernchen20/blob/master/scrapy/projects.json), we need an HtmlReader, to be used between HttpOpener and XmlDecoder (parsing HTML to XML, e.g. with https://jsoup.org/) in a workflow to create JSON index data for https://github.com/orgs/hbz/projects/4.

The text was updated successfully, but these errors were encountered:

Parse HTML with jsoup, write XML. See example in test. See #312

To use with decode-xml, but how to test? See #312

fsteeg · 2020-01-24T13:38:13Z

Current state, on http://test.lobid.org/fix (which includes metafacture-html from branch 312-html), use:

Data:

<div id=1><h1>Faust</h1><b>Goethe</b></div><div id=2><h1>Prozess</h1><b>Kafka

Flux:

html-to-xml|decode-xml|handle-generic-xml("div")|fix|encode-json

Fix:

map(_id, id)
map(h1.value,title)
map(b.value,author)
/*map(_else)*/

fsteeg · 2020-01-29T10:14:27Z

To test a real world example, I did:

Open https://www.hoou.de/sitemap.xml
Find the first entry with materials
Open https://www.hoou.de/materials/tutorial-lernen-lernen
Copy content of view-source:https://www.hoou.de/materials/tutorial-lernen-lernen
Paste as data in http://test.lobid.org/fix
Flux: html-to-xml|decode-xml|handle-generic-xml("html")|fix|encode-json
Fix: map(head.title.value,title)

The full input data fails due to an org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 29901; The value of attribute "value" associated with an element type "input" must not contain the '<' character. at org.metafacture.xml.XmlDecoder.process(XmlDecoder.java:69). To verify the basic idea, I used just the beginning of the HTML source, until the head.title element, as input data:

<!DOCTYPE html> <html lang="de"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="msapplication-TileColor" content="#ffffff"> <meta name="msapplication-TileImage" content="/assets/images/favicon/ms-icon-144x144-082c0e8e414258ffcec59a1e87679f9c.png"> <meta name="viewport" content="width=device-width, initial-scale=1">  <meta name="ember-cli-head-start" content> <meta property="title" content="Tutorial: Lernen lernen - HOOU"> <title>Tutorial: Lernen lernen - HOOU</title>

With this, we get a JSON result:

{"title":"Tutorial: Lernen lernen - HOOU"}

The problem above is caused by value="<embed src="https://https://www.hoou.de/m/undefined?embedded=true" style="overflow:hidden;height:100vh;width:100%"></embed>" attributes. If we remove these (there are 3 in this example), we can process the full input. If we use map(_else) as Fix, we can see the field structure, and pick out more information with a Fix like this:

map(head.title.value,title)
map(body.div.div.div.div.div.div.div.p.value,content)

We get a JSON result:

{"title":"Tutorial: Lernen lernen - HOOU","content":"Das Bewusstsein und die Kenntnis über Ihren Lernstil kann Ihnen helfen, Ihren Lernansatz und damit auch den Lernerfolg zu optimieren. In diesem Modul reflektieren Sie Ihren Lernstil und dessen Implikationen und entwickeln individuelle Lernstrategien. Zudem hilft Ihnen das Wissen über unterschiedliche Lernstile beim Lernen in der Gruppe oder bei der Teamarbeit."}

I tried a few other input examples, but all HOOU content has these value="<embed attributes, and other input from other sites (from https://github.com/programmieraffe/oerhoernchen20/blob/master/scrapy/projects.json) has other XML parsing issues, like undeclared or unclosed entities. I think this proves what @dr0i already suspected when we discussed this: converting these real world HTML inputs to XML does not really work.

Since the problem happens when the XML is parsed (at org.metafacture.xml.XmlDecoder.process(XmlDecoder.java:69)), not when the HTML is parsed, I think this could be a viable approach: parse the HTML with jsoup and generate metadata events directly, without the intermediate XML representation. In a Flux, this would basically work like this: decode-html|fix|encode-json. Internally, it would require a setup similar to XmlDecoder, but extending DefaultObjectPipe<Reader, StreamReceiver>, passing on the HTML DOM elements, instead of doing the SAX parsing.

With `decode-html` flux command See #312

Set generated record ID, only process content of leaf nodes See #312

fsteeg · 2020-02-04T14:36:22Z

Implemented new HtmlDecoder described above, the basic approach seems to work, deployed to http://test.lobid.org/fix:

Data: (Full, unmodified) source of https://www.hoou.de/materials/tutorial-lernen-lernen

Flux: decode-html|fix|encode-json(prettyPrinting="true")

Fix:

map(html.head.title.value,title)
map(html.body.div.div.div.div.div.div.div.p.value, description)

Output:

{
  "title" : "Tutorial: Lernen lernen - HOOU",
  "description" : "Das Bewusstsein und die Kenntnis ..."
}

I suggest we leave this open until we are done with some full real-world scenarios. For these, I think we should continue with support for full Flux workflows in the UI (metafacture/metafacture-fix#6) and conditionals in Fix (metafacture/metafacture-fix#10). @acka47, you might still want to play around with this a little already (use map(_else) as Fix to see the full field structure).

See #312 See hbz/oerindex#3

See #312

fsteeg · 2020-03-16T10:54:26Z

metafacture/metafacture-fix#6 is closed and #313 is merged, closing.

See #312

See: #312 https://gitlab.com/oersi/oersi-etl/-/issues/27 https://jsoup.org/cookbook/extracting-data/selector-syntax

See #312

…github.com/metafacture/metafacture-fix

fsteeg added Enhancement Stream modules labels Jan 22, 2020

fsteeg self-assigned this Jan 22, 2020

fsteeg changed the title ~~Add HtmlHandler~~ Add HtmlReader Jan 22, 2020

fsteeg added a commit that referenced this issue Jan 23, 2020

Basic HtmlReader with html-to-xml flux command

1dfddca

Parse HTML with jsoup, write XML. See example in test. See #312

fsteeg mentioned this issue Jan 23, 2020

Add HTML input support #313

Merged

fsteeg added a commit that referenced this issue Jan 24, 2020

Output to Reader, not String

4cc023d

To use with decode-xml, but how to test? See #312

fsteeg mentioned this issue Jan 27, 2020

Support full Flux with URL input in web UI metafacture/metafacture-fix#6

Closed

fsteeg changed the title ~~Add HtmlReader~~ Add HTML input support Jan 29, 2020

fsteeg added a commit that referenced this issue Feb 4, 2020

Add HtmlDecoder and tests

f2ae2e9

With `decode-html` flux command See #312

fsteeg added a commit that referenced this issue Feb 4, 2020

Process document recursively

6605128

Set generated record ID, only process content of leaf nodes See #312

fsteeg mentioned this issue Feb 4, 2020

Support conditionals metafacture/metafacture-fix#10

Closed

This was referenced Feb 5, 2020

Process HTML DOM elements with metafacture-fix hbz/oerindex#2

Closed

Process embedded JSON in HTML with metafacture-fix hbz/oerindex#3

Closed

fsteeg added a commit that referenced this issue Feb 6, 2020

Add ScriptExtractor with extract-script Flux command

3447246

See #312 See hbz/oerindex#3

fsteeg added a commit that referenced this issue Feb 7, 2020

Remove html-to-xml component

3427a67

See #312

fsteeg closed this as completed Mar 16, 2020

fsteeg mentioned this issue Mar 16, 2020

Release 5.2.0 #318

Closed

fsteeg added a commit that referenced this issue Apr 7, 2020

Add metafacture-html to metafacture-runner config

b884850

See #312

fsteeg added a commit that referenced this issue Nov 11, 2020

Add extract-element flux command based on CSS-style selector

97d9e8c

See: #312 https://gitlab.com/oersi/oersi-etl/-/issues/27 https://jsoup.org/cookbook/extracting-data/selector-syntax

fsteeg added a commit that referenced this issue Dec 8, 2020

Add extract-element flux command based on CSS-style selector

442f447

See: #312 https://gitlab.com/oersi/oersi-etl/-/issues/27 https://jsoup.org/cookbook/extracting-data/selector-syntax

fsteeg added a commit that referenced this issue Mar 25, 2021

Don't duplicate element value when adding it as a subfield

3710938

See #312

dr0i modified the milestone: Next release Apr 13, 2021

fsteeg added a commit that referenced this issue Apr 19, 2021

Don't duplicate element value when adding it as a subfield

227a13a

See #312

blackwinter pushed a commit that referenced this issue Dec 13, 2024

Merge #312 from branch '311-updateDependencyMfc5.7.0-rc1' of https://…

ef44f8c

…github.com/metafacture/metafacture-fix

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add HTML input support #312

Add HTML input support #312

fsteeg commented Jan 22, 2020 •

edited

Loading

fsteeg commented Jan 24, 2020

fsteeg commented Jan 29, 2020 •

edited

Loading

fsteeg commented Feb 4, 2020

fsteeg commented Mar 16, 2020

Add HTML input support #312

Add HTML input support #312

Comments

fsteeg commented Jan 22, 2020 • edited Loading

fsteeg commented Jan 24, 2020

fsteeg commented Jan 29, 2020 • edited Loading

fsteeg commented Feb 4, 2020

fsteeg commented Mar 16, 2020

fsteeg commented Jan 22, 2020 •

edited

Loading

fsteeg commented Jan 29, 2020 •

edited

Loading