Lines: support multiple RegEx-es in lines entries #417

rmilecki · 2022-10-01T21:32:35Z

So far "first_line", "line" and "last_line" could contain a single
RegEx only. Some invoices have lines that use more than one format. To
simplify parsin them allow all 3 entries to contain list of RegEx-es.

Example:
fields:
  lines:
    parser: lines
    start: Item\s+Discount\s+Price$
    end: \s+Total
    line:
      - Items group:\s+(?P<group>.+)
      - (?P<description>.+)\s+(?P<discount>\d+.\d+)\s+(?P<price>\d+\d+)

rmilecki · 2022-10-01T21:39:37Z

This is my alternative to the #378 that should really work this time.

@bosd: I believe you can parse your Mekro.pdf with:

# -*- coding: utf-8 -*-
issuer: Mekro
fields:
  amount: To Pay\s+(\d+.\d{2})
  amount_untaxed: Netto totaal[:]\s+(\d+[,]\d{2})
  date: Invoicedate\s.?\s+(\d{2}-\d{2}-\d{4})\s+\d{2}[:]\d{2}
  invoice_number: Invoicenumber[:]\s+(\S+)
  iban:
    parser: static
    value: NL44INGB0702593702
  partner_coc:
    parser: regex
    regex: '33166113'
  partner_website:
    parser: regex
    regex: mekro.nl
  lines:
    parser: lines
    start: Barcode
    line:
      - (?P<barcode>(\d{13}))\s+(?P<name>(\w+(?:\s\S+)*))\s+(?P<qty>(\d))\s+(?P<uom>\w+)\s+(?P<price_unit>(\d+.\d+))\s+(?P<discount>\d+.\d+)
      - ---(?P<line_note>.*ITEMS)---
    end: Netto totaal
keywords:
  - Mekro
  - NL001799434B01
options:
  date_formats:
    - '%d %m %Y'
  currency: EUR
  languages:
    - en
  decimal_separator: ','

that template gives me:

"lines": [
    {
        "line_note": "FOOD ITEMS"
    },
    {
        "barcode": "2231012001992",
        "name": "KROKETBROODJES",
        "qty": "2",
        "uom": "KG",
        "price_unit": "1,00",
        "discount": "0,0"
    },
    {
        "barcode": "8713009019455",
        "name": "Oil",
        "qty": "3",
        "uom": "L",
        "price_unit": "0,50",
        "discount": "0,0"
    },
    {
        "barcode": "8713009019475",
        "name": "Apple",
        "qty": "1",
        "uom": "KG",
        "price_unit": "50,0",
        "discount": "0,0"
    },
    {
        "line_note": "OTHER ITEMS"
    },
    {
        "barcode": "8713009019375",
        "name": "programmer",
        "qty": "1",
        "uom": "Hour",
        "price_unit": "50,0",
        "discount": "100,0"
    },
    {
        "barcode": "0013009019475",
        "name": "Sticker",
        "qty": "1",
        "uom": "pce",
        "price_unit": "0,0",
        "discount": "0,0"
    }
]

which I believe is what you expected.

bosd · 2022-10-07T16:00:15Z

@rmilecki Thanks for your efforts to produce an alternative with cleaner code.

Running the first test, on quality hosting example file.
Purpose to check if contentate function is working.

This is the result:

[
    {
        "issuer": "QualityHosting AG",
        "amount": 34.73,
        "amount_untaxed": 34.73,
        "date": "2014-05-07",
        "invoice_number": "30064443",
        "vat": "DE 232 446 240",
        "currency": "EUR",
        "lines": [
            {
                "pos": "1",
                "qty": 1.0,
                "desc": "Small Business StandardExchange 2010\nGrundgebühr pro Einheit\nDienst: OUDJQ_strukan\n01.05.14-31.05.14",
                "price": 3.89
            },
            {
                "pos": "2",
                "qty": 1.0,
                "desc": "Small Business QualityExchange 2010\nGrundgebühr pro Einheit\nDienst: OUDJQ_schneider\n01.05.14-31.05.14",
                "price": 5.39
            },
            {
                "pos": "3",
                "qty": 1.0,
                "desc": "Small Business QualityExchange 2010\nGrundgebühr pro Einheit\nDienst: OUDJQ_minar\n01.05.14-31.05.14",
                "price": 5.39
            },
            {
                "pos": "4",
                "qty": 1.0,
                "desc": "Small Business QualityExchange 2010\nGrundgebühr pro Einheit\nDienst: OUDJQ_mayr\n01.05.14-31.05.14",
                "price": 5.39
            },
            {
                "pos": "5",
                "qty": 1.0,
                "desc": "Small Business QualityExchange 2010\nGrundgebühr pro Einheit\nDienst: OUDJQ_jenewein\n01.05.14-31.05.14",
                "price": 5.39
            },
            {
                "pos": "6",
                "qty": 1.0,
                "desc": "Small Business QualityExchange 2010\nGrundgebühr pro Einheit\nDienst: OUDJQ_jauernik\n01.05.14-31.05.14\nQualityHosting AG - Uferweg 40-42 - D-63571 Gelnhausen\niViveLabs Ltd.\n93B Sai Yu Chung\nYuen Long, N.T.\nHong Kong\nPos.            Menge      Beschreibung                                                            Rabatt %     VK-Preis     Zeilenbetrag\nOhne      Ohne MwSt.\nMwSt.",
                "price": 5.39
            },
            {
                "pos": "7",
                "qty": 1.0,
                "desc": "Small Business StandardExchange 2010\nGrundgebühr pro Einheit\nDienst: OUDJQ_office\n01.05.14-31.05.14\n",
                "price": 3.89
            }
        ],
        "desc": "Invoice from QualityHosting AG"
    }
]

Note: it contantate correctly.

But something odd is going on in the last line of the first page.
It (incorrectly) adds the footer content to the line:

                "pos": "6",
                "qty": 1.0,
                "desc": "Small Business QualityExchange 2010\nGrundgebühr pro Einheit\nDienst: OUDJQ_jauernik\n01.05.14-31.05.14\nQualityHosting AG - Uferweg 40-42 - D-63571 Gelnhausen\niViveLabs Ltd.\n93B Sai Yu Chung\nYuen Long, N.T.\nHong Kong\nPos.            Menge      Beschreibung                                                            Rabatt %     VK-Preis     Zeilenbetrag\nOhne      Ohne MwSt.\nMwSt.",
                "price": 5.39
            },

The output from pdftotext for this part is:

                             Grundgebühr pro Einheit
                              Dienst: OUDJQ_jauernik
                              01.05.14-31.05.14

QualityHosting AG                                 Vorstand: Christian Heit (Vorsitz),         Bankverbindung
Uferweg 40-42                                     Markus Oestreicher                          Kreissparkasse Gelnhausen

note the line
QualityHosting AG Vorstand: Christian Heit (Vorsitz),

It does not contain a white space character at the beginning of the line.
According to the invoice template it should not match.

line: '^\s+(?P<desc>.+)$'

But it does

rmilecki · 2022-10-07T16:03:32Z

@bosd: can you paste or send me full pdftotext output, please? Feel free to replace your private data there with random text. That will make it much much easier for me to debug that problem.

bosd · 2022-10-07T16:11:41Z

Here it is:
(No personal info there, this is one of the example files included in this library)

DEBUG:invoice2data.main:START pdftotext result ===========================
DEBUG:invoice2data.main:
      QualityHosting AG - Uferweg 40-42 - D-63571 Gelnhausen




  iViveLabs Ltd.
  93B Sai Yu Chung
  Yuen Long, N.T.
  Hong Kong




Rechnung                                                                                Seite 1

Rechnungsnr.                 30064443                                                   Kundennr.              47774
Rechnungsdatum               7. Mai 2014



   Pos.            Menge      Beschreibung                                                          Rabatt %     VK-Preis     Zeilenbetrag
                                                                                                                    Ohne      Ohne MwSt.
                                                                                                                   MwSt.
                              Contract No. CON02858


       1                 1    Small Business StandardExchange 2010                                                     3,89           3,89
                              Grundgebühr pro Einheit
                              Dienst: OUDJQ_strukan
                              01.05.14-31.05.14
       2                 1    Small Business QualityExchange 2010                                                      5,39           5,39
                              Grundgebühr pro Einheit
                              Dienst: OUDJQ_schneider
                              01.05.14-31.05.14
       3                 1    Small Business QualityExchange 2010                                                      5,39           5,39
                              Grundgebühr pro Einheit
                              Dienst: OUDJQ_minar
                              01.05.14-31.05.14
       4                 1    Small Business QualityExchange 2010                                                      5,39           5,39
                              Grundgebühr pro Einheit
                              Dienst: OUDJQ_mayr
                              01.05.14-31.05.14
       5                 1    Small Business QualityExchange 2010                                                      5,39           5,39
                              Grundgebühr pro Einheit
                              Dienst: OUDJQ_jenewein
                              01.05.14-31.05.14
       6                 1    Small Business QualityExchange 2010                                                      5,39           5,39
                              Grundgebühr pro Einheit
                              Dienst: OUDJQ_jauernik
                              01.05.14-31.05.14

QualityHosting AG                                 Vorstand: Christian Heit (Vorsitz),         Bankverbindung
Uferweg 40-42                                     Markus Oestreicher                          Kreissparkasse Gelnhausen
D-63571 Gelnhausen                                Aufsichtsrat: Hans Jürgen                   Kto-Nr. 48567
Tel. +49 6051 916 44 10                           Habermann (Vorsitz)                         Blz: 507 500 94
Fax +49 6051 916 44 29                            Registergericht Hanau | HRB 13302           IBAN DE30507500940000048567
Im Internet: www.qualityhosting.de                UStId DE 232 446 240                        SWIFT HELADEF1GEL
eMail: info@qualityhosting.de                     Steuer-Nr. 044 241 601 03

  QualityHosting AG - Uferweg 40-42 - D-63571 Gelnhausen




  iViveLabs Ltd.
  93B Sai Yu Chung
  Yuen Long, N.T.
  Hong Kong




Rechnung                                                                                Seite 2

Rechnungsnr.                 30064443                                                   Kundennr.                47774
Rechnungsdatum               7. Mai 2014

   Pos.            Menge      Beschreibung                                                            Rabatt %     VK-Preis     Zeilenbetrag
                                                                                                                      Ohne      Ohne MwSt.
                                                                                                                     MwSt.
       7                 1    Small Business StandardExchange 2010                                                       3,89           3,89
                              Grundgebühr pro Einheit
                              Dienst: OUDJQ_office
                              01.05.14-31.05.14


                                                                                                    Total EUR                         34,73



Zahlungsform                                 Banküberweisung
Zahlungsbedingungen                          14 Tage netto
Zahlungsziel                                 21.05.14

Für Rückfragen bzgl. dieser Rechnung wenden SIe sich bitte per E-Mail unter der Angabe Ihrer Kunden- und Rechnungsnummer an
buchhaltung@qualityhosting.de.

Einwände gegen die Ihnen berechneten Lieferungen und Leistungen sind schriftlich innerhalb 4 Wochen ab Rechnungsdatum unserer
Buchhaltung anzuzeigen. Nach Ablauf dieser Frist gelten die Beträge als genehmigt. Im Falle einer Rücklastschrift der Beträge ohne
Verschulden der QualityHosting AG berechnen wir für die uns entstandenen Kosten ein Entgeld von 15,00 EUR. Unabhängig davon
behalten wir uns die Einstellung unserer Leistungen bis zum Ausgleich unserer Forderungen ausdrücklich vor.




QualityHosting AG                                 Vorstand: Christian Heit (Vorsitz),         Bankverbindung
Uferweg 40-42                                     Markus Oestreicher                          Kreissparkasse Gelnhausen
D-63571 Gelnhausen                                Aufsichtsrat: Hans Jürgen                   Kto-Nr. 48567
Tel. +49 6051 916 44 10                           Habermann (Vorsitz)                         Blz: 507 500 94
Fax +49 6051 916 44 29                            Registergericht Hanau | HRB 13302           IBAN DE30507500940000048567
Im Internet: www.qualityhosting.de                UStId DE 232 446 240                        SWIFT HELADEF1GEL
eMail: info@qualityhosting.de                     Steuer-Nr. 044 241 601 03


DEBUG:invoice2data.main:END pdftotext result =============================

rmilecki · 2022-10-08T20:12:46Z

@bosd: OK, I just found it's about QualityHosting.pdf.

That problem you reported - about parsing lines in QualityHosting.pdf - is caused by the way src/invoice2data/extract/templates/de/de.qualityhosting.yml is constructed. It has nothing to to with changes from this pull request.

de.qualityhosting.yml parses lines incorrectly without this pull request changes
de.qualityhosting.yml parses lines incorrectly with those changes

So while I agree de.qualityhosting.yml needs to be fixed it has really nothing to do with this pull request. I'm happy to help you fixing de.qualityhosting.yml if you need to parse QualityHosting invoices. It shouldn't be considered a blocked for this pull request however.

bosd · 2022-10-09T11:28:20Z

@rmilecki

de.qualityhosting.yml parses lines incorrectly without this pull request changes

I agree with you on this one.

de.qualityhosting.yml parses lines incorrectly with those changes

I think that is something that needs to be fixed. as of in #378 where this pr is an alternative implementation of that.

I'll respectfully disagree with you on changing the invoice template.
Yes, the invoice template is not optimal.
(It's a different topic, but as performance wise the . character (meta escape) should be avoided in python regexes.)

In this case, it is possible to add a last line rule, as of your proposal in #422.
However, I found many practical cases where it is impossible to define a lastline in the template.
In that case, all non-matching lines (like footers) should be discarded, until it finds a new first_line match.

Maybe better to leave suboptimal tests and examples in this library. Just as an showcase. (Same goes for the OCR examples in this repo). It is definityle helping us to find these corner cases.

However, the template set aside.

My analysis of whats happening here.
Instead of passing one line to the regex pattern match, it looks like it is trying to match the regex across the whole content.
(Did not dive into the code yet to verify this).

The regex is as follows:
\s+
matches any whitespace character (equivalent to [\r\n\t\f\v ])

matches the previous token between one and unlimited times, as many times as possible.

When looking at the line:

QualityHosting AG                                 Vorstand: Christian Heit (Vorsitz),         Bankverbindung

It does not contain a whitespace character at the beginning of the line.
So it should not be used in the output.
(This is the implementation of #378).

However, if we feed the following into the parser:


QualityHosting AG                                 Vorstand: Christian Heit (Vorsitz),         Bankverbindung

An linebreak \n has been added in the line above our footer. Now the regex
\s+(?P<desc>.+)
Is matching on the first line of the footer. It is matching because it is using the linebreak from the previous line.

This should not happen, as the linebreak is clearly on another line.

rmilecki · 2022-10-16T20:10:44Z

@bosd: in your analysis of \s+ treating I think you misread input invoice. That line: '^\s+(?P<desc>.+)$' in de.qualityhosting.yml works as you expected. It matches only those lines that start with whitespaces.

When looking at the line:

QualityHosting AG Vorstand: Christian Heit (Vorsitz), Bankverbindung

It does not contain a whitespace character at the beginning of the line.
So it should not be used in the output.

I agree with you. I believe you are correct. It should not be used in the output and it isn't used in the output.

So things work just like you expect them.

Please check this pdftotext output with my comments (scroll RIGHT please!):

       6                 1    Small Business QualityExchange 2010                                                      5,39           5,39
                              Grundgebühr pro Einheit
                              Dienst: OUDJQ_jauernik
                              01.05.14-31.05.14

QualityHosting AG                                 Vorstand: Christian Heit (Vorsitz),         Bankverbindung                    ← doesn't start with whitespace = doesn't appear in the output
Uferweg 40-42                                     Markus Oestreicher                          Kreissparkasse Gelnhausen         ← doesn't start with whitespace = doesn't appear in the output
D-63571 Gelnhausen                                Aufsichtsrat: Hans Jürgen                   Kto-Nr. 48567                     ← doesn't start with whitespace = doesn't appear in the output
Tel. +49 6051 916 44 10                           Habermann (Vorsitz)                         Blz: 507 500 94                   ← doesn't start with whitespace = doesn't appear in the output
Fax +49 6051 916 44 29                            Registergericht Hanau | HRB 13302           IBAN DE30507500940000048567       ← doesn't start with whitespace = doesn't appear in the output
Im Internet: www.qualityhosting.de                UStId DE 232 446 240                        SWIFT HELADEF1GEL                 ← doesn't start with whitespace = doesn't appear in the output
eMail: info@qualityhosting.de                     Steuer-Nr. 044 241 601 03

  QualityHosting AG - Uferweg 40-42 - D-63571 Gelnhausen                                                                        ← starts with whitespace = appears in the output = expected




  iViveLabs Ltd.                                                                                                                ← starts with whitespace = appears in the output = expected
  93B Sai Yu Chung                                                                                                              ← starts with whitespace = appears in the output = expected
  Yuen Long, N.T.                                                                                                               ← starts with whitespace = appears in the output = expected
  Hong Kong                                                                                                                     ← starts with whitespace = appears in the output = expected




Rechnung                                                                                Seite 2                                 ← doesn't start with whitespace = doesn't appear in the output

Rechnungsnr.                 30064443                                                   Kundennr.                47774          ← doesn't start with whitespace = doesn't appear in the output
Rechnungsdatum               7. Mai 2014

   Pos.            Menge      Beschreibung                                                            Rabatt %     VK-Preis     Zeilenbetrag
                                                                                                                      Ohne      Ohne MwSt.
                                                                                                                     MwSt.
       7                 1    Small Business StandardExchange 2010                                                       3,89           3,89
                              Grundgebühr pro Einheit
                              Dienst: OUDJQ_office
                              01.05.14-31.05.14

So I think everything works correct and just like described that you expect them to.

bosd · 2022-10-19T12:15:59Z

@rmilecki Thanks for the very clear information.
Makes sense to change the template, to only include a line when there are at least X amount of spaces on the beginning of the line.

rmilecki · 2022-10-21T13:05:54Z

@bosd: so could we have this one merged now, please?

It's a clear implementation, solves actual problem you reported, doesn't seem to regress anything. I find it a nice feature.

It's not meant to solve all cases our templates can't handle now. But it does solve one and I believe it's worth to have it.

We can work on handling more cases in further changes (e.g. #407, #423) but we need to start moving forward with something. I'm happy to discuss and work together on other cases later. At the same time I'd like to start merging proposed features.

bosd · 2022-10-21T13:36:52Z

@rmilecki I want to move this one forward as well. As I really want to have this functionality.
Yet, first I would like to test is a bit more thourougly.
It is quite big, to test this, (will use real use case templates and invoices)
Hope I can do that this weekend, and give the approval then.

bosd · 2022-10-21T19:51:25Z

@rmilecki Thanks for collaborating on this and all your efforts! Let's Merge!! 🎉 ✨

Tested this code against a bunch of pdf's and templates locally with great success!!!!

Some notes:

The parsing of different "blocks" is not working with the following syntax.

lines:
 - start: Efficient Invoice Handling
   end:  5555 NH TommyCity
   line: (?P<test>(Sessamestreet 46))
 - start: Barcode
   end:  Netto totaal
   line: (?P<barcode>(\d{13}))\s+(?P<name>(\w+(?:\s\S+)*))\s+(?P<qty>(\d))\s+(?P<uom>\w+)\s+(?P<price_unit>(\d+.\d+))\s+(?P<discount>\d+.\d+)

I tried a variant with the new syntax, but could not make it work.
What would be the right syntax to parse multiple (different) blocks of lines??
Since this is related to #423 , which is about the parsing of multiple similar blocks. I would not consider it to block this pr.

The documentation / examples need updating. To show the correct syntax to parse multiple lines.
eg.

    line:
      - (?P<barcode>(\d{13}))\s+(?P<name>(\w+(?:\s\S+)*))\s+(?P<qty>(\d))\s+(?P<uom>\w+)\s+(?P<price_unit>(\d+.\d+))\s+(?P<discount>\d+.\d+)
      - ---(?P<line_note>.*ITEMS)---

(I will take care of no 2, as it is in my pipeline to provide a new real invoice)

So far "first_line", "line" and "last_line" could contain a single RegEx only. Some invoices have lines that use more than one format. To simplify parsin them allow all 3 entries to contain list of RegEx-es. Example: fields: lines: parser: lines start: Item\s+Discount\s+Price$ end: \s+Total line: - Items group:\s+(?P<group>.+) - (?P<description>.+)\s+(?P<discount>\d+.\d+)\s+(?P<price>\d+\d+) Signed-off-by: Rafał Miłecki <rafal@milecki.pl>

bosd

Tested extensively! LGTM!! 🎉 🥇

rmilecki requested review from m3nu and bosd October 1, 2022 21:32

bosd added this to the 0.4.0 release milestone Oct 7, 2022

rmilecki mentioned this pull request Oct 15, 2022

templates: de: improve parsing QualityHosting lines #422

Open

bosd mentioned this pull request Oct 21, 2022

parsers: lines: support multiple occurrence of blocks to parse #423

Merged

bosd force-pushed the parsers-lines-multiple-regexes branch from a5558a4 to 81d4cfd Compare October 21, 2022 19:52

bosd approved these changes Oct 21, 2022

View reviewed changes

bosd merged commit 01529a5 into invoice-x:master Oct 21, 2022

bosd mentioned this pull request Oct 21, 2022

Add posibility to parse multiple line definitions #378

Closed

rmilecki mentioned this pull request Oct 22, 2022

Lines Feature: Capturing multiple positions AND the respective order number #346

Closed

bosd mentioned this pull request Oct 22, 2022

parsers: lines: support "rules" for multiple sets of regexes #407

Closed

rmilecki mentioned this pull request Oct 22, 2022

tests: add some tests for the "lines" parser #427

Merged

rmilecki deleted the parsers-lines-multiple-regexes branch October 24, 2022 15:09

rmilecki mentioned this pull request Feb 3, 2023

tests: add custom test for parsing lines #415

Closed

rmilecki mentioned this pull request Feb 18, 2023

lines: support "rules" field for multiple sets of parsing regexes #463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lines: support multiple RegEx-es in lines entries #417

Lines: support multiple RegEx-es in lines entries #417

rmilecki commented Oct 1, 2022 •

edited

Loading

rmilecki commented Oct 1, 2022

bosd commented Oct 7, 2022

rmilecki commented Oct 7, 2022

bosd commented Oct 7, 2022 •

edited

Loading

rmilecki commented Oct 8, 2022

bosd commented Oct 9, 2022 •

edited

Loading

rmilecki commented Oct 16, 2022 •

edited

Loading

bosd commented Oct 19, 2022

rmilecki commented Oct 21, 2022

bosd commented Oct 21, 2022

bosd commented Oct 21, 2022

bosd left a comment

Lines: support multiple RegEx-es in lines entries #417

Lines: support multiple RegEx-es in lines entries #417

Conversation

rmilecki commented Oct 1, 2022 • edited Loading

rmilecki commented Oct 1, 2022

bosd commented Oct 7, 2022

rmilecki commented Oct 7, 2022

bosd commented Oct 7, 2022 • edited Loading

rmilecki commented Oct 8, 2022

bosd commented Oct 9, 2022 • edited Loading

rmilecki commented Oct 16, 2022 • edited Loading

bosd commented Oct 19, 2022

rmilecki commented Oct 21, 2022

bosd commented Oct 21, 2022

bosd commented Oct 21, 2022

bosd left a comment

Choose a reason for hiding this comment

rmilecki commented Oct 1, 2022 •

edited

Loading

bosd commented Oct 7, 2022 •

edited

Loading

bosd commented Oct 9, 2022 •

edited

Loading

rmilecki commented Oct 16, 2022 •

edited

Loading