UnParse - Email parsing library

[McPizza0]

We will release our own email parsing library to extract replies/ signatures from incoming emails
This topic will serve as a place to gather information and links to possible approaches.

[McPizza0]

Wikipedia for a good laugh: https://en.wikipedia.org/wiki/Posting_style

Current libs

• https://github.com/lever/planer : maintained : removes replies : coffeescript : its great because it can parse both plain text and HTML email content
• https://github.com/crisp-oss/email-reply-parser: maintained : removes replies: javascript : only works on plain text emails, so would lose all formatting
• https://github.com/mailgun/talon : unmaintained : quotations or signatures : python : a lot of refrences to it, but last update was in 2016. training dataset also available
• https://github.com/yourtempo/tempo-email-parser : unmaintained : updated in our repo, some functionality broken : typescript : RIP Tempo - but they left us a nice gift. We are currently using this in our monorepo, though some stuff is broken.
  ^^this also has some great issues and prs for all the sig/reply stuff - including how far they got into stuff https://github.com/yourtempo/tempo-email-parser/projects/1?query=is%3Aopen+sort%3Aupdated-desc

Implementations

• https://github.com/Foundry376/Mailspring : maintained : handles both sig and reply : typescript : mainspring is an email client, the extractions stuff looks to be client side, but some of the logic could be brought over.

[McPizza0]

Another alternative is an LLM
we will have a local one anyway for some other uses, maybe its easier to do for this too 😂

[McPizza0]

Signature Matching
we can use the html data in these links to delete the signatures from emails
most providers add attributes to their signature divs
https://github.com/M66B/FairEmail/blob/a6ff1bb03026a661694d9f0c6c7c273465b0f73f/app/src/main/java/eu/faircode/email/HtmlHelper.java#L2784

More links coming soon

[ezra-en]

https://discord.com/channels/1113119653246545961/1186608361093988392/1213485141964357652
Playground for parser rule-writing is an idea we could work on down the line, especially if we're not putting in LLMs/vector search in right now