Add clean_ligatures to core cleaners

[walsha2]

Background

Ligatures can sometimes show up during the text extraction process when they should not. Very common examples of this are with the Latin f related ligatures which can be very subtle to spot by eye (see example below), but can wreak havoc later.

"ff": "ff",
"fi": "fi",
"fl": "fl",
"ffi": "ffi",
"ffl": "ffl",

Several libraries already do something like this. Most recently, pdfplumber added this sort of capability as part of the text extraction process, see jsvine/pdfplumber#598

Instead of incorporating any sort of breaking change to the PDF text processing in unstructured, it is best to add this as another cleaner and allow users to opt in. In turn, the clean_ligatures method has been added in this PR - with accompanying tests.

Example

Here is an example PDF that causes the issue. For example: Benefits, which should be Benefits.

example.pdf

curl -X 'POST' \
    'https://api.unstructured.io/general/v0/general' \
    -H 'accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: ${UNSTRUCTURED_API_KEY}' \
    -F 'files=@example.pdf' \
    -s | jq -C .

Notes

An initial list of mappings was added with the most common ligatures. There is some subjectivity to this, but this should be a relatively safe starting set. Can always be expanded as needed.

[cragwolfe]

@walsha2 , please add a bullet near the top of CHANGELOG.md. this is a great cleaning addition, thanks!
i'm tempted for this to be included by default in many partition cases, tbh, but that can follow in a later PR.

[walsha2]

add a bullet near the top of CHANGELOG.md

Done.

i'm tempted for this to be included by default in many partition cases

I thought about it as well, but I just started messing with unstructured this week. Will defer that decision about partition incorporation to someone more in the know! 😄

---

First PR to get feet wet with repo. Awesome package. Looking forward to making more contributions in the future!

[walsha2]

@cragwolfe I cant really make sense of the CI actions. Not sure why the CI / lint (3.8) is failing. Does not look to be PR related at all.

make: black: No such file or directory

https://github.com/Unstructured-IO/unstructured/actions/runs/6105531406/job/16569692294?pr=1326

[cragwolfe]

@cragwolfe I cant really make sense of the CI actions. Not sure why the CI / lint (3.8) is failing. Does not look to be PR related at all.

make: black: No such file or directory

https://github.com/Unstructured-IO/unstructured/actions/runs/6105531406/job/16569692294?pr=1326

yes, github CI was having a rough time for some unknown reason -- frustrating! it seems to be fine now.
i just merged in changes to main, let's see how it goes 🤞