extract_text() returns a unicode character \ufb03 LATIN SMALL LIGATURE FFI instead of the letters ffi when it comes across the word Office,

[colemanr03]

I am extracting information from over 1000 large PDF files. I am looking for rooms in a sketch with lots of pictures.

I love pdfplumber, it is so much better than the other methods I found. I do have one small problem.

When I find a room that is labeled 'Office', the exctract_Text routine returns \ufb03 LATIN SMALL LIGATURE FFI instead of the letters ffi

Office/2nd Floor instead of Office/2ndFloor.

They look the same but the first one has \ufb03 LATIN SMALL LIGATURE FFI instead of the letters ffi

Have you seen anything like this? Do you have a work around?

Thank you

[samkit-jain]

Hi @colemanr03 Appreciate your interest in the library. Could you please provide more details like the version of pdfplumber you are using, the PDF (redacting any sensitive information) that is causing the issue and a minimum reproducible code example?

[jsvine]

Just chiming in to echo @samkit-jain: Are you able to share the PDF, @colemanr03? It'd help resolve this issue, as I haven't come across another PDF that demonstrates this particular situation.

[jeffkile]

I'm running into the same issue. I can reproduce it consistently with this PDF: https://www.ck12info.org/wp-content/uploads/2008/12/CK12_Earth_Science_rev.pdf

"Fiction" and "Scientific" on page 9 (first page of chapter 1) end up as fiction and scientific (it may not be obvious at first glance but the "fi"s are a single character in these words)

I'm using version 0.8.0

[jsvine]

Hi @jeffkile, and thanks. For reference's sake, here's what I think you're pointing to:

import pdfplumber                                                                                         
pdf = pdfplumber.open("CK12_Earth_Science_rev.pdf")                                                       
page = pdf.pages[8]
print(page.extract_text())

... produces:

Chapter 1
What is Earth Science?
1.1 Nature of Science
Lesson Objectives
• Explain the importance of asking questions.
• State the steps of the scientific method.
• Describe the three major types of scientific models.
• Use appropriate safety precautions inside and outside the science laboratory.
Introduction
Think of your favorite science fiction movie. What is it about? Maybe it’s about spaceships
going to distant planets, or people being cloned in laboratories, or undersea civilizations, or
robots that walk among us. These entertaining imaginings are make-believe fantasies, that’s
why they’re called science “fiction.” They are not real. But why are they called “science”
fiction?
The answer is that science uses a disciplined process to answer questions. In science, “disci-
plined” does not mean well-behaved. It means following orderly steps in order to come up
with the best answers. Science involves observing, wondering, categorizing, communicating,
calculating, analyzing, and much more. In order to convert creativity into reality, we need
science. In order to travel beyond where anyone has gone before, we need science. In order
to understand the world, make sense of it, and conserve it, we need science. In order to
confirm our best guesses about the universe and the things in it, we need science. Science
fiction stories extend and expand on all the ideas of science and technology in creative ways.
1
www.ck12.org

... and that the word we would recognize as fiction appears as fiction (i.e., fi-ligature followed by ction).

I'm inferring (correct me if I'm mistaken), that you'd prefer .extract_text(...) to convert all fi ligatures to the two-character representation, fi.

That makes sense to me and, although it would introduce a breaking change for some users, it probably fits the most common use-case. Still, I'd want some way for users to preserve ligatures in the extracted text, perhaps through a preserve_ligatures=True parameter.

I'd be curious for your thoughts @samkit-jain!

[samkit-jain]

@jsvine As a user, yes, I too, would prefer to have it read fi instead of fi. Can add a new parameter but wouldn't it be too much for just one ligature? Or planning to add support for many?

[prakhs123]

This was bothering me as well, so I am using
text.replace('fi', 'fi')
in my code

[jeffkile]

Yeah I'm doing something similar with
text.replace("ff", "ff").replace("fi", "fi").replace("fl", "fl").replace("ffi", "ffi").replace("ffl", "ffl")

The problem is these are just the ones I've found so far, there may be other ones that I haven't found yet which users could see in production, so a generalized fix would really be preferred.

[colemanr03]

Sorry, that I did not supply the pdf file. I see someone else did. I found a similar work around with a text replace. I did not realize there were others I needed to look for. I can't believe that anyone would want their English text converted to ligatures.

[jsvine]

With v0.9.0, pdfplumber's text-extraction methods now expand the most common Latin-alphabet ligatures into their constituent characters. (It does not do so for ligatures that are considered to be their own letter, such as German's ß.) Let me know if/how it works for you! Thanks to commenters here for the suggestions and examples.

[Tom-Hudson]

@jsvine I am using 0.9.0 and I am still getting occasions where fi are being substituted with strange chars:

I am just using extract_text method. Any ideas?

[jsvine]

@Tom-Hudson Thank you for flagging. Can you share the PDF? (Hard to diagnose the issue without it. If it's a document you don't want to share publicly, you can email it to me — my address is in my GitHub profile.)

[Tom-Hudson]

@jsvine I have just emailed you the PDF now. Thanks so much for looking into this!

[Tom-Hudson]

Posting here to benefit anyone who encounters a similar issue and wrongly blames pdfplumber like me 😞

@jsvine was super helpful and pointed out that the PDF I was generating had null unicode characters to represent some ligatures.

We generate PDFs with Chromium from HTML and after reading this issue on StackOverflow: https://stackoverflow.com/questions/39504775/disabling-font-ligatures-css-letter-combining, I added CSS to disable ligatures in Chrome which solved my problem!

body {
  font-variant-ligatures: none;
  font-feature-settings: "liga" 0;
}

Here is the before and after which you can see in the PDF:

Before

After

After some testing, the actual cause was a combination of Chromium using ligatures and our usage of a webfont - I didn't dig too deep but it feels like there is something broken with ligatures and webfonts when printing to a PDF. If I re-enable ligatures with the default font, the ligatures are present but pdfplumber sorts them out because they use the correct unicode characters.

Thanks again @jsvine!