fix(pdf): Fix garbled text in PDF loaders

[engkimo]

Hi team,

This PR implements the first part of the fix for the garbled text issue (#29555), starting with PDF loaders. This is part of a larger effort to address encoding issues across multiple document loaders.

Changes

• Add PDFEncoding enum for standardized encoding options
• Implement PDFEncodingConfig for centralized encoding management
• Enhance BasePDFLoader with encoding configuration support
• Update PDFMinerParser to handle encoding properly
• Add fallback mechanism for encoding errors

Next Steps

This PR focuses on PDF loaders as the first implementation. Similar fixes will be needed for:

• Docx2txtLoader
• PyPDFLoader
• TextLoader
• UnstructuredExcelLoader
• UnstructuredWordDocumentLoader

Notes

• No breaking changes to existing API
• Default encoding remains UTF-8
• Users can now specify custom encoding configurations when needed
• Fixes regression introduced in o3 release for PDF loaders

Environment

• Tested on Python 3.13.1
• Compatible with:
  • langchain 0.3.17
  • langchain_community 0.3.16
  • langchain_core 0.3.33

Related issue: #29555

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Feb 3, 2025 3:37pm

[efriis]

hey there! is the issue that there isn't a way to pass in an encoding to these loaders currently, or is the issue that there isn't an encoding abstraction that is compatible with these different loaders?

I would rather just make sure we're passing through kwargs you might want to use to configure these underlying libraries, and skip standardizing the PDF encoding definition because different PDF libraries might have other flags you might want to configure as well, and we're not aiming to be a full abstraction layer for all pdf parsing libraries' configuration

[engkimo]

@efriis
Hi! Thank you for the clarification. The encoding issues affect multiple loaders (PDFMinerLoader, Docx2txtLoader, PyPDFLoader, TextLoader, UnstructuredExcelLoader, UnstructuredWordDocumentLoader, etc.).

Would this approach work better?

1. Add **kwargs to the BaseLoader class init
2. Pass these kwargs through to each underlying document parsing library
3. Document the encoding options available for each loader type

This would:

• Maintain flexibility for each document library's unique features
• Allow consistent encoding configuration across different loader types
• Keep LangChain's role as a thin wrapper rather than a full abstraction layer

Does this align better with LangChain's philosophy while addressing the encoding issues across multiple loader types?