Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update pdf2parquet to Docling v2 #756

Merged
merged 16 commits into from
Oct 31, 2024
Merged

Update pdf2parquet to Docling v2 #756

merged 16 commits into from
Oct 31, 2024

Conversation

dolfim-ibm
Copy link
Member

@dolfim-ibm dolfim-ibm commented Oct 30, 2024 •
edited
Loading

Why are these changes needed?

The new Docling v2 allows to

  1. Process more input formats: PDF, DOCX, PPTX, HTML, Markdown, ASCII Docs
  2. Faster PDF backend
  3. Improvements in the generated document
  4. New DoclingDocument
  5. Additional export format to plain text

In progress

  • upgrade dependencies and adapt code
  • add safe-guard for the download of model weights on multi-processes
  • implement flushing mechanism for creating batches of files
  • update doc_chunk tansform
  • propagate parameters to kfp

Related issue number (if any).

@dolfim-ibm
update to docling v2 and expose new parameters
f90f134 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
update to docling v2
1436480 
new parameters and input formats
faster backend
revalidated the test results

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
add lock
8c22f0d 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
add batch_size
d55e6bd 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
update parameter in README
261230c 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
fix multilock with default parameters
e396e16 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
use multilock with fix
5095c1b 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
propagate new param to kfp
f62e6de 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
update new models download in Dockerfile
7e5ea90 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
update doc_chunk with new docling v2
e929903 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
update to 2.3.1 with initialize_pipeline
b4eb978 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@touma-I touma-I requested a review from daw3rd October 30, 2024 18:10
@dolfim-ibm dolfim-ibm marked this pull request as ready for review October 30, 2024 18:27
@touma-I touma-I self-requested a review October 30, 2024 18:32
@dolfim-ibm
remove debug log
8395a25 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
daw3rd
daw3rd requested changes Oct 30, 2024
View reviewed changes
Copy link
Member

@daw3rd daw3rd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do any of. your tests exercise self.buffer and flush_binary()?

transforms/language/doc_chunk/python/src/doc_chunk_transform.py Show resolved Hide resolved
transforms/language/pdf2parquet/python/requirements.txt
@@ -1,6 +1,6 @@
data-prep-toolkit==0.2.2.dev1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be >= 0.2.2.dev1. dev1 may eventually go away.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here I rely on @daw3rd and @touma-I expertise. let me know what is the correct value to use

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@touma-I I defer to you on this one.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dolfim-ibm @daw3rd this will be reset during the release process or whenever we push to pypi. for now, I wouldn't worry about it. You're good.

transforms/language/pdf2parquet/python/src/pdf2parquet_transform.py Show resolved Hide resolved
@dolfim-ibm
improve parsing of metadata
4c693f9 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
add test case for batch_size
26b429a 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
notify users about the deprecated argument
269d732 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
daw3rd
daw3rd approved these changes Oct 31, 2024
View reviewed changes
Copy link
Member

@daw3rd daw3rd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM modulo @touma-I's call on the >= versioning.

@touma-I touma-I merged commit 5877e4d into dev Oct 31, 2024
14 checks passed
This was referenced Nov 1, 2024
Open
Open
Open
@shahrokhDaijavad shahrokhDaijavad mentioned this pull request Nov 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daw3rd daw3rd daw3rd approved these changes

@touma-I touma-I Awaiting requested review from touma-I

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dolfim-ibm @daw3rd @touma-I