Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat!: Docling v2 #117

Merged
merged 117 commits into from
Oct 16, 2024
Merged

feat!: Docling v2 #117

merged 117 commits into from
Oct 16, 2024

Conversation

cau-git
Copy link
Contributor

@cau-git cau-git commented Oct 1, 2024 •
edited by dolfim-ibm
Loading

  • Introduces extra abstraction level for document backends (see stub codes for HTML / Word / Powerpoint)
  • Introduces new interface for pipelines
    • Pipelines have more responsibility now, they need to implement execute, assemble_document and additional methods each.
  • Introduces automatic routing of detected format to a default backend
  • Introduces new top-level converter interface (changed arguments)

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the
    conventional commits.
  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

@cau-git
Reorganize imports from docling-core
abb6ddd 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Put stub for experimental format export
ac51a09 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Lots of import refactoring
12477c8 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Merge branch 'main' of github.com:DS4SD/docling into cau/experimental…
257d44a 
…-format
@cau-git
Add exporter methods to new types
d790731 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Add experimental output in glm_model
b54956c 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Merge from main
867e06f 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Switch everything to use label enum, and more
33373ac 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Update lockfile
850a521 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
[WIP] introducting extra backend abstraction and input formats
95c5395 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Apply renamings to DocItemLabel
3efc2bb 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Sync test data from main
48d8b7b 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Update GT test files for pages
ad2bd71 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Examples: Don't export experimental output by default
ba9d115 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Bump deepsearch-glm
0ee82a5 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Merge from main
9ffd1dc
@cau-git
Import rewrites, adapt to changes in docling-core
2461b56 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Renaming changes
91ab382 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Repinning
0a86529 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Merge branch 'cau/experimental-format' of github.com:DS4SD/docling in…
cd06d89 
…to cau/input-format-abstraction
@cau-git cau-git changed the title feat: Input format abstraction feat: Input format abstraction [WIP] Oct 1, 2024
@cau-git cau-git changed the base branch from main to cau/experimental-format October 1, 2024 14:51
@cau-git
Fundamental refactoring for multi-format support
1fa7cd9 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git cau-git force-pushed the cau/input-format-abstraction branch from 4941c9b to 1fa7cd9 Compare October 1, 2024 14:54
Working on a first version of DOCX native backend
cefc34e 
Signed-off-by: Maxim Lysak <mly@zurich.ibm.com>
@maxmnemonic
Copy link
Contributor

Implementing DOCX support...

cau-git and others added 3 commits October 7, 2024 12:35
@cau-git
Add comments
e613f7b 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Improved docx parsing
1346843 
Signed-off-by: Maxim Lysak <mly@zurich.ibm.com>
Added mspowerpoint backend first implementation, improvements on mswo…
bea9fc2 
…rd backend

Signed-off-by: Maxim Lysak <mly@zurich.ibm.com>
@maxmnemonic
Copy link
Contributor

Added PPTX support

@cau-git cau-git mentioned this pull request Oct 15, 2024
Closed
4 tasks
cau-git and others added 20 commits October 15, 2024 16:22
@cau-git
Merge from main and remove conflicts
84438bd 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@vagenas @cau-git
add v2 docs placeholder [skip ci] (#145)
c1794a7 
* add v2 docs placeholder [skip ci]

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* Remove conflicts

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Change output folder for examples.
40bb84d 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Adjust CI examples path
9d15f4d 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Add migration instructions to doc (wip)
74e0452 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@dolfim-ibm
add options to generate images
1cb11be 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
Merge branch 'cau/input-format-abstraction' of github.com:DS4SD/docli…
75feef2 
…ng into cau/input-format-abstraction
@dolfim-ibm
fix generation of images and adapt examples
cd8e3dc 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@cau-git
Add migration instructions to doc (cont)
df3ff47 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Update v2 documentation
8a25230
@dolfim-ibm
pin models, core and adapt example
dd2982c 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@cau-git
Update docling-core pinnings
c123e5a 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@dolfim-ibm
apply changes to the picture data annotations
d5f161d 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@cau-git
Update docling-core pinnings
515ab04 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Added and fixed origin for msword and mspowerpoint backend
a07a187 
Signed-off-by: Maxim Lysak <mly@zurich.ibm.com>
@cau-git
Update docling-core pinnings
5c862b5 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git
Merge branch 'cau/input-format-abstraction' of github.com:DS4SD/docli…
07206c5 
…ng into cau/input-format-abstraction
@cau-git
Documentation updates, remove DescriptionItem in DoclingDocument init
734d77c 
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@vagenas
update RAG examples and banner text
fb028b7 
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
@cau-git
Merge pull request #149 from DS4SD/update-rag-examples
0858ed9 
update RAG examples and banner text
@cau-git cau-git changed the title feat: Input format abstraction [WIP] feat!: Docling v2 Oct 16, 2024
@dolfim-ibm
simplify README, move content to docs
b7d9761 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm
pin core and glm
4cead1e 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
@dolfim-ibm dolfim-ibm marked this pull request as ready for review October 16, 2024 17:38
@dolfim-ibm
apply pre-commit with new path
4918e11 
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
dolfim-ibm
dolfim-ibm approved these changes Oct 16, 2024
View reviewed changes
Copy link
Contributor

@dolfim-ibm dolfim-ibm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@staar
Copy link

staar commented Oct 16, 2024

@dolfim-ibm @cau-git seems we are close! 🚀

@dolfim-ibm dolfim-ibm merged commit 7d3be0e into main Oct 16, 2024
4 checks passed
@dolfim-ibm dolfim-ibm deleted the cau/input-format-abstraction branch October 16, 2024 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dolfim-ibm dolfim-ibm dolfim-ibm approved these changes

Assignees

@cau-git cau-git

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support Html as native input document type Support Docx as native input document type
5 participants
@cau-git @maxmnemonic @staar @dolfim-ibm @vagenas