Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Acceleration of batch import jobs by enabling the use of sql batch operations #1291

Closed

Conversation

stephanr
Copy link
Member

To improve batch loading speed, it is critical to reduce the amount of database round trips for each operation.
To achieve this, it is required to prevent auto generated columns when idGenMode == IDGenMode.USE_EXISTING && idGenMode.isSkipResolveReferences() is used and all identifying column values are already known.

In this PR the handling of auto generated columns was optimized in InsertRow and also support for batching was introduced in InsertRowManager.

@stephanr stephanr added enhancement enhancement or improvement needs discussion requires discussion with contributor hard labels Mar 11, 2022
@stephanr
Copy link
Member Author

@tfr42 tfr42 added tools deegree command line tools (CLI) needs rebase PR is not up to date and needs rebase labels May 12, 2022
… requested and idGenMode == IDGenMode.USE_EXISTING && idGenMode.isSkipResolveReferences();

* Allow custom particle mapping MultiParticleConverter
@stephanr stephanr force-pushed the enhancement/optimize-batch-load branch from dda0ac5 to 5f613e8 Compare September 16, 2022 09:12
@stephanr stephanr removed the needs rebase PR is not up to date and needs rebase label Sep 16, 2022
@stephanr stephanr added in progress and removed needs discussion requires discussion with contributor hard labels Dec 21, 2022
@stephanr
Copy link
Member Author

At today's tmc meeting (see https://github.com/deegree/deegree3/wiki/tmc-meeting-minutes-20221221), we discussed this pull request and concluded that it is too specific to be included in its current form.
However, it would still be appreciated to keep opther fixes which are not related to SQL batching.

@lgoltz lgoltz added the needs rebase PR is not up to date and needs rebase label May 22, 2023
@stephanr
Copy link
Member Author

This PR will be closed for now

@stephanr stephanr closed this Nov 30, 2023
@tfr42 tfr42 removed the in progress label Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement enhancement or improvement needs rebase PR is not up to date and needs rebase tools deegree command line tools (CLI)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants