Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bulk import: recommend 128 inserts per batch for java #4399

Closed
bdarnell opened this issue Feb 20, 2019 · 2 comments
Closed

Bulk import: recommend 128 inserts per batch for java #4399

bdarnell opened this issue Feb 20, 2019 · 2 comments
Assignees
@rmloveland
Labels
C-doc-improvement O-external Origin: Issue comes from external users. P-1 High priority; must be done this release T-missing-info
Milestone
19.2

Comments

@bdarnell
Copy link
Contributor

bdarnell commented Feb 20, 2019 •
edited
Loading

PGJDBC's batching support is bizarre and magical: it only works with powers of two, and will split batches of other sizes up into multiple sub-batches. This means that a batch of size 128 can be 6x faster than a batch of size 250. We should adjust our docs to recommend batches of exactly 128 for java users (for other languages, around 500 is still optimal. Maybe change the recommendation to 512 in case other libraries are sensitive to powers of two).
@jseldess jseldess added O-external Origin: Issue comes from external users. T-missing-info A-sql P-1 High priority; must be done this release labels Feb 26, 2019
@jseldess jseldess added this to the 19.1 milestone Feb 26, 2019
@jseldess jseldess self-assigned this Apr 1, 2019
@jseldess
Copy link
Contributor

jseldess commented Apr 3, 2019

@rmloveland, can you factor this in while working on #4621 with @timveil?

@jseldess jseldess assigned rmloveland and unassigned jseldess Apr 3, 2019
@jseldess jseldess modified the milestones: 19.1, 19.2 Apr 30, 2019
rmloveland added a commit that referenced this issue May 30, 2019
@rmloveland
[WIP] Simplify JDBC code sample
70ecb33 
Fixes #4621, #3578, #4399.

Summary of changes:

- Update 'Build a Java App with CockroachDB' to use the Java DAO pattern

- Add a 'Recommended Practices' section that includes information about
  IMPORT, recommended batch size, and the JDBC INSERT rewriting flag

TODO:

- Update "secure" code sample to use certs.  Not worth doing yet until
  this code is reviewed by someone with Java expertise and updated to
  match good style, etc.  Once that is done, the secure code sample in
  this PR will be updated with the necessary cert munging.
@rmloveland rmloveland mentioned this issue May 30, 2019
Merged
rmloveland added a commit that referenced this issue Jun 6, 2019
@rmloveland
Simplify JDBC code sample
e3fcc9e 
Fixes #4621, #3578, #4399.

Summary of changes:

- Update 'Build a Java App with CockroachDB' to use:

  - the Java DAO pattern

  - JDBC

  - code for automatically retrying in case of txn retry errors

  - an example of fast bulk insertion using JDBC batching

- Add a 'Recommended Practices' section that includes information about
  IMPORT, recommended batch size, and the JDBC INSERT rewriting flag
rmloveland added a commit that referenced this issue Jun 17, 2019
@rmloveland
Simplify JDBC code sample
b5dd9ce 
Fixes #4621, #3578, #4399.

Summary of changes:

- Update 'Build a Java App with CockroachDB' to use:

  - the Java DAO pattern

  - JDBC

  - code for automatically retrying in case of txn retry errors

  - an example of fast bulk insertion using JDBC batching

- Add a 'Recommended Practices' section that includes information about
  IMPORT, recommended batch size, and the JDBC INSERT rewriting flag
rmloveland added a commit that referenced this issue Jun 20, 2019
@rmloveland
Simplify JDBC code sample
a739028 
Fixes #4621, #3578, #4399.

Summary of changes:

- Update 'Build a Java App with CockroachDB' to use:

  - the Java DAO pattern

  - JDBC

  - code for automatically retrying in case of txn retry errors

  - an example of fast bulk insertion using JDBC batching

- Add a 'Recommended Practices' section that includes information about
  IMPORT, recommended batch size, and the JDBC INSERT rewriting flag
rmloveland added a commit that referenced this issue Jun 24, 2019
@rmloveland
Simplify JDBC code sample
13c50fa 
Summary of changes:

- Update 'Build a Java App with CockroachDB' to use:
  - the Java DAO pattern
  - JDBC
  - code for automatically retrying in case of txn retry errors
  - an example of fast bulk insertion using JDBC batching

- Add a 'Recommended Practices' section that includes information about
  IMPORT, recommended batch size, and the JDBC INSERT rewriting flag

Does all of the above for versions 19.1 and 19.2.

Fixes #4621, #3578, #4399.
rmloveland added a commit that referenced this issue Jun 25, 2019
@rmloveland
Simplify JDBC code sample
e69a0d0 
Summary of changes:

- Update 'Build a Java App with CockroachDB' to use:
  - the Java DAO pattern
  - JDBC
  - code for automatically retrying in case of txn retry errors
  - an example of fast bulk insertion using JDBC batching

- Add a 'Recommended Practices' section that includes information about
  IMPORT, recommended batch size, and the JDBC INSERT rewriting flag

Does all of the above for versions 19.1 and 19.2.

Fixes #4621, #3578, #4399.
@rmloveland
Copy link
Contributor

Fixed by #4855

@MrPowerGamerBR MrPowerGamerBR mentioned this issue Sep 14, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rmloveland rmloveland

Labels
C-doc-improvement O-external Origin: Issue comes from external users. P-1 High priority; must be done this release T-missing-info
Projects
None yet
Milestone
19.2
Development

No branches or pull requests
3 participants
@bdarnell @rmloveland @jseldess