Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[IMPROVED] Bulk insert of messages #1140

Merged
merged 1 commit into from
Jan 8, 2021
Merged

Conversation

kozlovic
Copy link
Member

@kozlovic kozlovic commented Jan 8, 2021

A new configuration bulk_insert_limit switches the server from
the current insertion of messages within a SQL transaction, to
an "INSERT INTO MESSAGES () VALUES (),(),..." which can speed up
performance by several folds.

The server still may perform regular insert within transaction
if the limit is deemed too low.

This new configuration parameter is not enabled by default. It
needs to be explicitly set, either in configuration file or
from command line --sql_bulk_insert_limit <number here>.

Resolves #1132

Signed-off-by: Ivan Kozlovic ivan@synadia.com

A new configuration `bulk_insert_limit` switches the server from
the current insertion of messages within a SQL transaction, to
an "INSERT INTO MESSAGES () VALUES (),(),..." which can speed up
performance by several folds.

The server still may perform regular insert within transaction
if the limit is deemed too low.

This new configuration parameter is not enabled by default. It
needs to be explicitly set, either in configuration file or
from command line `--sql_bulk_insert_limit <number here>`.

Resolves #1132

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
@coveralls
Copy link

Coverage Status

Coverage increased (+0.07%) to 91.754% when pulling 20f9b79 on improve_sql_msg_insert_perf into 71d1dbc on master.

@kozlovic kozlovic requested a review from aricart January 8, 2021 22:06
@nats-io nats-io deleted a comment from kozlovic Jan 8, 2021
Copy link
Member

@aricart aricart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - sql parser for the win.

@kozlovic kozlovic merged commit 8be03bd into master Jan 8, 2021
@kozlovic kozlovic deleted the improve_sql_msg_insert_perf branch January 8, 2021 22:36
kozlovic added a commit that referenced this pull request Jan 9, 2021
This relates to #1140

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sql store performance enhancement
3 participants