Concurent safe batch saver #15247
Merged
Concurent safe batch saver #15247
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@miq-bot add_label enhancement |
|
@Ladas 'enhancement' is an invalid assignee, ignoring... |
|
@miq-bot add_label enhancement |
Helper methods for building batch SQL create/update. For create method, we also have a build_multi_selection_query that is able to fetch the batch created objects, using their unique indexes.
Fix rubocop issue if -> next
Include SqlHelper module to a saver base class
Extract batch_size to a base class
Extract delete_record method to a base class
Use base class delete_record method
Use a base class methods in concurent_safe saver
Add concurrent_batch_safe saver
Ladas
force-pushed
the
concurent_batch_safe_saver
branch
from
1a268ed
to
25840aa
Compare
|
Checked commits Ladas/manageiq@0e2a1f9~...25840aa with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0 |
agrare
approved these changes
Jun 11, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on:
Concurent batch safe saver strategy. Allowing us to do create&update in batched SQL queries, which will save a lot o time, using a remote db. (having 5ms network latency, we save ~1.5h on 1M of records being created/updated)
TBD:
A possbile candidate for replacing the INSERT ON CONFLICT UPDATE query is https://github.com/zdennis/activerecord-import/wiki/On-Duplicate-Key-Update
Also filled issue in Arel, since that would be the best to use:
rails/arel#485