Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ultimate batch saving speedup #15761

Merged
merged 14 commits into from
Aug 10, 2017
Merged

Commits on Aug 9, 2017

  1. Check if IC has serializable_keys? and transform them

    Check if IC has serializable_keys? and transform them, this saves
    some computing since we don not have to use :use_ar_object just
    because the model has: serialize <col_name>
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    23c6e47 View commit details
    Browse the repository at this point in the history
  2. Add TODOs for noticed possible bugs

    Add TODOs for noticed possible bugs
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    374f1fd View commit details
    Browse the repository at this point in the history
  3. Use to_sym for individual keys instead of symbolize_keys!

    Use to_sym for individual keys instead of symbolize_keys!
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    8c8a941 View commit details
    Browse the repository at this point in the history
  4. Performance tweaks for the base saver

    Performance tweaks for the base saver, getting repeated logic
    computed in initializer. Getting record_key abrtratcion for
    fetching attributes of non AR records.
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    e6d0c48 View commit details
    Browse the repository at this point in the history
  5. Perf tweaks of the batch saver

    Perf tweaks of the batch saver:
    1. Having iterator that fetches raw SQL without creationg AR
       objects.
    2. Select on the fetched data to reduce the mem size needed.
    3. Bumping batch_size to 10k, since the objects are 10x smaller
       than AR objects.
    And few small tweaks in the core saving
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    2894ecf View commit details
    Browse the repository at this point in the history
  6. Perf tweaks in SQL mixin

    Perf tweaks in SQL mixin:
    1. Store connection for the whole batch
    2. Precalculate pg_types for faster type_casting
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    b32d820 View commit details
    Browse the repository at this point in the history
  7. Fix bad comments

    Fix bad comments
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    84556e4 View commit details
    Browse the repository at this point in the history
  8. Fix rubocop issues

    Fix rubocop issues
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    7e34e96 View commit details
    Browse the repository at this point in the history
  9. Correct symbol vs string keys

    Correct symbol vs string keys
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    59b7895 View commit details
    Browse the repository at this point in the history
  10. Move batch size to attributes with setter

    Move batch size to attributes with setter, so we avoid computing
    it multiple times.
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    c80763d View commit details
    Browse the repository at this point in the history
  11. Remove redundant symbolize_keys

    Remove redundant symbolize_keys
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    7f09ddd View commit details
    Browse the repository at this point in the history
  12. Use faster inventory_object.id call

    Use faster inventory_object.id call
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    08ee63f View commit details
    Browse the repository at this point in the history
  13. Store primary_key_value to avoid multiple method calls

    Store primary_key_value to avoid multiple method calls
    Ladas committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    577e10f View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2017

  1. Use reorder for find_in_batches working properly

    Use reorder for find_in_batches working properly
    Ladas committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    60e14d1 View commit details
    Browse the repository at this point in the history