From f3b7f780c4413a02cfb19786e247a5c7a3f0690a Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 05:16:06 -0700 Subject: [PATCH 01/13] Add section about validate_uniqueness option Closes https://github.com/zdennis/activerecord-import/issues/430 --- README.markdown | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.markdown b/README.markdown index f0151b9d..fee3f60f 100644 --- a/README.markdown +++ b/README.markdown @@ -23,6 +23,7 @@ an 18 hour batch process to <2 hrs. ## Table of Contents +* [Uniqueness Validation](#uniqueness-validation) * [Callbacks](#callbacks) * [Additional Adapters](#additional-adapters) * [Requiring](#requiring) @@ -32,6 +33,14 @@ an 18 hour batch process to <2 hrs. * [Conflicts With Other Gems](#conflicts-with-other-gems) * [More Information](#more-information) +### Uniqueness Validation + +By default, `activerecord-import` will not validate for uniquness when importing records. Starting with `v0.27.0`, there is a parameter called `validate_uniqueness` that can be passed in to trigger this behavior. This option is provided with caution as there are many potential pitfalls. Please use with caution. + +```ruby +Book.import books, validate_uniqueness: true +``` + ### Callbacks ActiveRecord callbacks related to [creating](http://guides.rubyonrails.org/active_record_callbacks.html#creating-an-object), [updating](http://guides.rubyonrails.org/active_record_callbacks.html#updating-an-object), or [destroying](http://guides.rubyonrails.org/active_record_callbacks.html#destroying-an-object) records (other than `before_validation` and `after_validation`) will NOT be called when calling the import method. This is because it is mass importing rows of data and doesn't necessarily have access to in-memory ActiveRecord objects. From 6236a32d9d49206182c989b760d1ca8acb946a6f Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 05:34:19 -0700 Subject: [PATCH 02/13] Add section about not using array of hashes Closes https://github.com/zdennis/activerecord-import/issues/507 --- README.markdown | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/README.markdown b/README.markdown index fee3f60f..94077dd7 100644 --- a/README.markdown +++ b/README.markdown @@ -23,6 +23,7 @@ an 18 hour batch process to <2 hrs. ## Table of Contents +* [Array of Hashes](#array-of-hashes) * [Uniqueness Validation](#uniqueness-validation) * [Callbacks](#callbacks) * [Additional Adapters](#additional-adapters) @@ -33,6 +34,27 @@ an 18 hour batch process to <2 hrs. * [Conflicts With Other Gems](#conflicts-with-other-gems) * [More Information](#more-information) +## Array of Hashes + +Due to the counter-intuitive behavior that can occur when dealing with hashes instead of ActiveRecord objects, `activerecord-import` will raise an exception when passed an array of hashes. If you have an array of hash attributes, you should instead use them to instantiate an array of ActiveRecord objects and then pass that into `import`. + +See https://github.com/zdennis/activerecord-import/issues/507 for discussion. + +```ruby +arr = [ + { bar: 'abc' }, + { baz: 'xyz' }, + { bar: '123', baz: '456' } +] + +# An exception will be raised +Foo.import arr + +# better +arr.map! { |args| Foo.new(args) } +Foo.import arr +``` + ### Uniqueness Validation By default, `activerecord-import` will not validate for uniquness when importing records. Starting with `v0.27.0`, there is a parameter called `validate_uniqueness` that can be passed in to trigger this behavior. This option is provided with caution as there are many potential pitfalls. Please use with caution. From 017a815ed06563e84a76cd7452ad163d75a7e92a Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 05:55:48 -0700 Subject: [PATCH 03/13] Add section about counter cache logic Migrates https://github.com/zdennis/activerecord-import/wiki/Counter-Cache-Column --- README.markdown | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.markdown b/README.markdown index 94077dd7..03df1cf7 100644 --- a/README.markdown +++ b/README.markdown @@ -25,6 +25,7 @@ an 18 hour batch process to <2 hrs. * [Array of Hashes](#array-of-hashes) * [Uniqueness Validation](#uniqueness-validation) +* [Counter Cache](#counter-cache) * [Callbacks](#callbacks) * [Additional Adapters](#additional-adapters) * [Requiring](#requiring) @@ -63,6 +64,13 @@ By default, `activerecord-import` will not validate for uniquness when importing Book.import books, validate_uniqueness: true ``` +### Counter Cache + +When running `import`, `activerecord-import` does not automatically update counter cache columns. To update these columns, you will need to do one of the following: + +* Provide values to the column as an argument on your object that is passed in. +* Manually update the column after the record has been imported. + ### Callbacks ActiveRecord callbacks related to [creating](http://guides.rubyonrails.org/active_record_callbacks.html#creating-an-object), [updating](http://guides.rubyonrails.org/active_record_callbacks.html#updating-an-object), or [destroying](http://guides.rubyonrails.org/active_record_callbacks.html#destroying-an-object) records (other than `before_validation` and `after_validation`) will NOT be called when calling the import method. This is because it is mass importing rows of data and doesn't necessarily have access to in-memory ActiveRecord objects. From 7a043861f7570d5fc43b843eea3f8b669110e777 Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 06:03:41 -0700 Subject: [PATCH 04/13] Add section about running tests Migrates https://github.com/zdennis/activerecord-import/wiki/How-to-run-the-tests/ --- README.markdown | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/README.markdown b/README.markdown index 03df1cf7..c490bdda 100644 --- a/README.markdown +++ b/README.markdown @@ -34,6 +34,8 @@ an 18 hour batch process to <2 hrs. * [Load Path Setup](#load-path-setup) * [Conflicts With Other Gems](#conflicts-with-other-gems) * [More Information](#more-information) +* [Contributing](#contributing) + * [Running Tests](#running-tests) ## Array of Hashes @@ -218,6 +220,28 @@ For more information on activerecord-import please see its wiki: https://github. To document new information, please add to the README instead of the wiki. See https://github.com/zdennis/activerecord-import/issues/397 for discussion. +### Contributing + +#### Running Tests + +The first thing you need to do is set up your database(s): + +* copy `test/database.yml.sample` to `test/database.yml` +* modify `test/database.yml` for your database settings +* create databases as needed + +After that, you can run the tests. They run against multiple tests and ActiveRecord versions. + +This is one example of how to run the tests: + +```ruby +rm Gemfile.lock +AR_VERSION=4.2 bundle install +AR_VERSION=4.2 bundle exec rake test:postgresql test:sqlite3 test:mysql2 +``` + +Once you have pushed up your changes, you can find your CI results [here](https://travis-ci.org/zdennis/activerecord-import/). + # License This is licensed under the ruby license. From c8267579e1bec7649e79cee04ca31933a73ce7df Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 06:07:23 -0700 Subject: [PATCH 05/13] Add section about ActiveRecord timestamps Migrates https://github.com/zdennis/activerecord-import/wiki/Note:-ActiveRecord-Timestamps --- README.markdown | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/README.markdown b/README.markdown index c490bdda..11e15e2a 100644 --- a/README.markdown +++ b/README.markdown @@ -26,6 +26,7 @@ an 18 hour batch process to <2 hrs. * [Array of Hashes](#array-of-hashes) * [Uniqueness Validation](#uniqueness-validation) * [Counter Cache](#counter-cache) +* [ActiveRecord Timestamps](#activerecord-timestamps) * [Callbacks](#callbacks) * [Additional Adapters](#additional-adapters) * [Requiring](#requiring) @@ -73,6 +74,16 @@ When running `import`, `activerecord-import` does not automatically update count * Provide values to the column as an argument on your object that is passed in. * Manually update the column after the record has been imported. +### ActiveRecord Timestamps + +If you're familiar with ActiveRecord you're probably familiar with its timestamp columns: created_at, created_on, updated_at, updated_on, etc. When importing data the timestamp fields will continue to work as expected and each timestamp column will be set. + +Should you wish to specify those columns, you may use the option @timestamps: false@. + +However, it is also possible to set just @:created_at@ in specific records. In this case despite using @timestamps: true@, @:created_at@ will be updated only in records where that field is @nil@. Same rule applies for record associations when enabling the option @recursive: true@. + +If you are using custom time zones, these will be respected when performing imports as well as long as @ActiveRecord::Base.default_timezone@ is set, which for practically all Rails apps it is + ### Callbacks ActiveRecord callbacks related to [creating](http://guides.rubyonrails.org/active_record_callbacks.html#creating-an-object), [updating](http://guides.rubyonrails.org/active_record_callbacks.html#updating-an-object), or [destroying](http://guides.rubyonrails.org/active_record_callbacks.html#destroying-an-object) records (other than `before_validation` and `after_validation`) will NOT be called when calling the import method. This is because it is mass importing rows of data and doesn't necessarily have access to in-memory ActiveRecord objects. From e8c28a27db700209d713c715bce297a577e5aa0c Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 06:15:55 -0700 Subject: [PATCH 06/13] Add section with examples Migrates https://github.com/zdennis/activerecord-import/wiki/Examples --- README.markdown | 90 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 89 insertions(+), 1 deletion(-) diff --git a/README.markdown b/README.markdown index 11e15e2a..ebca1e8d 100644 --- a/README.markdown +++ b/README.markdown @@ -23,6 +23,10 @@ an 18 hour batch process to <2 hrs. ## Table of Contents +* [Examples](#examples) + * [Columns and Arrays](#columns-and-arrays) + * [ActiveRecord Models](#activerecord-models) + * [Batching](#batching) * [Array of Hashes](#array-of-hashes) * [Uniqueness Validation](#uniqueness-validation) * [Counter Cache](#counter-cache) @@ -38,7 +42,91 @@ an 18 hour batch process to <2 hrs. * [Contributing](#contributing) * [Running Tests](#running-tests) -## Array of Hashes +### Examples + +#### Columns and Arrays + +The @#import@ method can take an array of column names (string or symbols) and an array of arrays. Each child array represents an individual record and its list of values in the same order as the columns. This is the fastest import mechanism and also the most primitive. + +```ruby +columns = [ :title, :author ] +values = [ ['Book1', 'FooManChu'], ['Book2', 'Bob Jones'] ] + +# Importing without model validations +Book.import columns, values, :validate => false + +# Import with model validations +Book.import columns, values, :validate => true + +# when not specified :validate defaults to true +Book.import columns, values +``` + +#### ActiveRecord Models + +The @#import@ method can take an array of models. The attributes will be pulled off from each model by looking at the columns available on the model. + +```ruby +books = [ + Book.new(:title => "Book 1", :author => "FooManChu"), + Book.new(:title => "Book 2", :author => "Bob Jones") +] + +# without validations +Book.import books, :validate => false + +# with validations +Book.import books, :validate => true + +# when not specified :validate defaults to true +Book.import books +``` + +The @#import@ method can take an array of column names and an array of models. The column names are used to determine what fields of data should be imported. The following example will only import books with the @:title@ field: + +```ruby +books = [ + Book.new(:title => "Book 1", :author => "FooManChu"), + Book.new(:title => "Book 2", :author => "Bob Jones") +] +columns = [ :title ] + +# without validations +Book.import columns, books, :validate => false + +# with validations +Book.import columns, books, :validate => true + +# when not specified :validate defaults to true +Book.import columns, books + +# result in table books +# title | author +#--------|-------- +# Book 1 | NULL +# Book 2 | NULL + +``` + +#### Batching + +The @#import@ method can take a @:batch_size@ option to control the number of rows to insert per INSERT statement. The default is the total number of records being inserted so there is a single INSERT statement. + +```ruby +books = [ + Book.new(:title => "Book 1", :author => "FooManChu"), + Book.new(:title => "Book 2", :author => "Bob Jones"), + Book.new(:title => "Book 1", :author => "John Doe"), + Book.new(:title => "Book 2", :author => "Richard Wright") +] +columns = [ :title ] + +# 2 INSERT statements for 4 records +Book.import columns, books, :batch_size => 2 + +``` + +### Array of Hashes Due to the counter-intuitive behavior that can occur when dealing with hashes instead of ActiveRecord objects, `activerecord-import` will raise an exception when passed an array of hashes. If you have an array of hash attributes, you should instead use them to instantiate an array of ActiveRecord objects and then pass that into `import`. From 1ffbffaaee67b48d10e1b409605b671d39639ba7 Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 06:25:13 -0700 Subject: [PATCH 07/13] Add sections for duplicate key options + move uniqueness validation option Migrates https://github.com/zdennis/activerecord-import/wiki/On-Duplicate-Key-Ignore Migrates https://github.com/zdennis/activerecord-import/wiki/On-Duplicate-Key-Update --- README.markdown | 127 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 118 insertions(+), 9 deletions(-) diff --git a/README.markdown b/README.markdown index ebca1e8d..c493458c 100644 --- a/README.markdown +++ b/README.markdown @@ -27,8 +27,11 @@ an 18 hour batch process to <2 hrs. * [Columns and Arrays](#columns-and-arrays) * [ActiveRecord Models](#activerecord-models) * [Batching](#batching) +* [Options](#options) + * [Duplicate Key Ignore](#duplicate-key-ignore) + * [Duplicate Key Update](#duplicate-key-update) + * [Uniqueness Validation](#uniqueness-validation) * [Array of Hashes](#array-of-hashes) -* [Uniqueness Validation](#uniqueness-validation) * [Counter Cache](#counter-cache) * [ActiveRecord Timestamps](#activerecord-timestamps) * [Callbacks](#callbacks) @@ -126,6 +129,120 @@ Book.import columns, books, :batch_size => 2 ``` +### Options + +#### Duplicate Key Ignore + +[MySQL](http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html), [SQLite](https://www.sqlite.org/lang_insert.html), and [PostgreSQL](https://www.postgresql.org/docs/current/static/sql-insert.html#SQL-ON-CONFLICT) (9.5+) support `on_duplicate_key_ignore` which allows you to skip records if a primary or unique key constraint is violated. + +```ruby +book = Book.create! title: "Book1", author: "FooManChu" +book.title = "Updated Book Title" +book.author = "Bob Barker" + +Book.import [book], on_duplicate_key_ignore: true + +book.reload.title # => "Book1" (stayed the same) +book.reload.author # => "FooManChu" (stayed the same) +``` + +The option `:on_duplicate_key_ignore` is bypassed when `:recursive` is enabled for [PostgreSQL imports](https://github.com/zdennis/activerecord-import/wiki#recursive-example-postgresql-only). + +#### Duplicate Key Update + +MySQL, PostgreSQL (9.5+), and SQLite (3.24.0+) support @on duplicate key update@ (also known as "upsert") which allows you to specify fields whose values should be updated if a primary or unique key constraint is violated. + +One big difference between MySQL and PostgreSQL support is that MySQL will handle any conflict that happens, but PostgreSQL requires that you specify which columns the conflict would occur over. SQLite models its upsert support after PostgreSQL. + +h2. Basic Update + +```ruby +book = Book.create! title: "Book1", author: "FooManChu" +book.title = "Updated Book Title" +book.author = "Bob Barker" + +# MySQL version +Book.import [book], on_duplicate_key_update: [:title] + +# PostgreSQL version +Book.import [book], on_duplicate_key_update: {conflict_target: [:id], columns: [:title]} + +# PostgreSQL shorthand version (conflict target must be primary key) +Book.import [book], on_duplicate_key_update: [:title] + +book.reload.title # => "Updated Book Title" (changed) +book.reload.author # => "FooManChu" (stayed the same) +``` + +h2. Using the value from another column + +```ruby +book = Book.create! title: "Book1", author: "FooManChu" +book.title = "Updated Book Title" + +# MySQL version +Book.import [book], on_duplicate_key_update: {author: :title} + +# PostgreSQL version (no shorthand version) +Book.import [book], on_duplicate_key_update: { + conflict_target: [:id], columns: {author: :title} +} + +book.reload.title # => "Book1" (stayed the same) +book.reload.author # => "Updated Book Title" (changed) +``` + +h2. Using Custom SQL + +```ruby +book = Book.create! title: "Book1", author: "FooManChu" +book.author = "Bob Barker" + +# MySQL version +Book.import [book], on_duplicate_key_update: "author = values(author)" + +# PostgreSQL version +Book.import [book], on_duplicate_key_update: { + conflict_target: [:id], columns: "author = excluded.author" +} + +# PostgreSQL shorthand version (conflict target must be primary key) +Book.import [book], on_duplicate_key_update: "author = excluded.author" + +book.reload.title # => "Book1" (stayed the same) +book.reload.author # => "Bob Barker" (changed) +``` + +h2. PostgreSQL Using constraints + +```ruby +book = Book.create! title: "Book1", author: "FooManChu", edition: 3, published_at: nil +book.published_at = Time.now + +# in migration +execute <<-SQL + ALTER TABLE books + ADD CONSTRAINT for_upsert UNIQUE (title, author, edition); + SQL + +# PostgreSQL version +Book.import [book], on_duplicate_key_update: {constraint_name: :for_upsert, columns: [:published_at]} + + +book.reload.title # => "Book1" (stayed the same) +book.reload.author # => "FooManChu" (stayed the same) +book.reload.edition # => 3 (stayed the same) +book.reload.published_at # => 2017-10-09 (changed) +``` + +#### Uniqueness Validation + +By default, `activerecord-import` will not validate for uniquness when importing records. Starting with `v0.27.0`, there is a parameter called `validate_uniqueness` that can be passed in to trigger this behavior. This option is provided with caution as there are many potential pitfalls. Please use with caution. + +```ruby +Book.import books, validate_uniqueness: true +``` + ### Array of Hashes Due to the counter-intuitive behavior that can occur when dealing with hashes instead of ActiveRecord objects, `activerecord-import` will raise an exception when passed an array of hashes. If you have an array of hash attributes, you should instead use them to instantiate an array of ActiveRecord objects and then pass that into `import`. @@ -147,14 +264,6 @@ arr.map! { |args| Foo.new(args) } Foo.import arr ``` -### Uniqueness Validation - -By default, `activerecord-import` will not validate for uniquness when importing records. Starting with `v0.27.0`, there is a parameter called `validate_uniqueness` that can be passed in to trigger this behavior. This option is provided with caution as there are many potential pitfalls. Please use with caution. - -```ruby -Book.import books, validate_uniqueness: true -``` - ### Counter Cache When running `import`, `activerecord-import` does not automatically update counter cache columns. To update these columns, you will need to do one of the following: From f1a45ebbd0f7cf4a96e4f9ec4921ecd3a56b6361 Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 06:28:05 -0700 Subject: [PATCH 08/13] Add section on supported adapters Migrates https://github.com/zdennis/activerecord-import/wiki/Supported-Database-Adapters --- README.markdown | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/README.markdown b/README.markdown index c493458c..83e5da4a 100644 --- a/README.markdown +++ b/README.markdown @@ -35,6 +35,7 @@ an 18 hour batch process to <2 hrs. * [Counter Cache](#counter-cache) * [ActiveRecord Timestamps](#activerecord-timestamps) * [Callbacks](#callbacks) +* [Supported Adapters](#supported-adapters) * [Additional Adapters](#additional-adapters) * [Requiring](#requiring) * [Autoloading via Bundler](#autoloading-via-bundler) @@ -319,6 +320,24 @@ end Book.import valid_books, validate: false ``` +### Supported Adapters + +The following database adapters are currently supported: + +* MySQL - supports core import functionality plus on duplicate key update support (included in activerecord-import 0.1.0 and higher) +* MySQL2 - supports core import functionality plus on duplicate key update support (included in activerecord-import 0.2.0 and higher) +* PostgreSQL - supports core import functionality (included in activerecord-import 0.1.0 and higher) +* SQLite3 - supports core import functionality (included in activerecord-import 0.1.0 and higher) +* Oracle - supports core import functionality through DML trigger (available as an external gem: "activerecord-import-oracle_enhanced":https://github.com/keeguon/activerecord-import-oracle_enhanced) +* SQL Server - supports core import functionality (available as an external gem: "activerecord-import-sqlserver":https://github.com/keeguon/activerecord-import-sqlserver) + +If your adapter isn't listed here, please consider creating an external gem as described in the README to provide support. If you do, feel free to update this wiki to include a link to the new adapter's repository! + +To test which features are supported by your adapter, use the following methods on a model class: +* supports_import?(*args) +* supports_on_duplicate_key_update? +* supports_setting_primary_key_of_imported_objects? + ### Additional Adapters Additional adapters can be provided by gems external to activerecord-import by providing an adapter that matches the naming convention setup by activerecord-import (and subsequently activerecord) for dynamically loading adapters. This involves also providing a folder on the load path that follows the activerecord-import naming convention to allow activerecord-import to dynamically load the file. From b3f69b6b329fd67ddef4240cd3f153f5fb60c6b8 Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 06:34:41 -0700 Subject: [PATCH 09/13] Add relevant examples from the home wiki page Migrates https://github.com/zdennis/activerecord-import/wiki/Home/ --- README.markdown | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/README.markdown b/README.markdown index 83e5da4a..ad98a990 100644 --- a/README.markdown +++ b/README.markdown @@ -21,12 +21,21 @@ and then the reviews: That would be about 4M SQL insert statements vs 3, which results in vastly improved performance. In our case, it converted an 18 hour batch process to <2 hrs. +The gem provides the following high-level features: + +* activerecord-import can work with raw columns and arrays of values (fastest) +* activerecord-import works with model objects (faster) +* activerecord-import can perform validations (fast) +* activerecord-import can perform on duplicate key updates (requires MySQL or Postgres 9.5+) + ## Table of Contents * [Examples](#examples) + * [Introduction](#introduction) * [Columns and Arrays](#columns-and-arrays) * [ActiveRecord Models](#activerecord-models) * [Batching](#batching) + * [Recursive](#recursive) * [Options](#options) * [Duplicate Key Ignore](#duplicate-key-ignore) * [Duplicate Key Update](#duplicate-key-update) @@ -48,6 +57,29 @@ an 18 hour batch process to <2 hrs. ### Examples +#### Introduction + +Without `activerecord-import`, you'd write something like this: + +```ruby +10.times do |i| + Book.create! :name => "book #{i}" +end +``` + +This would end up making 10 SQL calls. YUCK! With `activerecord-import`, you can instead do this: + +```ruby +```ruby +books = [] +10.times do |i| + books << Book.new(:name => "book #{i}") +end +Book.import books # or use import! +``` + +and only have 1 SQL call. Much better! + #### Columns and Arrays The @#import@ method can take an array of column names (string or symbols) and an array of arrays. Each child array represents an individual record and its list of values in the same order as the columns. This is the fastest import mechanism and also the most primitive. @@ -127,7 +159,22 @@ columns = [ :title ] # 2 INSERT statements for 4 records Book.import columns, books, :batch_size => 2 +``` +#### Recursive + +NOTE: This only works with PostgreSQL. + +Assume that Books has_many Reviews. + +```ruby +books = [] +10.times do |i| + book = Book.new(:name => "book #{i}") + book.reviews.build(:title => "Excellent") + books << book +end +Book.import books, recursive: true ``` ### Options From 225a645429ae7d87af2baea057b5d9bfa0b4bad2 Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 06:46:55 -0700 Subject: [PATCH 10/13] Fix up formatting issues from Wiki to README copy-paste --- README.markdown | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/README.markdown b/README.markdown index ad98a990..538d589e 100644 --- a/README.markdown +++ b/README.markdown @@ -82,7 +82,7 @@ and only have 1 SQL call. Much better! #### Columns and Arrays -The @#import@ method can take an array of column names (string or symbols) and an array of arrays. Each child array represents an individual record and its list of values in the same order as the columns. This is the fastest import mechanism and also the most primitive. +The `import` method can take an array of column names (string or symbols) and an array of arrays. Each child array represents an individual record and its list of values in the same order as the columns. This is the fastest import mechanism and also the most primitive. ```ruby columns = [ :title, :author ] @@ -100,7 +100,7 @@ Book.import columns, values #### ActiveRecord Models -The @#import@ method can take an array of models. The attributes will be pulled off from each model by looking at the columns available on the model. +The `import` method can take an array of models. The attributes will be pulled off from each model by looking at the columns available on the model. ```ruby books = [ @@ -118,7 +118,7 @@ Book.import books, :validate => true Book.import books ``` -The @#import@ method can take an array of column names and an array of models. The column names are used to determine what fields of data should be imported. The following example will only import books with the @:title@ field: +The `import` method can take an array of column names and an array of models. The column names are used to determine what fields of data should be imported. The following example will only import books with the `title` field: ```ruby books = [ @@ -146,7 +146,7 @@ Book.import columns, books #### Batching -The @#import@ method can take a @:batch_size@ option to control the number of rows to insert per INSERT statement. The default is the total number of records being inserted so there is a single INSERT statement. +The `import` method can take a `batch_size` option to control the number of rows to insert per INSERT statement. The default is the total number of records being inserted so there is a single INSERT statement. ```ruby books = [ @@ -198,11 +198,11 @@ The option `:on_duplicate_key_ignore` is bypassed when `:recursive` is enabled f #### Duplicate Key Update -MySQL, PostgreSQL (9.5+), and SQLite (3.24.0+) support @on duplicate key update@ (also known as "upsert") which allows you to specify fields whose values should be updated if a primary or unique key constraint is violated. +MySQL, PostgreSQL (9.5+), and SQLite (3.24.0+) support `on duplicate key update` (also known as "upsert") which allows you to specify fields whose values should be updated if a primary or unique key constraint is violated. One big difference between MySQL and PostgreSQL support is that MySQL will handle any conflict that happens, but PostgreSQL requires that you specify which columns the conflict would occur over. SQLite models its upsert support after PostgreSQL. -h2. Basic Update +Basic Update ```ruby book = Book.create! title: "Book1", author: "FooManChu" @@ -222,7 +222,7 @@ book.reload.title # => "Updated Book Title" (changed) book.reload.author # => "FooManChu" (stayed the same) ``` -h2. Using the value from another column +Using the value from another column ```ruby book = Book.create! title: "Book1", author: "FooManChu" @@ -240,7 +240,7 @@ book.reload.title # => "Book1" (stayed the same) book.reload.author # => "Updated Book Title" (changed) ``` -h2. Using Custom SQL +Using Custom SQL ```ruby book = Book.create! title: "Book1", author: "FooManChu" @@ -261,7 +261,7 @@ book.reload.title # => "Book1" (stayed the same) book.reload.author # => "Bob Barker" (changed) ``` -h2. PostgreSQL Using constraints +PostgreSQL Using constraints ```ruby book = Book.create! title: "Book1", author: "FooManChu", edition: 3, published_at: nil @@ -323,11 +323,11 @@ When running `import`, `activerecord-import` does not automatically update count If you're familiar with ActiveRecord you're probably familiar with its timestamp columns: created_at, created_on, updated_at, updated_on, etc. When importing data the timestamp fields will continue to work as expected and each timestamp column will be set. -Should you wish to specify those columns, you may use the option @timestamps: false@. +Should you wish to specify those columns, you may use the option `timestamps: false`. -However, it is also possible to set just @:created_at@ in specific records. In this case despite using @timestamps: true@, @:created_at@ will be updated only in records where that field is @nil@. Same rule applies for record associations when enabling the option @recursive: true@. +However, it is also possible to set just `:created_at` in specific records. In this case despite using `timestamps: true`, `:created_at` will be updated only in records where that field is `nil`. Same rule applies for record associations when enabling the option `recursive: true`. -If you are using custom time zones, these will be respected when performing imports as well as long as @ActiveRecord::Base.default_timezone@ is set, which for practically all Rails apps it is +If you are using custom time zones, these will be respected when performing imports as well as long as `ActiveRecord::Base.default_timezone` is set, which for practically all Rails apps it is ### Callbacks @@ -375,15 +375,15 @@ The following database adapters are currently supported: * MySQL2 - supports core import functionality plus on duplicate key update support (included in activerecord-import 0.2.0 and higher) * PostgreSQL - supports core import functionality (included in activerecord-import 0.1.0 and higher) * SQLite3 - supports core import functionality (included in activerecord-import 0.1.0 and higher) -* Oracle - supports core import functionality through DML trigger (available as an external gem: "activerecord-import-oracle_enhanced":https://github.com/keeguon/activerecord-import-oracle_enhanced) -* SQL Server - supports core import functionality (available as an external gem: "activerecord-import-sqlserver":https://github.com/keeguon/activerecord-import-sqlserver) +* Oracle - supports core import functionality through DML trigger (available as an external gem: [activerecord-import-oracle_enhanced](https://github.com/keeguon/activerecord-import-oracle_enhanced) +* SQL Server - supports core import functionality (available as an external gem: [activerecord-import-sqlserver](https://github.com/keeguon/activerecord-import-sqlserver) If your adapter isn't listed here, please consider creating an external gem as described in the README to provide support. If you do, feel free to update this wiki to include a link to the new adapter's repository! To test which features are supported by your adapter, use the following methods on a model class: -* supports_import?(*args) -* supports_on_duplicate_key_update? -* supports_setting_primary_key_of_imported_objects? +* `supports_import?(*args)` +* `supports_on_duplicate_key_update?` +* `supports_setting_primary_key_of_imported_objects?` ### Additional Adapters From 0e15c5b2aa14207f617f5ec7d9ab42b013b0ce87 Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 06:47:15 -0700 Subject: [PATCH 11/13] Add name to contributors list --- README.markdown | 1 + 1 file changed, 1 insertion(+) diff --git a/README.markdown b/README.markdown index 538d589e..a59dbbe2 100644 --- a/README.markdown +++ b/README.markdown @@ -536,3 +536,4 @@ Zach Dennis (zach.dennis@gmail.com) * Thibaud Guillaume-Gentil * Mark Van Holstyn * Victor Costan +* Dillon Welch From d0a94a21195a44a9cd6afd35d3e67981ccb87f7f Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 2 Nov 2018 14:56:53 -0700 Subject: [PATCH 12/13] Make updates on hashes examples --- README.markdown | 91 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 22 deletions(-) diff --git a/README.markdown b/README.markdown index a59dbbe2..f74741d3 100644 --- a/README.markdown +++ b/README.markdown @@ -33,6 +33,7 @@ The gem provides the following high-level features: * [Examples](#examples) * [Introduction](#introduction) * [Columns and Arrays](#columns-and-arrays) + * [Hashes](#hashes) * [ActiveRecord Models](#activerecord-models) * [Batching](#batching) * [Recursive](#recursive) @@ -40,7 +41,6 @@ The gem provides the following high-level features: * [Duplicate Key Ignore](#duplicate-key-ignore) * [Duplicate Key Update](#duplicate-key-update) * [Uniqueness Validation](#uniqueness-validation) -* [Array of Hashes](#array-of-hashes) * [Counter Cache](#counter-cache) * [ActiveRecord Timestamps](#activerecord-timestamps) * [Callbacks](#callbacks) @@ -98,6 +98,74 @@ Book.import columns, values, :validate => true Book.import columns, values ``` +#### Hashes + +The `import` method can take an array of hashes. The keys map to the column names in the database. + +```ruby +values = [{ title: 'Book1', author: 'FooManChu' }, { title: 'Book2', author: 'Bob Jones'}] + +# Importing without model validations +Book.import values, validate: false + +# Import with model validations +Book.import values, validate: true + +# when not specified :validate defaults to true +Book.import values +``` +h2. Import Using Hashes and Explicit Column Names + +The `import` method can take an array of column names and an array of hash objects. The column names are used to determine what fields of data should be imported. The following example will only import books with the `title` field: + +```ruby +books = [ + { title: "Book 1", author: "FooManChu" }, + { title: "Book 2", author: "Bob Jones" } +] +columns = [ :title ] + +# without validations +Book.import columns, books, validate: false + +# with validations +Book.import columns, books, validate: true + +# when not specified :validate defaults to true +Book.import columns, books + +# result in table books +# title | author +#--------|-------- +# Book 1 | NULL +# Book 2 | NULL + +``` + +Using hashes will only work if the columns are consistent in every hash of the array. If this does not hold, an exception will be raised. There are two workarounds: use the array to instantiate an array of ActiveRecord objects and then pass that into `import` or divide the array into multiple ones with consistent columns and import each one separately. + +See https://github.com/zdennis/activerecord-import/issues/507 for discussion. + +```ruby +arr = [ + { bar: 'abc' }, + { baz: 'xyz' }, + { bar: '123', baz: '456' } +] + +# An exception will be raised +Foo.import arr + +# better +arr.map! { |args| Foo.new(args) } +Foo.import arr + +# better... though somewhat defeats the purpose of activerecord-import +Foo.import [{ bar: 'abc' }] +Foo.import [{ baz: 'xyz' }] +Foo.import [{ bar: '123', baz: '456' }] +``` + #### ActiveRecord Models The `import` method can take an array of models. The attributes will be pulled off from each model by looking at the columns available on the model. @@ -291,27 +359,6 @@ By default, `activerecord-import` will not validate for uniquness when importing Book.import books, validate_uniqueness: true ``` -### Array of Hashes - -Due to the counter-intuitive behavior that can occur when dealing with hashes instead of ActiveRecord objects, `activerecord-import` will raise an exception when passed an array of hashes. If you have an array of hash attributes, you should instead use them to instantiate an array of ActiveRecord objects and then pass that into `import`. - -See https://github.com/zdennis/activerecord-import/issues/507 for discussion. - -```ruby -arr = [ - { bar: 'abc' }, - { baz: 'xyz' }, - { bar: '123', baz: '456' } -] - -# An exception will be raised -Foo.import arr - -# better -arr.map! { |args| Foo.new(args) } -Foo.import arr -``` - ### Counter Cache When running `import`, `activerecord-import` does not automatically update counter cache columns. To update these columns, you will need to do one of the following: From 7aad08d78cb51d0858a7b23c3a8e1fedd29de5c5 Mon Sep 17 00:00:00 2001 From: Dillon Welch Date: Fri, 9 Nov 2018 10:58:51 -0800 Subject: [PATCH 13/13] Make another update on hash examples --- README.markdown | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.markdown b/README.markdown index f74741d3..0b9a8968 100644 --- a/README.markdown +++ b/README.markdown @@ -160,10 +160,10 @@ Foo.import arr arr.map! { |args| Foo.new(args) } Foo.import arr -# better... though somewhat defeats the purpose of activerecord-import -Foo.import [{ bar: 'abc' }] -Foo.import [{ baz: 'xyz' }] -Foo.import [{ bar: '123', baz: '456' }] +# better +arr.group_by(&:keys).each_value do |v| + Foo.import v +end ``` #### ActiveRecord Models