GORM V2 Release Note Draft

GORM 2.0 Release Note (Draft)

GORM 2.0 is rewritten from scratch based on feedback we received in the last few years, it introduces some incompatible-API change.

GORM 2.0 is not yet released, currently, in the public beta stage, it has been used stably in few production services, but we are still actively collecting user suggestions, feedbacks to achieve a better GORM V2 before the final release, If everything goes well, the final release date will be the time we reach 20k stars!

(The release note is still work-in-progress, it contains most major changes, for details, please checkout http://gorm.io when we finish its rewritten)

Highlights

• Performance
• Modularity
• Context, Batch Insert, Prepared Statment, DryRun Mode, Join Preload, Find To Map, FindInBatches
• SavePoint/RollbackTo/Nested Transaction Support
• Association improvements (On Delete/Update), Modify Join Table for Many2Many, Association Mode for batch data
• SQL Builder, Upsert, Locking, Optimizer/Index/Comment Hints supports
• Multiple fields support for auto creating/updating time, which also support unix (nano) seconds
• Field permissions support: readonly, writeonly, createonly, updateonly, ignored
• All new Migrator, Logger
• Naming strategy (Unified table name, field name, join table name, foreign key, checker, index name rule)
• Better customized data type support (e.g: JSON)
• All new plugin system, Hooks API

Upgrading？

• GORM's developments moved to github.com/go-gorm, and the import path changed to gorm.io/gorm, for previous projects, you can keep using github.com/jinzhu/gorm
• Database drivers have been split into separate projects, e.g: github.com/go-gorm/sqlserver, and its import path also changed

Install

go get gorm.io/gorm@v0.2.13  // public beta version

Usage

import (
  "gorm.io/gorm"
  "gorm.io/driver/sqlite"
  "gorm.io/driver/mysql"
  "gorm.io/driver/postgres"
  "gorm.io/driver/sqlserver"
)

func init() {
  // during initialization, you can use `gorm.Config` to change few configurations, e.g: NowFunc, which no longer allow modification through global variables
  db, err := gorm.Open(sqlite.Open("gorm.db"), &gorm.Config{NowFunc: func() time.Time { return time.Now().Local() }})
  db, err := gorm.Open(mysql.Open("gorm:gorm@tcp(localhost:9910)/gorm?charset=utf8&parseTime=True"), &gorm.Config{})
  db, err := gorm.Open(postgres.Open("user=gorm password=gorm dbname=gorm port=9920 TimeZone=Asia/Shanghai"), &gorm.Config{})

  // most CRUD API kept compatibility
  db.AutoMigrate(&Product{})
  db.Create(&user)
  db.First(&user, 1)
  db.Model(&user).Update("Age", 18)
  db.Model(&user).Omit("Role").Updates(map[string]interface{}{"Name": "jinzhu", "Role": "admin"})
  db.Delete(&user)
}

Major Features

Context Support

• All database operations support context with WithContext method

// Single session mode
DB.WithContext(ctx).Find(&users)

// Continuous session mode
tx := DB.WithContext(ctx)
tx.First(&user, 1)
tx.Model(&user).Update("role", "admin")

• Logger accepts context for tracing

Batch Insert

• Just pass slice data to Create, GORM will generate a single SQL statement to insert all the data and backfill primary key values
• If the data contains associations, all new association objects will be inserted with another SQL
• Batch inserted data will call its Hooks methods (Before/After Create/Save)

var users = []User{{Name: "jinzhu1"}, {Name: "jinzhu2"}, {Name: "jinzhu3"}}
DB.Create(&users)

for _, user := range users {
  user.ID // 1,2,3
}

Prepared Statment Mode

• Prepared Statement Mode creates prepared stmt for executed SQL and caches them to speed up future calls
• Prepared Statement Mode can be used globally or a session

// globally mode, all operations will create prepared stmt and cache to speed up
db, err := gorm.Open(sqlite.Open("gorm.db"), &gorm.Config{PrepareStmt: true})

// continuous session mode
tx := DB.Session(&Session{PrepareStmt: true})
tx.First(&user, 1)
tx.Find(&users)
tx.Model(&user).Update("Age", 18)

DryRun Mode

Generate SQL without executing, can be used to check or test generated SQL

// globally mode
db, err := gorm.Open(sqlite.Open("gorm.db"), &gorm.Config{DryRun: true})

stmt := db.Find(&user, 1).Statement
stmt.SQL.String() //=> SELECT * FROM `users` WHERE `id` = $1 // PostgreSQL
stmt.SQL.String() //=> SELECT * FROM `users` WHERE `id` = ?  // MySQL
stmt.Vars         //=> []interface{}{1}

// session mode
stmt := DB.Session(&Session{DryRun: true}).First(&user, 1).Statement
stmt.SQL.String() //=> SELECT * FROM `users` WHERE `id` = $1 ORDER BY `id`
stmt.Vars         //=> []interface{}{1}

Join Preload

• Preload loads the association data in a separate query, Join Preload will loads association data using inner join
• It can handle null association

DB.Joins("Company").Joins("Manager").Joins("Account").First(&user, 1)
DB.Joins("Company").Joins("Manager").Joins("Account").First(&user, "users.name = ?", "jinzhu")
DB.Joins("Company").Joins("Manager").Joins("Account").Find(&users, "users.id IN ?", []int{1,2,3,4,5})

Find To Map

Support scan result to map[string]interface{} or []map[string]interface{}

var result map[string]interface{}
DB.Model(&User{}).First(&result, "id = ?", 1)

var results []map[string]interface{}
DB.Model(&User{}).Find(&results)

FindInBatches

Query and process records in batch

// batch size 100
result := DB.Where("processed = ?", false).FindInBatches(&results, 100, func(tx *gorm.DB, batch int) error {
  // batch processing
  for _, result := range results {
    result.Processed = true
  }
  DB.Save(&results)

  tx.RowsAffected // number of records in this batch

  batch // Batch 1, 2, 3

  // returns error will stop future batches
  return nil
})

result.Error // returned error
result.RowsAffected // number of records in all batches

SavePoint, RollbackTo Support

tx := DB.Begin()
tx.Create(&user1)

tx.SavePoint("sp1")
tx.Create(&user2)
tx.RollbackTo("sp1") // Rollback user2

tx.Commit() // Commit user1

Nested Transaction Support

DB.Transaction(func(tx *gorm.DB) error {
  tx.Create(&user1)

  tx.Transaction(func(tx2 *gorm.DB) error {
    tx.Create(&user2)
    return errors.New("rollback user2") // Rollback user2
  })

  tx.Transaction(func(tx2 *gorm.DB) error {
    tx.Create(&user3)
    return nil
  })

  return nil
})

// Commit user1, user3

Association improvements

• Belongs To, Single-Table Belongs To, Has One, Has One Polymorphic, Has Many, Has Many Polymorphic, Single-Table Has Many, Many2Many, Single-Table Many2Many re-implement, fixed some edge case issues
• Easier to use tag to specify foreign keys (when not following the naming convention)

// Belongs To: `ForeignKey` specifies foreign key field owned by the current model, `References` specifies the association's primary key
// Has One/Many: `ForeignKey` specifies foreign key for the association, `References` specifies the current model's primary key
// Many2Many: `ForeignKey` specifies the current model's primary key, `JoinForeignKey` specifies join table's foreign key that refers to `ForeignKey`
//            `References` specifies the association's primary key, `JoinReferences` specifies join table's foreign key that refers to `References`
// For multiple foreign keys, it can be separated by commas

type Profile struct {
  gorm.Model
  Refer string
  Name  string
}

type User struct {
  gorm.Model
  Profile   Profile `gorm:"ForeignKey:ProfileID;References:Refer"`
  ProfileID int
}

// Many2Many with multiple primary keys
type Tag struct {
  ID     uint   `gorm:"primary_key"`
  Locale string `gorm:"primary_key"`
  Value  string
}

type Blog struct {
  ID         uint   `gorm:"primary_key"`
  Locale     string `gorm:"primary_key"`
  Subject    string
  Body       string
  // Multiple foreign keys using all primary fields
  Tags       []Tag `gorm:"many2many:blog_tags;"`
  // Using `ID` as foreign key
  SharedTags []Tag `gorm:"many2many:shared_blog_tags;ForeignKey:id;References:id"`
  // Using `ID`, `Locale` as foreign key for Blog, Using `ID` as foreign key for Tag
  LocaleTags []Tag `gorm:"many2many:locale_blog_tags;ForeignKey:id,locale;References:id"`
}

Association On Delete/Update Support

type Profile struct {
  gorm.Model
  Refer string
  Name  string
}

type User struct {
  gorm.Model
  Profile   Profile `gorm:"Constraint:OnUpdate:CASCADE,OnDelete:CASCADE;"`
  ProfileID int
}

clause.Associations with Select, Omit, Preload

// Preload all associations when query user
DB.Preload(clause.Associations).First(&user)

// Skip save all associations when creating user
DB.Omit(clause.Associations).Create(&user)

Modify Join Table for Many2Many

Easier to setup Many2Many's JoinTable，the JoinTable can be a full-featured model, like having Soft Delete，Hooks supports, and define more fields

type Person struct {
  ID        int
  Name      string
  Addresses []Address `gorm:"many2many:person_addresses;"`
}

type Address struct {
  ID   uint
  Name string
}

type PersonAddress struct {
  PersonID  int
  AddressID int
  CreatedAt time.Time
  DeletedAt gorm.DeletedAt
}

func (PersonAddress) BeforeCreate(db *gorm.DB) error {
  // ...
}

// PersonAddress must defined all required foreign keys, or it will raise error
err := DB.SetupJoinTable(&Person{}, "Addresses", &PersonAddress{})

Association Mode

• Deleted method Related to avoid some usage issue, replace it with gorm.Model(&user).Association("Pets").Find(&pets)
• Association supports batch data, e.g:

// Find all roles for all users
gorm.Model(&users).Association("Role").Find(&roles)

// Delete User A from all users's team
gorm.Model(&users).Association("Team").Delete(&userA)

// Unduplicated count of members in all user's team
gorm.Model(&users).Association("Team").Count()

// For `Append`, `Replace` with batch data, arguments's length need to equal to data's length
var users = []User{user1, user2, user3}
// e.g: we have 3 users, Append userA to user1's team, append userB to user2's team, append userA, userB and userC to user3's team
gorm.Model(&users).Association("Team").Append(&userA, &userB, &[]User{userA, userB, userC})
// Reset user1's team to userA，reset user2's team to userB, reset user3's team to userA, userB and userC
gorm.Model(&users).Association("Team").Replace(&userA, &userB, &[]User{userA, userB, userC})

Multiple fields support for auto creating/updating time, which also support unix (nano) seconds (when the data type is int)

type User struct {
  CreatedAt time.Time // Set to current time when it is zero on creating
  UpdatedAt int       // Set to current time on updaing or when it is zero on creating
  Updated   int64 `gorm:"autoupdatetime:nano"` // Use unix nano seconds as updating time
  Created   int64 `gorm:"autocreatetime"` // Use unix seconds as creating time
}

Field permissions support: readonly, writeonly, createonly, updateonly, ignored

type User struct {
  Name string `gorm:"<-:create"` // allow read and create
  Name string `gorm:"<-:update"` // allow read and update
  Name string `gorm:"<-"`        // allow read and write (create and update)
  Name string `gorm:"->:false;<-:create"` // createonly
  Name string `gorm:"->"` // readonly
  Name string `gorm:"-"`  // ignored
}

All new Migrator

• Migrator will create database foreign keys (use DisableForeignKeyConstraintWhenMigrating during initialization to disable this feature) (and ignore or delete foreign key constraint when DropTable)
• Migrator is more independent, providing better support for each database and unified API interfaces. we can design better migrate tools based on it (for example SQLite doesn't support ALTER COLUMN, DROP COLUMN, GORM will create a new table as the one you are trying to change, copy all data, drop the old table, rename the new table)
• Support to set check constraint through the tag
• Enhanced tag setting for index

type UserIndex struct {
  Name  string `gorm:"check:named_checker,(name <> 'jinzhu')"`
  Name2 string `gorm:"check:(age > 13)"`
  Name4 string `gorm:"index"`
  Name5 string `gorm:"index:idx_name,unique"`
  Name6 string `gorm:"index:,sort:desc,collate:utf8,type:btree,length:10,where:name3 != 'jinzhu'"`
}

Naming Strategy

You can setup naming strategy (table name, field name, join table name, foreign key, checker, index name) during initialization, which is convenient to modify the naming rules (such as adding table name prefix), for details, please refer to https://github.com/go-gorm/gorm/blob/master/schema/naming.go#L14

db, err := gorm.Open(sqlite.Open("gorm.db"), &gorm.Config{
  NamingStrategy: schema.NamingStrategy{TablePrefix: "t_", SingularTable: true},
})

Logger

In addition to the context support mentioned above, Logger has also optimized the following:

• Customize/turn off the colors in the log
• Slow SQL log, default slow SQL time is 100ms
• Optimized the SQL log format so that it can be copied and executed in a database console

Safely update, delete. do update/delete without any conditions would be prohibited

DB.Delete(&User{}) // returns error
DB.Model(&User{}).Update("role", "admin") // returns error

DB.Where("1=1").Delete(&User{}) // delete all records
DB.Model(&User{}).Where("1=1").Update("role", "admin") // update all records

Transaction Mode

By default, all GORM write operations run inside a transaction to ensure data consistency, you can disable it during initialization if it is not required

db, err := gorm.Open(sqlite.Open("gorm.db"), &gorm.Config{SkipDefaultTransaction: true})

Hooks

Before/After Create/Update/Save/Find/Delete must be defined as a method of type func(tx *gorm.DB) error, if defined as other types, a warning log will be printed and it won't take effect

func (user *User) BeforeCreate(tx *gorm.DB) error {
  // Modify current operation through tx.Statement, e.g:
  tx.Statement.Select("Name", "Age")
  tx.Statement.AddClause(clause.OnConflict{DoNothing: true})

  // Operations based on tx will runs inside same transaction without clauses of current one
  var role Role
  err := tx.First(&role, "name = ?", user.Role).Error // SELECT * FROM roles WHERE name = "admin"
  return err
}

Hooks can modify updating value without SetColumn

You can modify updating value without using SetColumn anymore (Note: need to declare hook methods with pointer receivers to modify its value)

func (user *User) BeforeUpdate(tx *gorm.DB) error {
  user.Age = 18
  return nil
}

TableName method

TableName will not allow dynamic table name, the return value of TableName will be cached for future

func (User) TableName() string {
  return "t_user"
}

Better customized data type support (JSON as an example)

GORM optimizes support for custom types, so you can define a data structure to support all databases

The following takes JSON as an example (supported MySQL, Postgres, refer: https://github.com/go-gorm/datatypes/blob/master/json.go)

import "gorm.io/datatypes"

type User struct {
  gorm.Model
  Name       string
  Attributes datatypes.JSON
}

DB.Create(&User{
  Name:       "jinzhu",
  Attributes: datatypes.JSON([]byte(`{"name": "jinzhu", "age": 18, "tags": ["tag1", "tag2"], "orgs": {"orga": "orga"}}`)),
}

// Query user having a role field in attributes
DB.First(&user, datatypes.JSONQuery("attributes").HasKey("role"))
// Query user having orgs->orga field in attributes
DB.First(&user, datatypes.JSONQuery("attributes").HasKey("orgs", "orga"))

Omit, Select optimizes

// When creating objects
DB.Select("Name", "Age").Create(&user) // Using the fields specified by Select
DB.Omit([]string{"Name", "Age"}).Create(&user) // Ignoring the field specified by Omit

// When updating objects
DB.Model(&user).Select("Name", "Age").Updates(map[string]interface{}{"name": "jinzhu", "age": 18, "role": "admin"})
DB.Model(&user).Omit([]string{"Role"}).Update(User{Name: "jinzhu", Role: "admin"})

SQL Builder

• GORM uses SQL Builder generates SQL internally
• When performing an operation, GORM creates a Statement object, all APIs add/change Clause for the Statement, at last, GORM generated SQL based on those clauses
• For different databases, Clauses may generate different SQL

Group Conditions

db.Where(
  db.Where("pizza = ?", "pepperoni").Where(db.Where("size = ?", "small").Or("size = ?", "medium")),
).Or(
  db.Where("pizza = ?", "hawaiian").Where("size = ?", "xlarge"),
).Find(&pizzas)

// SELECT * FROM pizzas WHERE (pizza = 'pepperoni' AND (size = 'small' OR size = 'medium')) OR (pizza = 'hawaiian' AND size = 'xlarge')

Upsert

clause.OnConflict provides compatible Upsert support for different databases (SQLite, MySQL, PostgreSQL, SQL Server)

import "gorm.io/gorm/clause"

// Do nothing on conflict
DB.Clauses(clause.OnConflict{DoNothing: true}).Create(&users)

DB.Clauses(clause.OnConflict{
  Columns:   []clause.Column{{Name: "id"}},
  DoUpdates: clause.Assignments(map[string]interface{}{"name": "jinzhu", "age": 18}),
}).Create(&users)
// MERGE INTO "users" USING *** WHEN NOT MATCHED THEN INSERT *** WHEN MATCHED THEN UPDATE SET ***; SQL Server
// INSERT INTO `users` *** ON DUPLICATE KEY UPDATE ***; MySQL

DB.Clauses(clause.OnConflict{
  Columns:   []clause.Column{{Name: "id"}},
  DoUpdates: clause.AssignmentColumns([]string{"name", "age"}),
}).Create(&users)
// MERGE INTO "users" USING *** WHEN NOT MATCHED THEN INSERT *** WHEN MATCHED THEN UPDATE SET "name"="excluded"."name"; SQL Server
// INSERT INTO "users" *** ON CONFLICT ("id") DO UPDATE SET "name"="excluded"."name", "age"="excluded"."age"; PostgreSQL
// INSERT INTO `users` *** ON DUPLICATE KEY UPDATE `name`=VALUES(name),`age=VALUES(age); MySQL

Locking

DB.Clauses(clause.Locking{Strength: "UPDATE"}).Find(&users) // SELECT * FROM `users` FOR UPDATE

DB.Clauses(clause.Locking{
  Strength: "SHARE",
  Table: clause.Table{Name: clause.CurrentTable},
}).Find(&users)
// SELECT * FROM `users` FOR SHARE OF `users`

Optimizer/Index/Comment Hints

import "gorm.io/hints"

// Optimizer Hints
DB.Clauses(hints.New("hint")).Find(&User{})
// SELECT * /*+ hint */ FROM `users`

// Index Hints
DB.Clauses(hints.UseIndex("idx_user_name")).Find(&User{})
// SELECT * FROM `users` USE INDEX (`idx_user_name`)

DB.Clauses(hints.ForceIndex("idx_user_name", "idx_user_id").ForJoin()).Find(&User{})
// SELECT * FROM `users` FORCE INDEX FOR JOIN (`idx_user_name`,`idx_user_id`)"

DB.Clauses(
	hints.ForceIndex("idx_user_name", "idx_user_id").ForOrderBy(),
	hints.IgnoreIndex("idx_user_name").ForGroupBy(),
).Find(&User{})
// SELECT * FROM `users` FORCE INDEX FOR ORDER BY (`idx_user_name`,`idx_user_id`) IGNORE INDEX FOR GROUP BY (`idx_user_name`)"

// Comment Hints
DB.Clauses(hints.Comment("select", "master")).Find(&User{})
// SELECT /*master*/ * FROM `users`;

DB.Clauses(hints.CommentBefore("insert", "node2")).Create(&user)
// /*node2*/ INSERT INTO `users` ...;

DB.Clauses(hints.CommentAfter("select", "node2")).Create(&user)
// /*node2*/ INSERT INTO `users` ...;

DB.Clauses(hints.CommentAfter("where", "hint")).Find(&User{}, "id = ?", 1)
// SELECT * FROM `users` WHERE id = ? /* hint */

New Plugin System API

Database drivers have been split into separate projects, they need to implement the gorm.Dialector interface, then we can use it with gorm.Open(dialector, &gorm.Config{}) to initialize gorm.DB

During initialization, database driver can register, modify or delete Callbacks method (create, query, update, delete, row, raw) of *gorm.DB, method type needs to be func(db *gorm.DB), refer to following for details:

• Sqlite initialization process: https://github.com/go-gorm/sqlite/blob/master/sqlite.go
• Register Callbacks: https://github.com/go-gorm/gorm/blob/master/callbacks/callbacks.go

Happy Hacking!