Skip to content

Commit

Permalink
feat: add MutatePresentColumns method
Browse files Browse the repository at this point in the history
Context:
Spanner has a hard limit on size of a mutation that is 20k cells, the
product of rows and columns. Schemas for protobuf messages containing
oneofs could be mapped to large number of nullable fields in Spanner.

When inserting rows to this schema, the majority of these nullable
fields will not be set, and setting the column to null is equivalent to
not setting it at all. This method, not setting columns at all,
drastically lowers the number of cells mutated in a mutation.

---

This change adds `row.MutatePresentColumns` which only mutates nullable
fields that are present.

* Nullable scalars are considered present if `column.Valid == true`
* Nullable arrays are considered present if `len(column) != 0`
* Nullable bytes are considered present if `len(colum) != 0`
  • Loading branch information
ericwenn committed Mar 30, 2021
1 parent ce6d1bb commit 8d69ab6
Show file tree
Hide file tree
Showing 10 changed files with 389 additions and 0 deletions.
41 changes: 41 additions & 0 deletions internal/codegen/databasecodegen/row.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ func (g RowCodeGenerator) GenerateCode(f *codegen.File) {
g.generateUnmarshalFunction(f)
g.generateMutationFunction(f)
g.generateMutationForColumnsFunction(f)
g.generateMutationForPresentColumnsFunction(f)
g.generatePrimaryKeyMethod(f)
}

Expand Down Expand Up @@ -171,6 +172,46 @@ func (g RowCodeGenerator) generateMutationForColumnsFunction(f *codegen.File) {
f.P("}")
}

func (g RowCodeGenerator) generateMutationForPresentColumnsFunction(f *codegen.File) {
f.P()
f.P("func (r *", g.Type(), ") MutatePresentColumns() (string, []string, []interface{}) {")
f.P("columns := make([]string, 0, len(r.", g.ColumnNamesMethod(), "()))")
// non-nullable fields
f.P("columns = append(")
f.P("columns,")
for _, column := range g.Table.Columns {
if column.NotNull {
f.P(strconv.Quote(string(column.Name)), ",")
}
}
f.P(")")
// nullable fields
for _, column := range g.Table.Columns {
if column.NotNull {
continue
}
f.P("if ", g.isPresentPredicate(column), " {")
f.P("columns = append(columns, ", strconv.Quote(string(column.Name)), ")")
f.P("}")
}
f.P("return r.MutateColumns(columns)")
f.P("}")
}

func (g RowCodeGenerator) isPresentPredicate(column *spanddl.Column) string {
switch {
case column.Type.Array:
return "len(r." + g.ColumnFieldName(column) + ") != 0"
case column.Type.Base == spansql.Bytes:
return "len(r." + g.ColumnFieldName(column) + ") != 0"
case !column.NotNull:
return "!r." + g.ColumnFieldName(column) + ".IsNull()"
default:
// columns that are non-nullable are always considered present
return "true"
}
}

func (g RowCodeGenerator) generateColumnNamesFunctions(f *codegen.File) {
spansqlPkg := f.Import("cloud.google.com/go/spanner/spansql")
f.P()
Expand Down
18 changes: 18 additions & 0 deletions internal/codegen/databasecodegen/testdata/1.sql.database.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

31 changes: 31 additions & 0 deletions internal/codegen/databasecodegen/testdata/2.sql.database.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

45 changes: 45 additions & 0 deletions internal/codegen/databasecodegen/testdata/3.sql.database.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

59 changes: 59 additions & 0 deletions internal/codegen/databasecodegen/testdata/4.sql.database.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions internal/codegen/databasecodegen/testdata/5.sql.database.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions internal/codegen/databasecodegen/testdata/6.sql.database.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 17 additions & 0 deletions internal/codegen/databasecodegen/testdata/7.sql.database.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 8d69ab6

Please sign in to comment.