The library implements a simple key-value abstraction to store algebraic, linked-data data types at AWS storage services: AWS DynamoDB and AWS S3.
The library encourages developers to use Golang struct to define domain models, write correct, maintainable code. Using this library, the application can achieve the ideal data model that would require a single request to DynamoDB and model one-to-one, one-to-many and even many-to-many relations. The library uses generic programming style to implement actual storage I/O, while expose external domain object as [T dynamo.Thing] with implicit conversion back and forth between a concrete struct(s). The library uses AWS Golang SDK v2 under the hood.
Essentially, the library implement a following generic key-value trait to access domain objects.
type KeyVal[T dynamo.Thing] interface {
Put(T) error
Get(T) (T, error)
Remove(T) (T, error)
Update(T) (T, error)
Match(T) ([]T, error)
}The library philosophy and use-cases are covered in depth at the post How To Model Any Relational Data in DynamoDB With dynamo library or continue reading the Getting started section.
The library requires Go 1.18 or later due to usage of generics.
The latest version of the library is available at its main branch. All development, including new features and bug fixes, take place on the main branch using forking and pull requests as described in contribution guidelines. The stable version is available via Golang modules.
- dynamo
- Inspiration
- Getting started
- How To Contribute
- License
Data types definition is an essential part of development with dynamo library. Golang structs declares domain of your application. Public fields are serialized into DynamoDB attributes, the field tag dynamodbav controls marshal/unmarshal processes.
The library demands from each structure implementation of Thing interface. This type acts as struct annotation -- Golang compiler raises an error at compile time if other data type is supplied to the dynamo library. Secondly, each structure defines unique "composite primary key". The library encourages definition of both partition and sort keys using a special data type curie.IRI. This type is a synonym to compact Internationalized Resource Identifiers, which facilitates linked-data, hierarchical structures and cheap relations between data items. curie.IRI is a synonym to the built-in string type so that anything castable to string suite to model the keys as alternative solution.
type Person struct {
Org curie.IRI `dynamodbav:"prefix,omitempty"`
ID curie.IRI `dynamodbav:"suffix,omitempty"`
Name string `dynamodbav:"name,omitempty"`
Age int `dynamodbav:"age,omitempty"`
Address string `dynamodbav:"address,omitempty"`
}
//
// Identity implements thing interface
func (p Person) HashKey() curie.IRI { return p.Org }
func (p Person) SortKey() curie.IRI { return p.ID }
//
// this data type is a normal Golang struct
// just create an instance, fill required fields
var person := Person{
Org: curie.IRI("University:Kiel"),
ID: curie.IRI("Professor:8980789222"),
Name: "Verner Pleishner",
Age: 64,
Address: "Blumenstrasse 14, Berne, 3013",
}This is it! Your application is ready to read/write data to/form DynamoDB tables.
Please see and try examples. Its cover all basic use-cases with runnable code snippets, check the post How To Model Any Relational Data in DynamoDB With dynamo library for deep-dive into library philosophy.
go run examples/keyval/main.go ddb:///my-tableThe following code snippet shows a typical I/O patterns
import (
"github.com/fogfish/dynamo/v2/service/ddb"
)
//
// Create dynamodb client and bind it with the table.
// The client is type-safe and support I/O with a single type (e.g. Person).
// Use options to specify params.
db, err := ddb.New[Person](ddb.WithTable("my-table"))
//
// Write the struct with Put
if err := db.Put(context.TODO(), person); err != nil {
}
//
// Lookup the struct using Get. This function takes input structure as key
// and return a new copy upon the completion. The only requirement - ID has to
// be defined.
val, err := db.Get(context.TODO(),
Person{
Org: curie.IRI("University:Kiel"),
ID: curie.IRI("Professor:8980789222"),
},
)
switch {
case nil:
// success
case recoverNotFound(err):
// not found
default:
// other i/o error
}
//
// Apply a partial update using Update function. This function takes
// a partially defined structure, patches the instance at storage and
// returns remaining attributes.
val, err := db.Update(context.TODO(),
Person{
Org: curie.IRI("University:Kiel"),
ID: curie.IRI("Professor:8980789222"),
Address: "Viktoriastrasse 37, Berne, 3013",
}
)
if err != nil { /* ... */ }
//
// Remove the struct using Remove give partially defined struct with ID
_, err := db.Remove(context.TODO(),
Person{
Org: curie.IRI("University:Kiel"),
ID: curie.IRI("Professor:8980789222"),
}
)
if err != nil { /* ... */ }The library enforces for "assert errors for behavior, not type" as the error handling strategy, see the post for details.
Use following behaviors to recover from errors:
type ErrorCode interface{ ErrorCode() string }
type NotFound interface { NotFound() string }
type PreConditionFailed interface { PreConditionFailed() bool }
type Conflict interface { Conflict() bool }
type Gone interface { Gone() bool }The library support definition of A ⟼ B relation for data elements. Let's consider message threads as a classical examples for such hierarchies:
A
├ B
├ C
│ ├ D
│ └ E
│ └ F
└ G
Composite sort key is core concept to organize hierarchies. It facilitates linked-data, hierarchical structures and cheap relations between data items. An application declares node path using composite sort key design pattern. For example, the root is thread:A, 2nd rank node ⟨thread:A, B⟩, 3rd rank node ⟨thread:A, C/D⟩ and so on ⟨thread:A, C/E/F⟩. Each id declares partition and sub nodes. The library implement a Match function, supply the node identity and it returns sequence of child elements.
//
// Match uses partition key to match DynamoDB entries.
// It returns a sequence of matched data elements.
db.Match(context.TODO(), Message{Thread: "thread:A"})
//
// Match uses partition key and partial sort key to match DynamoDB entries.
db.Match(context.TODO(), Message{Thread: "thread:A", ID: "C"})See advanced example for details on managing linked-data.
Hierarchical structures is the way to organize collections, lists, sets, etc. The Match returns a lazy Sequence that represents your entire collection. Sometimes, your need to split the collection into sequence of pages.
// 1. Set the limit on the stream
seq, cursor, err := db.Match(context.TODO(),
Message{Thread: "thread:A", ID: "C"},
dynamo.Limit(25),
)
// 2. Continue I/O with a new stream, supply the cursor
seq := db.Match(context.TODO(),
Message{Thread: "thread:A", ID: "C"},
dynamo.Limit(25),
cursor,
)Cross-linking of structured data is an essential part of type safe domain driven design. The library helps developers to model relations between data instances using familiar data type.
type Person struct {
Org curie.IRI `dynamodbav:"prefix,omitempty"`
ID curie.IRI `dynamodbav:"suffix,omitempty"`
Leader *curie.IRI `dynamodbav:"leader,omitempty"`
}ID and Leader are sibling, equivalent data types. ID is only used as primary identity, Leader is a "pointer" to linked-data. The library advices usage of compact Internationalized Resource Identifiers (curie.IRI) for this purpose. Semantic Web publishes structured data using this type so that it can be interlinked by applications.
Often, there is an established system of the types in the application. It is not convenient to inject dependencies to the dynamo library. Also, the usage of secondary indexes requires multiple projections of core type.
//
// original core type
type Person struct {
Org string `dynamodbav:"prefix,omitempty"`
ID string `dynamodbav:"suffix,omitempty"`
Name string `dynamodbav:"name,omitempty"`
Age int `dynamodbav:"age,omitempty"`
Country string `dynamodbav:"country,omitempty"`
}
//
// the core type projection that uses ⟨Org, ID⟩ as composite key
// e.g. this projection supports writes to DynamoDB table
type dbPerson Person
func (p dbPerson) HashKey() curie.IRI { return curie.IRI(p.Org) }
func (p dbPerson) SortKey() curie.IRI { return curie.IRI(p.ID) }
//
// the core type projection that uses ⟨Org, Name⟩ as composite key
// e.g. the projection support lookup of employer
type dbNamedPerson Person
func (p dbNamedPerson) HashKey() curie.IRI { return curie.IRI(p.Org) }
func (p dbNamedPerson) SortKey() curie.IRI { return curie.IRI(p.Name) }
//
// the core type projection that uses ⟨Country, Name⟩ as composite key
type dbCitizen Person
func (p dbCitizen) HashKey() curie.IRI { return curie.IRI(p.Country) }
func (p dbCitizen) SortKey() curie.IRI { return curie.IRI(p.Name) }Development of complex Golang application might lead developers towards Standard Package Layout. It becomes extremely difficult to isolate dependencies from core data types to this library and AWS SDK. The library support serialization of core type to dynamo using custom codecs
/*** core.go ***/
// 1. complex domain type is defined
type ID struct {/* ... */}
// 2. structure with core types is defined, no deps to dynamo library
type Person struct {
Org ID `dynamodbav:"prefix,omitempty"`
ID ID `dynamodbav:"suffix,omitempty"`
}
/*** aws/ddb/ddb.go ***/
import (
"github.com/fogfish/dynamo/service/ddb"
)
// 3. declare codecs for complex core domain type
type id core.ID
func (id) MarshalDynamoDBAttributeValue() (types.AttributeValue, error) {
/* ...*/
}
func (*id) UnmarshalDynamoDBAttributeValue(types.AttributeValue) error {
/* ...*/
}
// aws/ddb/ddb.go
// 2. type alias to core type implements dynamo custom codec
type dbPerson Person
// 3. custom codec for structure field is defined
var (
codecHashKey = ddb.Codec[dbPerson, id]("Org")
codecSortKey = ddb.Codec[dbPerson, id]("ID")
)
// 4. use custom codec
func (p dbPerson) MarshalDynamoDBAttributeValue() (types.AttributeValue, error) {
type tStruct dbPerson
return ddb.Encode(av, tStruct(p),
codecHashKey.Encode(id(p.Org)),
codecSortKey.Encode(id(p.ID))),
)
}
func (x *dbPerson) UnmarshalDynamoDBAttributeValue(av types.AttributeValue) error {
type tStruct *dbPerson
return ddb.Decode(av, tStruct(x),
codecHashKey.Decode((*id)(&x.Org)),
codecSortKey.Decode((*id)(&x.ID))),
)
}In Amazon DynamoDB, there is a concept of expressions to denote the attributes to read; indicate various conditions while doing I/O.
Projection expression defines attributes to be read from the table. The library automatically defines projection expression for each request. The expression is derived from the datatype definition.
// The type projects: prefix, suffix & name attributes.
type Identity struct {
Org curie.IRI `dynamodbav:"prefix,omitempty"`
ID curie.IRI `dynamodbav:"suffix,omitempty"`
Name string `dynamodbav:"name,omitempty"`
}
// The type projects: prefix, suffix, name, age & address.
type Person struct {
Org curie.IRI `dynamodbav:"prefix,omitempty"`
ID curie.IRI `dynamodbav:"suffix,omitempty"`
Name string `dynamodbav:"name,omitempty"`
Age int `dynamodbav:"age,omitempty"`
Address string `dynamodbav:"address,omitempty"`
}Condition expression helps to implement conditional manipulation of items. The expression defines boolean predicate to determine which items should be modified. If the condition expression evaluates to true, the operation succeeds; otherwise, the operation fails. The library defines a special type Schema, which translates a Golang declaration into DynamoDB syntax:
type Person struct {
Name string `dynamodbav:"name,omitempty"`
}
// defines the builder of conditional expression
var ifName = ddb.ClauseFor[Person, string]("Name")
db.Update(/* ... */, ifName.NotExists())
db.Update(/* ... */, ifName.Eq("Verner Pleishner"))See constraint.go for the list of supported conditional expressions:
- Comparison:
Eq,Ne,Lt,Le,Gt,Ge,Is - Unary checks:
Exists,NotExists - Set checks:
Between,In - String:
HasPrefix,Contains - Concurrency control:
Optimistic
The conditional expressions are composable using OneOf or AllOf expression. OneOf joins multiple constraint into higher-order constraint that is true when one of defined is true. It is OR expression. AllOf is conjunctive join.
db.Update(/* ... */,
ddb.OneOf(
ifName.NotExists(),
ifName.Eq("Verner Pleishner"),
),
)
db.Update(/* ... */,
ddb.AllOf(
ifName.HasPrefix("Verner"),
ifName.Contains("Pleishner"),
),
)Update expression specifies how update operation will modify the attributes of an item. Unfortunately, this abstraction do not fit into the key-value concept advertised by the library. However, update expression are useful to implement counters, set management, etc.
The dynamo library implements UpdateWith method together with simple DSL.
type Person struct {
Name string `dynamodbav:"name,omitempty"`
Age int `dynamodbav:"age,omitempty"`
}
// defines the builder of updater expression
var (
Name = ddb.UpdateFor[Person, string]("Name")
Age = ddb.UpdateFor[Person, int]("Age")
)
db.UpdateWith(context.Background(),
ddb.Updater(
Person{
Org: curie.IRI("University:Kiel"),
ID: curie.IRI("Professor:8980789222"),
},
Address.Set("Viktoriastrasse 37, Berne, 3013"),
Age.Inc(64),
),
)Optimistic Locking is a lightweight approach to ensure causal ordering of read, write operations to database. AWS made a great post about Optimistic Locking with Version Number.
The dynamo library implements type safe conditional expressions to achieve optimistic locking. This feature is vital when your serverless application concurrently updates same entity in the database.
Let's consider a following example.
type Person struct {
Org string `dynamodbav:"prefix,omitempty"`
ID string `dynamodbav:"suffix,omitempty"`
Name string `dynamodbav:"anothername,omitempty"`
}An optimistic locking on this structure is straightforward from DynamoDB perspective. Just make a request with conditional expression:
&dynamodb.UpdateItemInput{
ConditionExpression: "anothername = :anothername",
ExpressionAttributeValues: /* ":anothername" : {S: "Verner Pleishner"} */
}However, the application operates with struct types. How to define a condition expression on the field Name? Golang struct defines and refers the field by Name but DynamoDB stores it under the attribute anothername. Struct field dynamodbav tag specifies serialization rules. Golang does not support a typesafe approach to build a correspondence between Name ⟷ anothername. Developers have to utilize dynamodb attribute name(s) in conditional expression and Golang struct name in rest of the code. It becomes confusing and hard to maintain. The library defines set of helper types and functions to declare and use conditional expression in type safe manner:
type Person struct {
Org string `dynamodbav:"prefix,omitempty"`
ID string `dynamodbav:"suffix,omitempty"`
Name string `dynamodbav:"anothername,omitempty"`
}
// defines the builder of conditional expression
var Name = ddb.ClauseFor[Person, string]("Name")
val, err := db.Update(context.TODO(), &person, Name.Eq("Verner Pleishner"))
switch err.(type) {
case nil:
// success
case dynamo.PreConditionFailed:
// not found
default:
// other i/o error
}The library provides helper function to implement the concurency control. The optimistic locking constraint Optimistic is built-in OneOf combinator for NotExists or Eq.
db.Put(/* ... */, Name.Optimistic("Verner Pleishner"))See the go doc for all supported constraints.
The library supports batch interface to read/write objects from DynamoDB tables:
BatchGettakes sequence of keys and return sequence of values.BatchPuttakes sequence of object to store.BatchRemovetakes sequence of keys to delete.
The dynamo library is optimized to operate with generic Dynamo DB that declares both partition and sort keys with fixed names. Use the following schema:
const Schema = (): ddb.TableProps => ({
tableName: 'my-table',
partitionKey: {type: ddb.AttributeType.STRING, name: 'prefix'},
sortKey: {type: ddb.AttributeType.STRING, name: 'suffix'},
})If table uses other names for partitionKey and sortKey then config options allows to re-declare it.
db, err := ddb.New[Person](
ddb.WithTable("my-table"),
// Optionally set Global Secondary Index for the session
ddb.WithGlobalSecondaryIndex("my-index"),
// Optionally declare other keys to be user for attribute projection
ddd.WithHashKey("someHashKey"),
ddb.WithSortKey("someSortKey"),
)The following post discusses in depth and shows example DynamoDB table configuration and covers aspect of secondary indexes.
The library advances its simple I/O interface to AWS S3 bucket, allowing to persist data types to multiple storage simultaneously.
import (
"github.com/fogfish/dynamo/v2/service/ddb"
"github.com/fogfish/dynamo/v2/service/s3"
)
//
// Create client and bind it with DynamoDB the table
db, err := ddb.New(ddb.WithTable("my-table"))
//
// Create client and bind it with S3 bucket
db, err := s3.New(s3.WithBucket("my-bucket"))There are few fundamental differences about AWS S3 bucket
- use
s3schema of connection URI; - compose primary key is serialized to S3 bucket path. (e.g.
⟨thread:A, C/E/F⟩ ⟼ thread/A/_/C/E/F); - storage persists struct to JSON, use
jsonfield tags to specify serialization rules; - optimistic locking is not supported yet, any conditional expression is silently ignored;
Updateis not thread safe.
The library is MIT licensed and accepts contributions via GitHub pull requests:
- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Added some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request
The build and testing process requires Go version 1.13 or later.
build and test library.
git clone https://github.com/fogfish/dynamo
cd dynamo
go test ./...
staticcheck ./...Update dependency with go-check-updates
go-check-updates
go-check-updates -u --push githubThe commit message helps us to write a good release note, speed-up review process. The message should address two question what changed and why. The project follows the template defined by chapter Contributing to a Project of Git book.
If you experience any issues with the library, please let us know via GitHub issues. We appreciate detailed and accurate reports that help us to identity and replicate the issue.
