Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework of the Shape type system #102

Merged
merged 77 commits into from
Feb 4, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
e4200fc
Classes with shape properties supporting annotations
sam-goodwin Jan 2, 2020
5f69d72
Scaffolding for new shape system established
sam-goodwin Jan 3, 2020
2a1bcb7
Stash kinda workiung:
sam-goodwin Jan 3, 2020
4bd88a3
Type-safe JSON schema transformation of Shape AST
sam-goodwin Jan 3, 2020
ac6b9cb
Collection types
sam-goodwin Jan 3, 2020
5ae5b16
Timestamp type and nice refactors
sam-goodwin Jan 4, 2020
c5e51de
Comments
sam-goodwin Jan 4, 2020
1aac30b
Of and of utilities
sam-goodwin Jan 4, 2020
6865b2d
Centralize ShapeGuards
sam-goodwin Jan 4, 2020
ef1c7aa
Tests for ShapeGuards
sam-goodwin Jan 4, 2020
5f8bcc9
Support decorators
sam-goodwin Jan 4, 2020
c756b08
Stash
sam-goodwin Jan 4, 2020
7d3da58
Type-safe meta annotations
sam-goodwin Jan 4, 2020
786b97f
Type-safe annotation plumbing is wired up
sam-goodwin Jan 4, 2020
bc8fd6a
Maintain the root Shape type of applied metadata to cleanup signature…
sam-goodwin Jan 5, 2020
9aa1202
Support numeric constraints for json schema
sam-goodwin Jan 5, 2020
3d27a96
Shape runtime mappings
sam-goodwin Jan 5, 2020
e0f3013
Runtime type mappings
sam-goodwin Jan 5, 2020
dd08560
Make better use of Symbols to stash metadatra
sam-goodwin Jan 5, 2020
f2f01e0
Support declaration merging
sam-goodwin Jan 5, 2020
df6d5b0
Map Shape to DynamoDB AttributeValue AST
sam-goodwin Jan 5, 2020
8bdadc0
Attribute Value mapper is working and is type-safe
sam-goodwin Jan 5, 2020
dd1cf86
Framework in place for DDB query expressions
sam-goodwin Jan 5, 2020
4652e2e
Grooming
sam-goodwin Jan 5, 2020
5fc0ca1
Better AST for queries
sam-goodwin Jan 6, 2020
95ab4b1
Query compilation is working
sam-goodwin Jan 6, 2020
3766b2b
Rename query to dsl
sam-goodwin Jan 6, 2020
2f6527c
move member enumeration into the AttributeValue.Struct type itself
sam-goodwin Jan 6, 2020
f4409f5
Move type enumeration into the types for AttributeValue
sam-goodwin Jan 6, 2020
c139984
rename query to filter
sam-goodwin Jan 6, 2020
937f912
Support assign and increment/decrement update expressions
sam-goodwin Jan 6, 2020
5405430
List push and concat
sam-goodwin Jan 6, 2020
a3101a6
DynamoDB Table Client partial
sam-goodwin Jan 6, 2020
505aba5
Remove DDB visitor
sam-goodwin Jan 6, 2020
21a76b0
Support disabling ClassShape cache
sam-goodwin Jan 6, 2020
3e30ca1
fix ShapeGuards assert message
sam-goodwin Jan 6, 2020
63fbcfa
Remove ClassModel type alias and clean up Member derivations
sam-goodwin Jan 6, 2020
3a21a20
Grooming
sam-goodwin Jan 6, 2020
75f947c
DDB client unit tests
sam-goodwin Jan 6, 2020
a8fe5a5
hashCode, equals, data structures, json de/ser
sam-goodwin Jan 7, 2020
244ca08
Required and optional kets. Fixes to bugs for runtime types and json …
sam-goodwin Jan 8, 2020
ddc6179
Fix broken types introduced by Member.Map :(.
sam-goodwin Jan 8, 2020
073038c
Support array and map indexing
sam-goodwin Jan 8, 2020
fb7b262
JSON Path AST
sam-goodwin Jan 8, 2020
df17080
Dynamic and Binary Shapes
sam-goodwin Jan 8, 2020
f37275c
Groom
sam-goodwin Jan 8, 2020
2c7c463
Validator logic
sam-goodwin Jan 8, 2020
06243c1
Validate DynamoDB records
sam-goodwin Jan 8, 2020
f08fc85
Add path information to validation
sam-goodwin Jan 9, 2020
e9851be
add shapes-glue mappings
sam-goodwin Jan 10, 2020
e193945
Big stuff lol
sam-goodwin Jan 15, 2020
edd2a4c
Implement Record function to create classes dynamically
sam-goodwin Jan 24, 2020
bd7ad71
Fixing a bunch of packages
sam-goodwin Jan 28, 2020
3486324
Fix JSON Path
sam-goodwin Jan 28, 2020
dd2eb9f
punchcard is compiling
sam-goodwin Jan 28, 2020
c701f8d
Most tests are passing
sam-goodwin Jan 29, 2020
fa3fc08
Clean up how Data Type was configured for a data structure
sam-goodwin Jan 30, 2020
4a0efdf
Data lake compiles.
sam-goodwin Jan 30, 2020
c9341ab
Tests are mostly working
sam-goodwin Jan 30, 2020
4d639ce
Rename ClassShape to RecordShape
sam-goodwin Feb 4, 2020
55eaf19
Add README to the shape lib
sam-goodwin Feb 4, 2020
93531f8
More READMEs
sam-goodwin Feb 4, 2020
cadb188
Rename shape-glue to shape-hive
sam-goodwin Feb 4, 2020
9500be7
Update snapshot tests and fix empty ({}) Record bugs
sam-goodwin Feb 4, 2020
3f39c7e
Upgrade to cdk 1.22.0
sam-goodwin Feb 4, 2020
105d9a8
Fix bugs with DDB update expressions
sam-goodwin Feb 4, 2020
5437290
Fix bootstrap bugs - got hello-world example working
sam-goodwin Feb 4, 2020
7c7d977
Stream-processing example is working
sam-goodwin Feb 4, 2020
40055f9
Updates and cleanups for README.md. Support condition expressions for…
sam-goodwin Feb 4, 2020
f29d168
Fix snapshot tests
sam-goodwin Feb 4, 2020
503c229
Update docs
sam-goodwin Feb 4, 2020
ab775d5
Move all @punchcard/shape-json exports into the Json namespace
sam-goodwin Feb 4, 2020
764a1ac
Fix Glue typos on README
sam-goodwin Feb 4, 2020
d758678
Fix Glue typos on README
sam-goodwin Feb 4, 2020
e864e4f
Update package json
sam-goodwin Feb 4, 2020
6cbd347
Merge branch 'master' into struct-overhaul
sam-goodwin Feb 4, 2020
d130e1d
Fix package locks
sam-goodwin Feb 4, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
153 changes: 64 additions & 89 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@ To understand the internals, there is the guide:

# Tour

Initialize an App and Stack:
```ts
const app = new Core.App();
const stack = app.stack('hello-world');
```

## Runtime Code and Dependencies

Creating a Lambda Function is super simple - just create it and implement `handle`:
Expand All @@ -46,7 +52,7 @@ This will create the required IAM policies for your Function's IAM Role, add any
```ts
new Lambda.Function(stack, 'MyFunction', {
depends: topic,
handle: async (event, topic) => {
}, async (event, topic) => {
await topic.publish({
key: 'some key',
count: 1,
Expand All @@ -59,95 +65,83 @@ new Lambda.Function(stack, 'MyFunction', {
Furthermore, its interface is higher-level than what would normally be expected when using the `aws-sdk`, and it's also type-safe: the argument to the `publish` method is not an opaque `string` or `Buffer`, it is an `object` with keys and rich types such as `Date`. This is because data structures in punchcard, such as `Topic`, `Queue`, `Stream`, etc. are generic with statically declared types (like an `Array<T>`):

```ts
/**
* Message is a JSON Object with properties: `key`, `count` and `timestamp`.
*/
class NotificationType extends Record({
key: string,
count: integer,
timestamp
}) {}

const topic = new SNS.Topic(stack, 'Topic', {
/**
* Message is a JSON Object with properties: `key`, `count` and `timestamp`.
*/
shape: struct({
key: string(),
count: integer(),
timestamp
})
shape: NofiticationType
});
```

This `Topic` is now of type:
```ts
Topic<{
key: string;
count: number;
timestamp: Date;
}>
Topic<NotificationType>
```

## Type-Safe DynamoDB Expressions

This feature in punchcard becomes even more evident when using DynamoDB. To demonstrate, let's create a DynamoDB `Table` and use it in a `Function`:

```ts
// class describing the data in the DynamoDB Table
class TableRecord extends Record({
id: string,
count: integer
.apply(Minimum(0))
}) {}

// table of TableRecord, with a single hash-key: 'id'
const table = new DynamoDB.Table(stack, 'my-table', {
partitionKey: 'id',
attributes: {
id: string(),
count: integer({
minimum: 0
})
},
billingMode: BillingMode.PAY_PER_REQUEST
attributes: TableRecord,
key: 'id'
});
```

Now, when getting an item from DynamoDB, there is no need to use `AttributeValues` such as `{ S: 'my string' }`, like you would when using the low-level `aws-sdk`. You simply use ordinary javascript types:

```ts
const item = await table.get({
id: 'state'
});
const item = await table.get('state');
```

The interface is statically typed and derived from the definition of the `Table` - we specified the `partitionKey` as the `id` field which has type `string`, and so the object passed to the `get` method must correspond.

`PutItem` and `UpdateItem` have similarly high-level and statically checked interfaces. More interestingly, condition and update expressions are built with helpers derived (again) from the table definition:

```ts
// put an item if it doesn't already exist
await table.put({
item: {
id: 'state',
count: 1
},
if: item => DynamoDB.attribute_not_exists(item.id)
// put an item if it doesn't exist
await table.put(new TableRecord({
id: 'state',
count: 1
}), {
if: _ => _.id.notExists()
});

// increment the count property by 1
await table.update({
key: {
id: 'state'
},
actions: item => [
item.count.increment(1)
]
// increment the count property by 1 if it is less than 0
await table.update('state', {
actions: _ => [
_.count.increment(1)
],
if: _ => _.id.lessThan(0)
});
```

If you specified a `sortKey`:
To also specify `sortKey`, use a tuple of `TableRecord's` keys:

```ts
const table = new DynamoDB.Table(stack, 'my-table', {
partitionKey: 'id',
sortKey: 'count', // specify a sortKey
// ...
});
const table = new DynamoDB.Table(stack, 'my-table', TablerRecord, ['id', 'count']);
```

Then you can also build typesafe query expressions:
Now, you can also build typesafe query expressions:

```ts
await table.query({
key: {
id: 'id',
count: DynamoDB.greaterThan(1)
},
await table.query(['id', _ => _.count.greaterThan(1)], {
filter: _ => _.count.lessThan(0)
})
```
## Stream Processing
Expand All @@ -157,20 +151,14 @@ Punchcard has the concept of `Stream` data structures, which should feel similar
For example, given an SNS Topic:
```ts
const topic = new SNS.Topic(stack, 'Topic', {
shape: struct({
key: string(),
count: integer(),
timestamp
})
shape: NotificationType
});
```

You can attach a new Lambda Function to process each notification:
```ts
topic.notifications().forEach(stack, 'ForEachNotification', {
handle: async (notification) => {
console.log(`notification delayed by ${new Date().getTime() - notification.timestamp.getTime()}ms`);
}
topic.notifications().forEach(stack, 'ForEachNotification', {}, async (notification) => {
console.log(`notification delayed by ${new Date().getTime() - notification.timestamp.getTime()}ms`);
})
```

Expand All @@ -185,26 +173,24 @@ const queue = topic.toSQSQueue(stack, 'MyNewQueue');
We can then, perhaps, `map` over each message in the `Queue` and collect the results into a new AWS Kinesis `Stream`:

```ts
class LogData extends Record({
key: string,
count: integer,
tags: array(string)
timestamp
}) {}

const stream = queue.messages()
.map({
handle: async(message, e) => {
return {
...message,
tags: ['some', 'tags'],
};
}
.map(async (message, e) => ({
...message,
tags: ['some', 'tags'],
})
.toKinesisStream(stack, 'Stream', {
// partition values across shards by the 'key' field
partitionBy: value => value.key,

// type of the data in the stream
type: struct({
key: string(),
count: integer(),
tags: array(string()),
timestamp
})
shape: LogData
});
```

Expand All @@ -218,6 +204,7 @@ With data now flowing to S3, let's partition and catalog it in a `Glue.Table` (b

```ts
import glue = require('@aws-cdk/aws-glue');
import { Glue } from 'punchcard';

const database = stack.map(stack => new glue.Database(stack, 'Database', {
databaseName: 'my_database'
Expand All @@ -228,21 +215,9 @@ s3DeliveryStream.objects().toGlueTable(stack, 'ToGlue', {
columns: stream.type.shape,
partition: {
// Glue Table partition keys: minutely using the timestamp field
keys: {
year: integer(),
month: integer(),
day: integer(),
hour: integer(),
minute: integer()
},
get: record => ({
// define the mapping of a record to its Glue Table partition keys
year: record.timestamp.getUTCFullYear(),
month: record.timestamp.getUTCMonth(),
day: record.timestamp.getUTCDate(),
hour: record.timestamp.getUTCHours(),
minute: record.timestamp.getUTCMinutes(),
})
keys: Glue.Partition.Minutely,
// define the mapping of a record to its Glue Table partition keys
get: record => Glue.Partition.byMinute(record.timestamp)
}
});
```
Expand Down
2 changes: 2 additions & 0 deletions docs/1-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ const app = new Core.App();

// Constructs are created laziliy by mapping "into the" Build context
const stack = app.root.map(app => new cdk.Stack(app, 'MyStack'));
// you can also use the helper:
const stack = app.stack('MyStack');

Lambda.schedule(stack, 'MyFunction', {
schedule: Schedule.rate(cdk.Duration.minutes(1)),
Expand Down
22 changes: 10 additions & 12 deletions docs/2-creating-functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,26 @@
Creating a `Lambda.Function` is super simple - just instantiate it and implement `handle`:

```ts
new Lambda.Function(stack, 'MyFunction', {
handle: async() => console.log('Hello, World!')
});
new Lambda.Function(stack, 'MyFunction', {},
async() => console.log('Hello, World!')
);
```

It supports all the same properties as the AWS CDK's [`@aws-cdk/aws-lambda.Function`](https://github.com/aws/aws-cdk/tree/master/packages/%40aws-cdk/aws-lambda), so you can configure things such as memory:

```ts
new Lambda.Function(stack, 'MyFunction', {
handle: async() => console.log('Hello, World!'),
memorySize: 512;
memorySize: 512
}, async() => console.log('Hello, World!'),
});
```

# Request and Response Shape
By default, the type of the request and response is `any`, but you can also explicitly define the types:
```ts
new Lambda.Function<string, number>(stack, 'MyFunction', {
handle: async(str) => str.length
});
new Lambda.Function<string, number>(stack, 'MyFunction', {},
async(str) => str.length
);
```

These types will be serialized to and from JSON based on their JS types (using `JSON.stringify`). To explicitly control the serialization, explicitly provide a **Shape** for the `request` and `response` properties.
Expand All @@ -31,8 +31,7 @@ These types will be serialized to and from JSON based on their JS types (using `
new Lambda.Function(stack, 'MyFunction', {
request: string(),
response: integer(),
handle: async(str) => str.length
});
}, async(str) => str.length);
```

Now, the Punchcard framework will validate and serialzie the request and response according to their "Shape". (*See Part 4 - [Shapes: Type-Safe Schemas](4-shapes.md)*).
Expand All @@ -43,8 +42,7 @@ You can schedule a new `Lambda.Function` to do some work:
```ts
Lambda.schedule(stack, 'MyFunction', {
schedule: Schedule.rate(Duration.minutes(1)),
handle: async(request: CloudWatch.Event) => console.log('Hello, World!'),
});
}, async(request: CloudWatch.Event) => console.log('Hello, World!'));
```

Note: how the the type of `request` is a `CloudWatch.Event`, as it is regularly triggered by a scheduled CloudWatch Event
Expand Down
18 changes: 9 additions & 9 deletions docs/3-runtime-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@ To contact other services in your Function, data structures such as `SNS.Topic`,

First, create the Construct you want to access from Lambda, like an `SQS.Queue`:
```ts
import { string } from '@punchcard/shape';

const queue = new SQS.Queue(stack, 'MyQueue', {
shape: string()
shape: string // data type will be orindary strings
});
```

Expand All @@ -25,9 +27,8 @@ The result is that your `handle` function is now passed a `queue` client instanc
```ts
new Lambda.Function(stack, 'MyFunction', {
depends: queue,
handle: async (event, queue) => {
await queue.sendMessage('Hello, World!');
}
}, async (event, queue) => {
await queue.sendMessage('Hello, World!');
});
```

Expand Down Expand Up @@ -63,10 +64,9 @@ new Lambda.Function(stack, 'MyFunction', {
queue: queue.sendAccess(),
topic
}),
handle: async (event, ({queue, topic})) => {
await queue.sendMessage('Hello, SQS!');
await topic.publish('Hello, SNS!');
}
}, async (event, ({queue, topic})) => {
await queue.sendMessage('Hello, SQS!');
await topic.publish('Hello, SNS!');
});
```

Expand Down Expand Up @@ -120,7 +120,7 @@ const namedDependency: Dependency.Named<{
You may have noticed something strange about the definition of our `SQS.Queue`:
```ts
const queue: SQS.Queue<StringShape> = new SQS.Queue(this, 'Q', {
shape: string()
shape: string
});
```

Expand Down
Loading