Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(event-handler): add appsync batch resolvers #1998

Merged
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
107 commits
Select commit Hold shift + click to select a range
432093c
update changelog with latest changes
Mar 10, 2023
da1e745
Add batch processing to appsync handler
Mar 3, 2023
9c5cbd9
Extend router to accept List of events. Add functional test
Mar 3, 2023
4793efd
Add e2e tests
Mar 10, 2023
e3ca8c5
Add required package
Mar 10, 2023
d0fe867
Fix linter checks
Mar 10, 2023
bc45703
Refactor appsync resolver
Mar 23, 2023
b7a4391
Refactor code to use composition instead of inheritence
Mar 31, 2023
e1d2caa
Refactor appsync event handler
Apr 7, 2023
b1a49d6
merging from develop
leandrodamascena Apr 14, 2023
f201d25
Merge remote-tracking branch 'upstream/develop' into fix/1303-appsync…
leandrodamascena Apr 27, 2023
8fe6a8b
fix style
leandrodamascena Apr 27, 2023
252a796
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena May 5, 2023
bb87fa3
Merge branch 'develop' into fix/1303-appsync-batch-invoke
heitorlessa Jun 8, 2023
2714b0f
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Jun 8, 2023
986b497
Merge branch 'develop' into fix/1303-appsync-batch-invoke
heitorlessa Jun 12, 2023
b21a0a1
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Jun 28, 2023
a96df34
Merge branch 'develop' into fix/1303-appsync-batch-invoke
mploski Jul 14, 2023
d5bbd09
Add support for async batch processing
Jul 21, 2023
9f8625e
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Aug 15, 2023
659a044
Fixing sonarcloud error
leandrodamascena Aug 15, 2023
13b0bcd
Adding docstring + increasing coverage
leandrodamascena Aug 15, 2023
db74f0a
Adding missing test + increasing coverage
leandrodamascena Aug 15, 2023
1acfe6e
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Aug 29, 2023
b19f4d3
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Sep 4, 2023
c0c45c5
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Sep 5, 2023
56ce1fd
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Sep 27, 2023
5ac0e3c
Start writing docs
Oct 31, 2023
f248769
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Dec 7, 2023
0d2f7e9
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Jan 23, 2024
5f44a2b
Refactoring examples
leandrodamascena Jan 23, 2024
ab7d5b9
Refactoring code + examples + documentation
leandrodamascena Jan 25, 2024
4711cdf
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Jan 25, 2024
50894b9
Moving e2e tests to the right folder
leandrodamascena Jan 25, 2024
3ae6f73
Moving e2e tests to the right folder
leandrodamascena Jan 25, 2024
733b888
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Jan 26, 2024
99b03c5
Adding partial failure
leandrodamascena Jan 26, 2024
32be46c
Adding partial failure
leandrodamascena Jan 26, 2024
4d07d3c
Fixing docstring and examples
leandrodamascena Jan 27, 2024
5644225
Adding documentation about Handling Exceptions
leandrodamascena Jan 27, 2024
78d330b
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Jan 29, 2024
13ac0a1
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Jan 29, 2024
b928996
Fixing docstring
leandrodamascena Jan 29, 2024
45c8a38
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Feb 1, 2024
48ee55f
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Feb 2, 2024
88d2151
Merging from develop
leandrodamascena Feb 6, 2024
f81a003
Adding fine grained control when handling exceptions
leandrodamascena Feb 7, 2024
25fff55
Adding fine grained control when handling exceptions
leandrodamascena Feb 7, 2024
5a335f6
Merge branch 'develop' into fix/1303-appsync-batch-invoke
heitorlessa Feb 7, 2024
cbe7190
docs: add intro diagram
heitorlessa Feb 9, 2024
38a77e8
Merge branch 'develop' into fix/1303-appsync-batch-invoke
heitorlessa Feb 21, 2024
f10a02c
docs: fix wording (Tech debt)
heitorlessa Feb 9, 2024
f0c7b36
refactor: use async_ prefix for async code
heitorlessa Feb 21, 2024
db6d67d
refactor: move router to a separate file to ease maintenance
heitorlessa Feb 21, 2024
50134fc
refactor: rename BasePublic to BaseRouter
heitorlessa Feb 21, 2024
14635f5
refactor: undo router context composition to reduce complexity and ca…
heitorlessa Feb 21, 2024
774a725
refactor: reduce abstractions, use explicit methods over assignments
heitorlessa Feb 21, 2024
e36dd7c
refactor: move registry to a separate file; make it private
heitorlessa Feb 21, 2024
3cddfff
refactor: expand inline if for readability
heitorlessa Feb 21, 2024
17015df
refactor: short circuit upfront, complex after
heitorlessa Feb 22, 2024
54d892b
refactor: simplify arg name
heitorlessa Feb 22, 2024
0ad3383
refactor: add debug statements
heitorlessa Feb 22, 2024
eb340ec
fix(docs): use .context instead of previous ._router.context
heitorlessa Feb 22, 2024
7326582
refactor: use kwargs for explicitness
heitorlessa Feb 22, 2024
26353fa
refactor: use return_exceptions=True to reduce call stack
heitorlessa Feb 23, 2024
b27d497
chore: add notes on the beauty of return_exceptions
heitorlessa Feb 23, 2024
2a1c785
refactor: append suffix in exceptions
heitorlessa Feb 23, 2024
98a8aff
chore: if over elif in short-circuit
heitorlessa Feb 23, 2024
32fd640
chore: improve logging; glad I learned this new f-string trick
heitorlessa Feb 23, 2024
de823a6
chore: fix debug statement location due to null resolvers
heitorlessa Feb 23, 2024
1698778
revert: debug graceful error flag due to non-determinism async
heitorlessa Feb 23, 2024
3e036f9
revert: debug stmt due to mypy; moving elsewhere
heitorlessa Feb 23, 2024
01251bc
docs: docstring resolver (tech debt)
heitorlessa Feb 23, 2024
d280742
docs: minimal batch_resolver docstring
heitorlessa Feb 23, 2024
507d854
chore: complete resolver docstring
heitorlessa Feb 23, 2024
f213510
Merge branch 'develop' into fix/1303-appsync-batch-invoke
heitorlessa Feb 23, 2024
a805ac3
Removing payload exception
Feb 27, 2024
be8b056
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Feb 27, 2024
8592194
Updating poetry
Feb 27, 2024
90e73d0
merging develop
Feb 27, 2024
69fd6ae
Merging from develop
leandrodamascena Mar 14, 2024
e14c73f
Merging from develop
leandrodamascena Mar 14, 2024
857375c
Merging from develop
leandrodamascena Jun 24, 2024
9ed003e
Addressing Heitor's feedback
leandrodamascena Jun 24, 2024
d039211
Merge branch 'develop' into fix/1303-appsync-batch-invoke
leandrodamascena Jun 24, 2024
84c3590
Refactoring to support aggregate events
leandrodamascena Jun 24, 2024
4493165
Refactoring examples + docs
leandrodamascena Jun 24, 2024
a778dcd
Addressing Heitor's feedback
leandrodamascena Jun 25, 2024
615c6d7
Merging from develop
leandrodamascena Jun 25, 2024
b0e1e3e
docs: add diagram to visualize n+1 problem
heitorlessa Jun 26, 2024
fb631ea
docs: improve wording in lambda invoke
heitorlessa Jun 26, 2024
1ebc157
docs: add diagram where n+1 problem shifts to Lambda runtime
heitorlessa Jun 26, 2024
de73d47
docs: add diagram where n+1 problem shifts to Lambda runtime w/ error…
heitorlessa Jun 26, 2024
4414d6b
docs: highlight lambda response for non-errors
heitorlessa Jun 26, 2024
3764819
docs(setup): increase table of contents depth to 5 to help redis and …
heitorlessa Jun 26, 2024
8606136
Adding examples
leandrodamascena Jun 26, 2024
4c84e52
docs: explain N+1 problem and organize content into sub-sections
heitorlessa Jun 26, 2024
06480d4
docs: clean up batch resolvers section; add typing
heitorlessa Jun 26, 2024
925040e
docs: clean up no-aggregate processing section
heitorlessa Jun 26, 2024
2428ebd
docs: clean up raise on error section
heitorlessa Jun 26, 2024
92db4a2
Adding examples
leandrodamascena Jun 26, 2024
8f3af70
docs: clean up async section
heitorlessa Jun 26, 2024
bc6cd12
docs: fix highlights, add missing code annotation
heitorlessa Jun 26, 2024
a4173e0
docs: rename snippets to match advanced section
heitorlessa Jun 26, 2024
ba173b3
Merge branch 'develop' into fix/1303-appsync-batch-invoke
heitorlessa Jun 26, 2024
5d3616a
Merging from develop
leandrodamascena Jun 26, 2024
a612e38
Tests
leandrodamascena Jun 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 23 additions & 5 deletions aws_lambda_powertools/event_handler/appsync.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import logging
from typing import Any, Callable, Optional, Type, TypeVar
from itertools import groupby
from typing import Any, Callable, List, Optional, Type, TypeVar, Union

from aws_lambda_powertools.utilities.data_classes import AppSyncResolverEvent
from aws_lambda_powertools.utilities.typing import LambdaContext
Expand All @@ -10,7 +11,7 @@


class BaseRouter:
current_event: AppSyncResolverEventT # type: ignore[valid-type]
current_event: Union[AppSyncResolverEventT, List[AppSyncResolverEventT]] # type: ignore[valid-type]
leandrodamascena marked this conversation as resolved.
Show resolved Hide resolved
lambda_context: LambdaContext
context: dict

Expand Down Expand Up @@ -152,11 +153,28 @@ def lambda_handler(event, context):
If we could not find a field resolver
"""
# Maintenance: revisit generics/overload to fix [attr-defined] in mypy usage
BaseRouter.current_event = data_model(event)

BaseRouter.lambda_context = context

resolver = self._get_resolver(BaseRouter.current_event.type_name, BaseRouter.current_event.field_name)
response = resolver(**BaseRouter.current_event.arguments)
# If event is a list it means that AppSync sent batch request
if isinstance(event, list):
event_groups = [
{"field_name": field_name, "events": list(events)}
for field_name, events in groupby(event, key=lambda x: x["info"]["fieldName"])
]
if len(event_groups) > 1:
ValueError("batch with different field names. It shouldn't happen!")

appconfig_events = [data_model(event) for event in event_groups[0]["events"]]
BaseRouter.current_event = appconfig_events
resolver = self._get_resolver(appconfig_events[0].type_name, event_groups[0]["field_name"])
response = resolver()
else:
appconfig_event = data_model(event)
BaseRouter.current_event = appconfig_event
resolver = self._get_resolver(appconfig_event.type_name, appconfig_event.field_name)
response = resolver(**appconfig_event.arguments)

self.clear_context()

return response
Expand Down
19 changes: 19 additions & 0 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ cfn-lint = "0.74.1"
mypy = "^0.982"
types-python-dateutil = "^2.8.19.6"
httpx = "^0.23.3"
aws-cdk-aws-appsync-alpha = "^2.59.0a0"

[tool.coverage.run]
source = ["aws_lambda_powertools"]
Expand Down
19 changes: 19 additions & 0 deletions tests/e2e/event_handler/files/schema.graphql
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
schema {
query: Query
}

type Query {
getPost(post_id:ID!): Post
allPosts: [Post]
}

type Post {
post_id: ID!
author: String!
title: String
content: String
url: String
ups: Int
downs: Int
relatedPosts: [Post]
}
99 changes: 99 additions & 0 deletions tests/e2e/event_handler/handlers/appsync_resolver_handler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
from typing import List

from pydantic import BaseModel

from aws_lambda_powertools.event_handler import AppSyncResolver
from aws_lambda_powertools.utilities.typing import LambdaContext

app = AppSyncResolver()


posts = {
"1": {
"post_id": "1",
"title": "First book",
"author": "Author1",
"url": "https://amazon.com/",
"content": "SAMPLE TEXT AUTHOR 1",
"ups": "100",
"downs": "10",
},
"2": {
"post_id": "2",
"title": "Second book",
"author": "Author2",
"url": "https://amazon.com",
"content": "SAMPLE TEXT AUTHOR 2",
"ups": "100",
"downs": "10",
},
"3": {
"post_id": "3",
"title": "Third book",
"author": "Author3",
"url": None,
"content": None,
"ups": None,
"downs": None,
},
"4": {
"post_id": "4",
"title": "Fourth book",
"author": "Author4",
"url": "https://www.amazon.com/",
"content": "SAMPLE TEXT AUTHOR 4",
"ups": "1000",
"downs": "0",
},
"5": {
"post_id": "5",
"title": "Fifth book",
"author": "Author5",
"url": "https://www.amazon.com/",
"content": "SAMPLE TEXT AUTHOR 5",
"ups": "50",
"downs": "0",
},
}

posts_related = {
"1": [posts["4"]],
"2": [posts["3"], posts["5"]],
"3": [posts["2"], posts["1"]],
"4": [posts["2"], posts["1"]],
"5": [],
}


class Post(BaseModel):
post_id: str
author: str
title: str
url: str
content: str
ups: str
downs: str


@app.resolver(type_name="Query", field_name="getPost")
def get_post(post_id: str = "") -> dict:
post = Post(**posts[post_id]).dict()
return post


@app.resolver(type_name="Query", field_name="allPosts")
def all_posts() -> List[dict]:
return list(posts.values())


@app.resolver(type_name="Post", field_name="relatedPosts")
def related_posts() -> List[dict]:
posts = []
for resolver_event in app.current_event:
if resolver_event.source:
posts.append(posts_related[resolver_event.source["post_id"]])
return posts


def lambda_handler(event, context: LambdaContext) -> dict:
return app.resolve(event, context)
42 changes: 41 additions & 1 deletion tests/e2e/event_handler/infrastructure.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
from pathlib import Path
from typing import Dict, Optional

from aws_cdk import CfnOutput
from aws_cdk import CfnOutput, Duration, Expiration
from aws_cdk import aws_apigateway as apigwv1
from aws_cdk import aws_apigatewayv2_alpha as apigwv2
from aws_cdk import aws_apigatewayv2_authorizers_alpha as apigwv2authorizers
from aws_cdk import aws_apigatewayv2_integrations_alpha as apigwv2integrations
from aws_cdk import aws_appsync_alpha as appsync
from aws_cdk import aws_ec2 as ec2
from aws_cdk import aws_elasticloadbalancingv2 as elbv2
from aws_cdk import aws_elasticloadbalancingv2_targets as targets
Expand All @@ -21,6 +23,7 @@ def create_resources(self):
self._create_api_gateway_rest(function=functions["ApiGatewayRestHandler"])
self._create_api_gateway_http(function=functions["ApiGatewayHttpHandler"])
self._create_lambda_function_url(function=functions["LambdaFunctionUrlHandler"])
self._create_appsync_endpoint(function=functions["AppsyncResolverHandler"])

def _create_alb(self, function: Function):
vpc = ec2.Vpc.from_lookup(
Expand Down Expand Up @@ -84,3 +87,40 @@ def _create_lambda_function_url(self, function: Function):
# Maintenance: move auth to IAM when we create sigv4 builders
function_url = function.add_function_url(auth_type=FunctionUrlAuthType.AWS_IAM)
CfnOutput(self.stack, "LambdaFunctionUrl", value=function_url.url)

def _create_appsync_endpoint(self, function: Function):
api = appsync.GraphqlApi(
self.stack,
"Api",
name="e2e-tests",
schema=appsync.SchemaFile.from_asset(str(Path(self.feature_path, "files/schema.graphql"))),
authorization_config=appsync.AuthorizationConfig(
default_authorization=appsync.AuthorizationMode(
authorization_type=appsync.AuthorizationType.API_KEY,
api_key_config=appsync.ApiKeyConfig(
description="public key for getting data",
expires=Expiration.after(Duration.hours(25)),
name="API Token",
),
)
),
xray_enabled=False,
)
lambda_datasource = api.add_lambda_data_source("DataSource", lambda_function=function)

lambda_datasource.create_resolver(
"QueryGetAllPostsResolver",
type_name="Query",
field_name="allPosts",
)
lambda_datasource.create_resolver(
"QueryGetPostResolver",
type_name="Query",
field_name="getPost",
)
lambda_datasource.create_resolver(
"QueryGetPostRelatedResolver", type_name="Post", field_name="relatedPosts", max_batch_size=10
)

CfnOutput(self.stack, "GraphQLHTTPUrl", value=api.graphql_url)
CfnOutput(self.stack, "GraphQLAPIKey", value=api.api_key)
108 changes: 108 additions & 0 deletions tests/e2e/event_handler/test_appsync_resolvers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
import json

import pytest
from requests import Request

from tests.e2e.utils import data_fetcher


@pytest.fixture
def appsync_endpoint(infrastructure: dict) -> str:
return infrastructure["GraphQLHTTPUrl"]


@pytest.fixture
def appsync_access_key(infrastructure: dict) -> str:
return infrastructure["GraphQLAPIKey"]


@pytest.mark.xdist_group(name="event_handler")
def test_appsync_get_all_posts(appsync_endpoint, appsync_access_key):
# GIVEN
body = {
"query": "query MyQuery { allPosts { post_id }}",
"variables": None,
"operationName": "MyQuery",
}

# WHEN
response = data_fetcher.get_http_response(
Request(
method="POST",
url=appsync_endpoint,
json=body,
headers={"x-api-key": appsync_access_key, "Content-Type": "application/json"},
)
)

# THEN expect a HTTP 200 response and content return list of Posts
assert response.status_code == 200
assert response.content is not None

data = json.loads(response.content.decode("ascii"))["data"]

assert data["allPosts"] is not None
assert len(data["allPosts"]) > 0


@pytest.mark.xdist_group(name="event_handler")
def test_appsync_get_post(appsync_endpoint, appsync_access_key):
# GIVEN
post_id = "1"
body = {
"query": f'query MyQuery {{ getPost(post_id: "{post_id}") {{ post_id }} }}',
"variables": None,
"operationName": "MyQuery",
}

# WHEN
response = data_fetcher.get_http_response(
Request(
method="POST",
url=appsync_endpoint,
json=body,
headers={"x-api-key": appsync_access_key, "Content-Type": "application/json"},
)
)

# THEN expect a HTTP 200 response and content return Post id
assert response.status_code == 200
assert response.content is not None

data = json.loads(response.content.decode("ascii"))["data"]

assert data["getPost"]["post_id"] == post_id


@pytest.mark.xdist_group(name="event_handler")
def test_appsync_get_related_posts_batch(appsync_endpoint, appsync_access_key):
# GIVEN
post_id = "2"
related_posts_ids = ["3", "5"]

body = {
"query": f'query MyQuery {{ getPost(post_id: "{post_id}") {{ post_id relatedPosts {{ post_id }} }} }}',
"variables": None,
"operationName": "MyQuery",
}

# WHEN
response = data_fetcher.get_http_response(
Request(
method="POST",
url=appsync_endpoint,
json=body,
headers={"x-api-key": appsync_access_key, "Content-Type": "application/json"},
)
)

# THEN expect a HTTP 200 response and content return Post id with dependent Posts id's
assert response.status_code == 200
assert response.content is not None

data = json.loads(response.content.decode("ascii"))["data"]

assert data["getPost"]["post_id"] == post_id
assert len(data["getPost"]["relatedPosts"]) == len(related_posts_ids)
for post in data["getPost"]["relatedPosts"]:
assert post["post_id"] in related_posts_ids
Loading