diff --git a/lambda-bedrock-response-streaming-rust-cdk/.gitignore b/lambda-bedrock-response-streaming-rust-cdk/.gitignore new file mode 100644 index 000000000..2cdcf27b8 --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/.gitignore @@ -0,0 +1,10 @@ +*.js +!jest.config.js +*.d.ts +node_modules +.DS_Store + +# CDK asset staging directory +.cdk.staging +cdk.out + diff --git a/lambda-bedrock-response-streaming-rust-cdk/README.md b/lambda-bedrock-response-streaming-rust-cdk/README.md new file mode 100644 index 000000000..b21e457d8 --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/README.md @@ -0,0 +1,131 @@ +# AWS Lambda and Amazon Bedrock Response Streaming with Rust + +This CDK application demonstrates how to write a Lambda function, expose it as a function URL, use the Bedrock ConverseStream API to invoke the Anthropic Claude 3 Haiku model, and then stream a response back to the client using chunked transfer encoding. Written in Rust, it showcases how to efficiently stream responses from Amazon Bedrock to a client. The example serves as a practical illustration of implementing real-time, serverless communication between Bedrock's GenAI capabilities and a client application. + +Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. + +## Requirements + +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [Node and NPM](https://nodejs.org/en/download/) installed +* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) (AWS CDK) installed +* [Docker](https://docs.docker.com/engine/install/) installed and running locally (needed for Rust cross-platform Lambda build) +* [Rust](https://www.rust-lang.org/) 🦀 installed with v1.79.0 or higher +* [Cargo Lambda](https://www.cargo-lambda.info/) installed +* [cross](https://github.com/cross-rs/cross) compilation installed for Cargo Lambda: `cargo install cross --git https://github.com/cross-rs/cross` + +## Amazon Bedrock Setup Instructions + +You must request access to the Bedrock LLM model before you can use it. This example uses `Claude 3 Haiku`, so make sure you have `Access granted` to this model. For more information, see [Model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html). + +## Deployment Instructions + +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: + ```bash + git clone https://github.com/aws-samples/serverless-patterns + ``` +2. Change directory to the pattern's CDK directory: + ```bash + cd lambda-bedrock-response-streaming-rust-cdk/cdk + ``` +3. From the command line, use npm to install the development dependencies: + ```bash + npm install + ``` +4. If you haven't done so previously for this account, run this command to bootstrap CDK: + ```bash + cdk bootstrap + ``` +5. Review the CloudFormation template that CDK generates for your stack using the following AWS CDK CLI command: + ```bash + cdk synth + ``` +6. Use AWS CDK to deploy your AWS resources + ```bash + cdk deploy + ``` + + After the deployment completes, note the URL in the `Outputs` section at the end. The `BedrockStreamerStack.LambdaFunctionURL` followed by the Lambda Function URL (FURL) will be used to invoke the Lambda function. It should look something like: + + `https://{YOUR_ID_HERE}.lambda-url.{YOUR_REGION_HERE}.on.aws/` + +## How it works + +This pattern exposes a public Lambda Function URL (FURL) endpoint where requests can be made. When a request is made to the FURL, the Lambda function initiates a streaming request to Amazon Bedrock using the [ConverseStream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html) API. This allows the response from the LLM in Bedrock to start streaming back to the Lambda function as soon as generation begins, without waiting for the entire response to be ready. + +Generating Bedrock responses often takes a long time. By using a streaming approach, the Lambda function can immediately start processing the incoming response and writing the data chunks back to the client. The chunks are delievered in real-time to the connected client, providing an extremely interactive user experience. + +## Testing + +If you deployed the application without any code changes, you've deployed it with `IAM` authorization. You can see the CDK code for it in `./cdk/lib/bedrock-streamer-lambda/bedrock-streamer-stack.ts` + +```ts +// Create a Lambda Function URL +const lambdaUrl = streamingLambda.addFunctionUrl({ + authType: lambda.FunctionUrlAuthType.AWS_IAM, + invokeMode: lambda.InvokeMode.RESPONSE_STREAM, +}); +``` +Here you can see `authType: lambda.FunctionUrlAuthType.AWS_IAM` is instructing the Lambda function to use `IAM`. However, if you want to make requests to the FURL without credentials, you can change it to `authType: lambda.FunctionUrlAuthType.NONE`. But **BEWARE**, as doing so exposes the FURL to any unauthenticated user. + +To make a request to the FURL, you can use a number of different options. This example will use [curl](https://curl.se/) to make the request. + +To make the request using `IAM`, you'll need credentials from an `IAM User` or `IAM Role` to sign the requests. The credentials will need permission for the `lambda:InvokeFunctionUrl` action for the deployed Lambda function. + +1. In order to use the curl command with `IAM` auth enabled, you can use the `--user access_key:secret_access_key` option. It's a simple way to do it, but if you don't add the keys to environment variables, you could potentially expose them to your command-line history. So for this example, both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` should be exported as environment variables to your shell (`Bash` in this example) + + ```bash + curl -v 'https://{YOUR_ID_HERE}.lambda-url.{YOUR_REGION_HERE}.on.aws' \ + -H "X-Amz-Date: $(date -u '+%Y%m%dT%H%M%SZ')" \ + -H 'Content-Type: application/json' \ + --aws-sigv4 'aws:amz:{YOUR_REGION_HERE}:lambda' \ + --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \ + --data-raw '{"storyType": "CATS"}' \ + --no-buffer + ``` + If you're using a temporary IAM user or role, perhaps through an SSO login, you'll likely need to add the session to your `curl` request, so it looks something like this: + + ```bash + curl -v 'https://{YOUR_ID_HERE}.lambda-url.{YOUR_REGION_HERE}.on.aws' \ + -H "X-Amz-Security-Token: $AWS_SESSION_TOKEN" \ + -H "X-Amz-Date: $(date -u '+%Y%m%dT%H%M%SZ')" \ + -H 'Content-Type: application/json' \ + --aws-sigv4 'aws:amz:{YOUR_REGION_HERE}:lambda' \ + --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \ + --data-raw '{"storyType": "DOGS"}' \ + --no-buffer + ``` + where `AWS_SESSION_TOKEN` is the token for your AWS session. If you're authenticated with SSO, you can get all of these by running this command: + + ```bash + aws configure export-credentials --format env + ``` + +2. If you're not using `IAM` for your Lambda function, you don't need any access keys or sessions. You can simply use the following as an example + + ```bash + curl -v 'https://{YOUR_ID_HERE}.lambda-url.{YOUR_REGION_HERE}.on.aws' \ + -H 'Content-Type: application/json' \ + --data-raw '{"storyType": "HORSES"}' \ + --no-buffer + ``` + +For any of the commands you use, you should get streaming output as it's generated by the LLM. Here's an example of what you might see if you used a `--data-raw '{"storyType": "CATS"}'` entry + +```json +{"type":"other","message":null}{"type":"text","message":"Here"}{"type":"text","message":"'s a very"}{"type":"text","message":" short story about cats"}{"type":"text","message":":"}{"type":"text","message":"\n\nThe"}{"type":"text","message":" Curious"}{"type":"text","message":" F"}{"type":"text","message":"eline"}{"type":"text","message":"\n\nWhis"}{"type":"text","message":"kers t"}{"type":"text","message":"witched, eyes"}{"type":"text","message":" narrow"}{"type":"text","message":"ed,"}{"type":"text","message":" the"}{"type":"text","message":" sle"}{"type":"text","message":"ek tab"}{"type":"text","message":"by cat"}{"type":"text","message":" cr"}{"type":"text","message":"ept"}{"type":"text","message":" forwar"}{"type":"text","message":"d caut"}{"type":"text","message":"iously."}{"type":"text","message":" A"}{"type":"text","message":" new"}{"type":"text","message":" toy"}{"type":"text","message":" ha"}{"type":"text","message":"d caught its"}{"type":"text","message":" attention -"}{"type":"text","message":" a"}{"type":"text","message":" curious"}{"type":"text","message":" contraption that b"}{"type":"text","message":"linked an"}{"type":"text","message":"d whirred."}{"type":"text","message":" With"}{"type":"text","message":" a"}{"type":"text","message":" slight t"}{"type":"text","message":"ilt of the hea"}{"type":"text","message":"d, the"}{"type":"text","message":" feline p"}{"type":"text","message":"ounced,"}{"type":"text","message":" batting"}{"type":"text","message":" at"}{"type":"text","message":" the"}{"type":"text","message":" strange"}{"type":"text","message":" object"}{"type":"text","message":"."}{"type":"text","message":" A"}{"type":"text","message":" flash"}{"type":"text","message":" of light"}{"type":"text","message":","}{"type":"text","message":" a"}{"type":"text","message":" gentle"}{"type":"text","message":" hum"}{"type":"text","message":","}{"type":"text","message":" then silence"}{"type":"text","message":"."}{"type":"text","message":" The"}{"type":"text","message":" cat"}{"type":"text","message":" sat"}{"type":"text","message":" back"}{"type":"text","message":","}{"type":"text","message":" please"}{"type":"text","message":"d with its"}{"type":"text","message":" work,"}{"type":"text","message":" before"}{"type":"text","message":" cur"}{"type":"text","message":"ling up"}{"type":"text","message":" for"}{"type":"text","message":" a well"}{"type":"text","message":"-earned n"}{"type":"text","message":"ap,"}{"type":"text","message":" leaving"}{"type":"text","message":" the"}{"type":"text","message":" dism"}{"type":"text","message":"ant"}{"type":"text","message":"led ga"}{"type":"text","message":"dget behin"}{"type":"text","message":"d."}{"type":"other","message":null}{"type":"other","message":null}{"type":"other","message":null}* Connection #0 to host {YOUR_ID_HERE}.lambda-url.{YOUR_REGION_HERE}.on.aws left intact +``` + +## Cleanup + +You can use the following commands to destroy the AWS resources created during deployment. This assumes you're currently at the `lambda-bedrock-response-streaming-rust-cdk/cdk` directory in your terminal: + +```bash +cdk destroy +``` +---- +Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +SPDX-License-Identifier: MIT-0 diff --git a/lambda-bedrock-response-streaming-rust-cdk/cdk/.npmignore b/lambda-bedrock-response-streaming-rust-cdk/cdk/.npmignore new file mode 100644 index 000000000..c1d6d45dc --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/cdk/.npmignore @@ -0,0 +1,6 @@ +*.ts +!*.d.ts + +# CDK asset staging directory +.cdk.staging +cdk.out diff --git a/lambda-bedrock-response-streaming-rust-cdk/cdk/bin/bedrock-streamer.ts b/lambda-bedrock-response-streaming-rust-cdk/cdk/bin/bedrock-streamer.ts new file mode 100644 index 000000000..3ac6983dd --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/cdk/bin/bedrock-streamer.ts @@ -0,0 +1,21 @@ +#!/usr/bin/env node +import 'source-map-support/register'; +import * as cdk from 'aws-cdk-lib'; +import { BedrockStreamerStack } from '../lib/bedrock-streamer-stack'; + +const app = new cdk.App(); +new BedrockStreamerStack(app, 'BedrockStreamerStack', { + /* If you don't specify 'env', this stack will be environment-agnostic. + * Account/Region-dependent features and context lookups will not work, + * but a single synthesized template can be deployed anywhere. */ + + /* Uncomment the next line to specialize this stack for the AWS Account + * and Region that are implied by the current CLI configuration. */ + // env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION }, + + /* Uncomment the next line if you know exactly what Account and Region you + * want to deploy the stack to. */ + // env: { account: '123456789012', region: 'us-east-1' }, + + /* For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html */ +}); \ No newline at end of file diff --git a/lambda-bedrock-response-streaming-rust-cdk/cdk/cdk.json b/lambda-bedrock-response-streaming-rust-cdk/cdk/cdk.json new file mode 100644 index 000000000..18218baed --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/cdk/cdk.json @@ -0,0 +1,71 @@ +{ + "app": "npx ts-node --prefer-ts-exts bin/bedrock-streamer.ts", + "watch": { + "include": [ + "**" + ], + "exclude": [ + "README.md", + "cdk*.json", + "**/*.d.ts", + "**/*.js", + "tsconfig.json", + "package*.json", + "yarn.lock", + "node_modules", + "test" + ] + }, + "context": { + "@aws-cdk/aws-lambda:recognizeLayerVersion": true, + "@aws-cdk/core:checkSecretUsage": true, + "@aws-cdk/core:target-partitions": [ + "aws", + "aws-cn" + ], + "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true, + "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true, + "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true, + "@aws-cdk/aws-iam:minimizePolicies": true, + "@aws-cdk/core:validateSnapshotRemovalPolicy": true, + "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true, + "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true, + "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true, + "@aws-cdk/aws-apigateway:disableCloudWatchRole": true, + "@aws-cdk/core:enablePartitionLiterals": true, + "@aws-cdk/aws-events:eventsTargetQueueSameAccount": true, + "@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true, + "@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true, + "@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true, + "@aws-cdk/aws-route53-patters:useCertificate": true, + "@aws-cdk/customresources:installLatestAwsSdkDefault": false, + "@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true, + "@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true, + "@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true, + "@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true, + "@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true, + "@aws-cdk/aws-redshift:columnId": true, + "@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true, + "@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true, + "@aws-cdk/aws-apigateway:requestValidatorUniqueId": true, + "@aws-cdk/aws-kms:aliasNameRef": true, + "@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true, + "@aws-cdk/core:includePrefixInUniqueNameGeneration": true, + "@aws-cdk/aws-efs:denyAnonymousAccess": true, + "@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true, + "@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true, + "@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true, + "@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true, + "@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true, + "@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true, + "@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true, + "@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true, + "@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true, + "@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true, + "@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true, + "@aws-cdk/aws-eks:nodegroupNameAttribute": true, + "@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true, + "@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true, + "@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false + } +} diff --git a/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/.gitignore b/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/.gitignore new file mode 100644 index 000000000..c41cc9e35 --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/.gitignore @@ -0,0 +1 @@ +/target \ No newline at end of file diff --git a/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/Cargo.toml b/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/Cargo.toml new file mode 100644 index 000000000..e7048accf --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/Cargo.toml @@ -0,0 +1,21 @@ +[package] +name = "bedrock-streamer-url" +version = "0.1.0" +edition = "2021" + +[dependencies] +aws-config = "1.1.1" +aws-sdk-bedrockruntime = "1.53.0" +bytes = "1.5.0" +lambda_runtime = "0.13.0" +serde = { version = "1.0.193", features = ["derive"] } +serde_json = "1.0.108" +tokio = { version = "1.34.0", features = ["full"] } +tracing = { version = "0.1", features = ["log"] } +tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt"] } + +[profile.release] +opt-level = "z" +strip = true +lto = true +codegen-units = 1 \ No newline at end of file diff --git a/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/src/main.rs b/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/src/main.rs new file mode 100644 index 000000000..d85bce11c --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/src/main.rs @@ -0,0 +1,178 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +use aws_config::{meta::region::RegionProviderChain, BehaviorVersion}; +use aws_sdk_bedrockruntime::{ + types::{ContentBlock, ConversationRole, ConverseStreamOutput, Message}, + Client, +}; +use bytes::Bytes; +use lambda_runtime::{ + service_fn, + streaming::{channel, Body, Response}, + Error as LambdaError, LambdaEvent, +}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use tracing::{error, info}; + +const MODEL_ID: &str = "anthropic.claude-3-haiku-20240307-v1:0"; + +/// Bedrock story +#[derive(Debug, Deserialize)] +struct StoryRequest { + #[serde(rename = "storyType")] + story_type: String, +} + +/// Response for each stream record sent from Amazon Bedrock +#[derive(Debug, Serialize)] +struct BedrockResponse { + #[serde(rename = "type")] + response_type: String, + message: Option, +} + +/// Main Lambda handler here... +async fn function_handler(event: LambdaEvent) -> Result, LambdaError> { + let region_provider = RegionProviderChain::default_provider().or_else("us-east-1"); + let config = aws_config::defaults(BehaviorVersion::latest()) + .region(region_provider) + .load() + .await; + + info!("Received event...: {:?}", event); + + let body = event.payload + .get("body") + .and_then(Value::as_str) + .ok_or_else(|| LambdaError::from("Failed to parse request body"))?; + + match serde_json::from_str::(body) { + Ok(request) => { + info!("Received request...: {:?}", request); + let bedrock_client = Client::new(&config); + handle_story_request(bedrock_client, request).await + } + Err(e) => Err(LambdaError::from(format!( + "Failed to parse request body: {:?}", + e + ))), + } +} + +/// Handle the incoming story request +async fn handle_story_request( + bedrock_client: Client, + request: StoryRequest, +) -> Result, LambdaError> { + // Construct the prompt based on the type of story to create + let prompt: String = format!("Tell me a very short story about: {}", request.story_type); + info!("Bedrock story prompt...: {}", prompt); + + // Invoke Bedrock with prompt - process the stream + process_bedrock_stream(prompt, bedrock_client).await +} + +/// Process the Bedrock stream +async fn process_bedrock_stream( + prompt: String, + bedrock_client: Client, +) -> Result, LambdaError> { + // Create a communication channel between transmitter & receiver + let (mut tx, rx) = channel(); + + // Create a response using the Converse API + let bedrock_response = bedrock_client + .converse_stream() + .model_id(MODEL_ID) + .messages( + Message::builder() + .role(ConversationRole::User) + .content(ContentBlock::Text(prompt)) + .build() + .map_err(|_| LambdaError::from("failed to build message"))?, + ) + .send() + .await; + + let mut converse_stream = match bedrock_response { + Ok(output) => Ok(output.stream), + Err(e) => { + error!("Error in Bedrock response: {:?}", e); + Err(LambdaError::from(format!( + "Error in Bedrock response: {:?}", + e + ))) + } + }?; + + // Spawn a task to process the Bedrock stream + tokio::spawn(async move { + info!("Bedrock stream processing started..."); + loop { + let token = converse_stream.recv().await; + match token { + Ok(Some(output)) => { + info!("Bedrock response: {:?}", output); + let response = get_response(output).map_err(LambdaError::from)?; + let bytes = Bytes::from(serde_json::to_vec(&response).map_err(|e| { + LambdaError::from(format!("Failed to serialize response: {:?}", e)) + })?); + if let Err(e) = tx.send_data(bytes).await { + error!("Receiver dropped error: {:?}", e); + return Err(LambdaError::from( + "Receiver dropped error. Bedrock proccessing stopped.", + )); + } + } + Ok(None) => break Ok(()), + Err(e) => { + error!("Error receiving stream: {:?}", e); + return Err(LambdaError::from("Error receiving stream")); + } + } + } + }); + + info!("Bedrock stream processing complete..."); + Ok(Response::from(rx)) +} + +/// Process an output token from the Bedrock response stream +fn get_response(output: ConverseStreamOutput) -> Result { + match output { + ConverseStreamOutput::ContentBlockDelta(event) => match event.delta() { + Some(delta) => { + let text = delta.as_text().map_err(|e| { + LambdaError::from(format!("Failed to get text from delta: {:?}", e)) + })?; + Ok(BedrockResponse { + response_type: "text".into(), + message: Some(text.to_string()), + }) + } + None => Ok(BedrockResponse { + response_type: "message".into(), + message: Some("".into()), + }), + }, + _ => Ok(BedrockResponse { + response_type: "other".into(), + message: None, + }), + } +} + +/// Lambda Entry +#[tokio::main] +async fn main() -> Result<(), LambdaError> { + tracing_subscriber::fmt() + .with_max_level(tracing::Level::INFO) + .with_target(false) + .without_time() + .init(); + + lambda_runtime::run(service_fn(function_handler)).await?; + Ok(()) +} diff --git a/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-stack.ts b/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-stack.ts new file mode 100644 index 000000000..a15d9f267 --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/cdk/lib/bedrock-streamer-stack.ts @@ -0,0 +1,44 @@ +import * as cdk from 'aws-cdk-lib'; +import * as lambda from 'aws-cdk-lib/aws-lambda'; +import * as iam from 'aws-cdk-lib/aws-iam'; +import { Construct } from 'constructs'; +import { RustFunction } from 'cargo-lambda-cdk'; + +export class BedrockStreamerStack extends cdk.Stack { + constructor(scope: Construct, id: string, props?: cdk.StackProps) { + super(scope, id, props); + + // Rust Lambda Function + const streamingLambda = new RustFunction(this, 'BedrockStreamerURL', { + manifestPath: 'lib/bedrock-streamer-lambda/Cargo.toml', + bundling: { + architecture: lambda.Architecture.ARM_64, + cargoLambdaFlags: [ + '--compiler', + 'cross', + '--release', + ], + }, + timeout: cdk.Duration.seconds(60), + }); + + // Create a Lambda Function URL + const lambdaUrl = streamingLambda.addFunctionUrl({ + authType: lambda.FunctionUrlAuthType.AWS_IAM, + invokeMode: lambda.InvokeMode.RESPONSE_STREAM, + }); + + // Grant Lambda permission to invoke Bedrock + streamingLambda.addToRolePolicy(new iam.PolicyStatement({ + actions: ['bedrock:InvokeModelWithResponseStream'], + resources: ['*'], + })); + + // Output the Lambda Function URL + new cdk.CfnOutput(this, 'LambdaFunctionURL', { + value: lambdaUrl.url, + description: 'Lambda Function URL', + }); + + } +} \ No newline at end of file diff --git a/lambda-bedrock-response-streaming-rust-cdk/cdk/package.json b/lambda-bedrock-response-streaming-rust-cdk/cdk/package.json new file mode 100644 index 000000000..9730c4825 --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/cdk/package.json @@ -0,0 +1,24 @@ +{ + "name": "bedrock-streamer", + "version": "0.1.0", + "bin": { + "bedrock-streamer": "bin/bedrock-streamer.js" + }, + "scripts": { + "build": "tsc", + "watch": "tsc -w", + "cdk": "cdk" + }, + "devDependencies": { + "@types/node": "20.14.9", + "aws-cdk": "2.152.0", + "ts-node": "^10.9.2", + "typescript": "~5.5.3" + }, + "dependencies": { + "aws-cdk-lib": "2.152.0", + "cargo-lambda-cdk": "^0.0.22", + "constructs": "^10.0.0", + "source-map-support": "^0.5.21" + } +} diff --git a/lambda-bedrock-response-streaming-rust-cdk/cdk/tsconfig.json b/lambda-bedrock-response-streaming-rust-cdk/cdk/tsconfig.json new file mode 100644 index 000000000..aaa7dc510 --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/cdk/tsconfig.json @@ -0,0 +1,31 @@ +{ + "compilerOptions": { + "target": "ES2020", + "module": "commonjs", + "lib": [ + "es2020", + "dom" + ], + "declaration": true, + "strict": true, + "noImplicitAny": true, + "strictNullChecks": true, + "noImplicitThis": true, + "alwaysStrict": true, + "noUnusedLocals": false, + "noUnusedParameters": false, + "noImplicitReturns": true, + "noFallthroughCasesInSwitch": false, + "inlineSourceMap": true, + "inlineSources": true, + "experimentalDecorators": true, + "strictPropertyInitialization": false, + "typeRoots": [ + "./node_modules/@types" + ] + }, + "exclude": [ + "node_modules", + "cdk.out" + ] +} diff --git a/lambda-bedrock-response-streaming-rust-cdk/example-pattern.json b/lambda-bedrock-response-streaming-rust-cdk/example-pattern.json new file mode 100644 index 000000000..22cb371c5 --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/example-pattern.json @@ -0,0 +1,67 @@ +{ + "title": "AWS Lambda and Amazon Bedrock Response Streaming with Rust", + "description": "Create a Lambda function URL that streams a response from an Amazon Bedrock LLM.", + "language": "Rust", + "level": "200", + "framework": "CDK", + "introBox": { + "headline": "How it works", + "text": [ + "This pattern exposes a public Lambda Function URL (FURL) endpoint where requests can be made.", + "When a request is made to the FURL, the Lambda function initiates a streaming request to Amazon Bedrock", + "using the ConverseStream API. This allows the response from the LLM in Bedrock to start streaming back", + "to the Lambda function as soon as generation begins, without waiting for the entire response to be ready." + ] + }, + "gitHub": { + "template": { + "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/lambda-bedrock-response-streaming-rust-cdk", + "templateURL": "serverless-patterns/lambda-bedrock-response-streaming-rust-cdk", + "projectFolder": "lambda-bedrock-response-streaming-rust-cdk", + "templateFile": "cdk/lib/bedrock-streamer-stack.ts" + } + }, + "resources": { + "bullets": [ + { + "text": "Lambda Function URLs", + "link": "https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html" + }, + { + "text": "Amazon Bedrock", + "link": "https://aws.amazon.com/bedrock/" + }, + { + "text": "Cloud Development Kit", + "link": "https://docs.aws.amazon.com/cdk/v2/guide/home.html" + }, + { + "text": "AWS SDK for Rust", + "link": "https://aws.amazon.com/sdk-for-rust/" + } + ] + }, + "deploy": { + "text": [ + "cdk deploy" + ] + }, + "testing": { + "text": [ + "See the GitHub repo for detailed testing instructions." + ] + }, + "cleanup": { + "text": [ + "Delete the stack: cdk destroy." + ] + }, + "authors": [ + { + "name": "Phil Callister", + "image": "https://serverlessland.com/assets/images/resources/contributors/phil-callister.jpg", + "bio": "I'm an Enterprise Solutions Architect at AWS, with a focus on Financial Services. As a passionate builder, I enjoy helping customers create innovative solutions to achieve their business objectives.", + "linkedin": "https://www.linkedin.com/in/philcallister/" + } + ] +} diff --git a/lambda-bedrock-response-streaming-rust-cdk/lambda-bedrock-response-streaming-rust-cdk.json b/lambda-bedrock-response-streaming-rust-cdk/lambda-bedrock-response-streaming-rust-cdk.json new file mode 100644 index 000000000..b74dcc0e3 --- /dev/null +++ b/lambda-bedrock-response-streaming-rust-cdk/lambda-bedrock-response-streaming-rust-cdk.json @@ -0,0 +1,86 @@ +{ + "title": "AWS Lambda and Amazon Bedrock Response Streaming with Rust", + "description": "Create a Lambda function URL that streams a response from an Amazon Bedrock LLM.", + "language": "Rust", + "level": "200", + "framework": "CDK", + "introBox": { + "headline": "How it works", + "text": [ + "This pattern exposes a public Lambda Function URL (FURL) endpoint where requests can be made.", + "When a request is made to the FURL, the Lambda function initiates a streaming request to Amazon Bedrock", + "using the ConverseStream API. This allows the response from the LLM in Bedrock to start streaming back", + "to the Lambda function as soon as generation begins, without waiting for the entire response to be ready." + ] + }, + "gitHub": { + "template": { + "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/lambda-bedrock-response-streaming-rust-cdk", + "templateURL": "serverless-patterns/lambda-bedrock-response-streaming-rust-cdk", + "projectFolder": "lambda-bedrock-response-streaming-rust-cdk", + "templateFile": "cdk/lib/bedrock-streamer-stack.ts" + } + }, + "resources": { + "bullets": [ + { + "text": "Lambda Function URLs", + "link": "https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html" + }, + { + "text": "Amazon Bedrock", + "link": "https://aws.amazon.com/bedrock/" + }, + { + "text": "Cloud Development Kit", + "link": "https://docs.aws.amazon.com/cdk/v2/guide/home.html" + }, + { + "text": "AWS SDK for Rust", + "link": "https://aws.amazon.com/sdk-for-rust/" + } + ] + }, + "deploy": { + "text": [ + "cdk deploy" + ] + }, + "testing": { + "text": [ + "See the GitHub repo for detailed testing instructions." + ] + }, + "cleanup": { + "text": [ + "Delete the stack: cdk destroy." + ] + }, + "authors": [ + { + "name": "Phil Callister", + "image": "https://serverlessland.com/assets/images/resources/contributors/phil-callister.jpg", + "bio": "I'm an Enterprise Solutions Architect at AWS, with a focus on Financial Services. As a passionate builder, I enjoy helping customers create innovative solutions to achieve their business objectives.", + "linkedin": "philcallister" + } + ], + "patternArch": { + "icon1": { + "x": 20, + "y": 50, + "service": "lambda", + "label": "AWS Lambda" + }, + "icon2": { + "x": 80, + "y": 50, + "service": "bedrock", + "label": "Amazon Bedrock" + }, + "line1": { + "from": "icon1", + "to": "icon2", + "label": "" + } + } +}