AWS Bedrock support

[csheaff]

Hello, it would be grand to be able to use AWS models from Amazon Bedrock, such as Anthropic Claude Sonnet 3.5.

[karthink]

Can you provide a link to their API documentation?

[csheaff]

...looking for a way to use just http requests but i'm not sure it's possible.

[karthink]

I'm not familiar with AWS Bedrock. How do you access models (or other computation) running there?

[csheaff]

the easiest approaches are to use the aws cli on the command line or a python sdk. But i'm guessing what would be most convenient here is being able to send https requests with lisp.

relevant? pokepay/aws-sdk-lisp#35

[csheaff]

This is as close as i can find to the payload structure:

https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html

On the command line one would would do:

aws bedrock-runtime converse \
--model-id amazon.titan-text-express-v1 \
--messages '[{"role": "user", "content": [{"text": "Describe the purpose of a \"hello world\" program in one line."}]}]' \
--inference-config '{"maxTokens": 512, "temperature": 0.5, "topP": 0.9}'

[karthink]

This is as close as i can find to the payload structure:

https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html

This makes it seem like you can make http requests? Sorry, I'm not understanding how this service is structured.

If you can make http requests I can add support for it to gptel.

[csheaff]

yes i think so. one could authenticate using environmental variables

AWS_SESSION_TOKEN, AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID, AWS_REGION

...or just have the user enter them in during config. In my case i have to renew my credentials often for security reasons, so being able to update them without closing emacs and re-exporting the variables in the terminal would be a plus.

[karthink]

Sorry, I don't follow how environment variables are relevant to making a http request.

Is there a curl command you can run to receive model responses from AWS Bedrock?

[swapneils]

Is there a curl command you can run to receive model responses from AWS Bedrock?

Curl has native support for the AWS signing method (see e.g. this article), so this should be possible.
This reddit discussion seems to have a sample curl command to invoke Bedrock.

On a separate note, it would be nice to support CLI backends as well as REST HTTP backends, since then the work of actually calling a service can be offloaded to its native CLI tools (q chat, llama-cli, etc).

Note since I'm AWS employed: The above is purely my own knowledge and opinions, and not communication on behalf of my employer. This also applies to all future communications in this thread unless explicitly specified otherwise.

[karthink]

@swapneils Thank you for the pointer -- this should be possible now.

@csheaff So this can be done using Curl, but someone will need to write an AWS bedrock backend for gptel. Unfortunately we can't inherit the OpenAI backend since the payload structure is different. PRs are welcome, you can copy gptel-openai.el or gptel-anthropic.el and modify it.

[csheaff]

thanks @karthink . I'll try to find some time but it might be tough.

[JGalego]

@karthink @csheaff Found this issue by accident. Maybe this will be useful for a future implementation >>> cl-bedrock? Includes support for InvokeModel, Converse and ApplyGuardrail APIs.