BricksLLM: AI Gateway For Putting LLMs In Production

Tip

A managed version of BricksLLM is also available! It is production ready, and comes with a dashboard to make interacting with BricksLLM easier. Try us out for free today!

BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM:

• Set LLM usage limits for users on different pricing tiers
• Track LLM usage on a per user and per organization basis
• Block or redact requests containing PIIs
• Improve LLM reliability with failovers, retries and caching
• Distribute API keys with rate limits and cost limits for internal development/production use cases
• Distribute API keys with rate limits and cost limits for students

Features

• PII detection and masking
• Rate limit
• Cost control
• Cost analytics
• Request analytics
• Caching
• Request Retries
• Failover
• Model access control
• Endpoint access control
• Native support for all OpenAI endpoints
• Native support for Anthropic
• Native support for Azure OpenAI
• Native support for vLLM
• Native support for Deepinfra
• Support for custom deployments
• Integration with custom models
• Datadog integration
• Logging with privacy control

Getting Started

The easiest way to get started with BricksLLM is through BricksLLM-Docker.

Step 1 - Clone BricksLLM-Docker repository

git clone https://github.com/bricks-cloud/BricksLLM-Docker

Step 2 - Change to BricksLLM-Docker directory

cd BricksLLM-Docker

Step 3 - Deploy BricksLLM locally with Postgresql and Redis

docker compose up

You can run this in detach mode use the -d flag: docker compose up -d

Step 4 - Create a provider setting

curl -X PUT http://localhost:8001/api/provider-settings \
   -H "Content-Type: application/json" \
   -d '{
          "provider":"openai",
          "setting": {
             "apikey": "YOUR_OPENAI_KEY"
          }
      }'   

Copy the id from the response.

Step 5 - Create a Bricks API key

Use id from the previous step as settingId to create a key with a rate limit of 2 req/min and a spend limit of 25 cents.

curl -X PUT http://localhost:8001/api/key-management/keys \
   -H "Content-Type: application/json" \
   -d '{
	      "name": "My Secret Key",
	      "key": "my-secret-key",
	      "tags": ["mykey"],
        "settingIds": ["ID_FROM_STEP_FOUR"],
        "rateLimitOverTime": 2,
        "rateLimitUnit": "m",
        "costLimitInUsd": 0.25
      }'   

Congratulations you are done!!!

Then, just redirect your requests to us and use OpenAI as you would normally. For example:

curl -X POST http://localhost:8002/api/providers/openai/v1/chat/completions \
   -H "Authorization: Bearer my-secret-key" \
   -H "Content-Type: application/json" \
   -d '{
          "model": "gpt-3.5-turbo",
          "messages": [
              {
                  "role": "system",
                  "content": "hi"
              }
          ]
      }'

Or if you're using an SDK, you could change its baseURL to point to us. For example:

// OpenAI Node SDK v4
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: "some-secret-key", // key created earlier
  baseURL: "http://localhost:8002/api/providers/openai/v1", // redirect to us
});

How to Update?

For updating to the latest version

docker pull luyuanxin1995/bricksllm:latest

For updating to a particular version

docker pull luyuanxin1995/bricksllm:1.4.0

Documentation

Environment variables

Name	type	description	default
POSTGRESQL_HOSTS	required	Hosts for Postgresql DB. Separated by ,	localhost
POSTGRESQL_DB_NAME	optional	Name for Postgresql DB.
POSTGRESQL_USERNAME	required	Postgresql DB username
POSTGRESQL_PASSWORD	required	Postgresql DB password
POSTGRESQL_SSL_MODE	optional	Postgresql SSL mode	disable
POSTGRESQL_PORT	optional	The port that Postgresql DB runs on	5432
POSTGRESQL_READ_TIME_OUT	optional	Timeout for Postgresql read operations	2m
POSTGRESQL_WRITE_TIME_OUT	optional	Timeout for Postgresql write operations	5s
REDIS_HOSTS	required	Host for Redis. Separated by ,	localhost
REDIS_PASSWORD	optional	Redis Password
REDIS_PORT	optional	The port that Redis DB runs on	6379
REDIS_READ_TIME_OUT	optional	Timeout for Redis read operations	1s
REDIS_WRITE_TIME_OUT	optional	Timeout for Redis write operations	500ms
IN_MEMORY_DB_UPDATE_INTERVAL	optional	The interval BricksLLM API gateway polls Postgresql DB for latest key configurations	1s
STATS_PROVIDER	optional	"datadog" or Host:Port(127.0.0.1:8125) for statsd.
PROXY_TIMEOUT	optional	Timeout for proxy HTTP requests.	600s
NUMBER_OF_EVENT_MESSAGE_CONSUMERS	optional	Number of event message consumers that help handle counting tokens and inserting event into db.	3
AWS_SECRET_ACCESS_KEY	optional	It is for PII detection feature.	5s
AWS_ACCESS_KEY_ID	optional	It is for using PII detection feature.	5s
AMAZON_REGION	optional	Region for AWS.	us-west-2
AMAZON_REQUEST_TIMEOUT	optional	Timeout for amazon requests.	5s
AMAZON_CONNECTION_TIMEOUT	optional	Timeout for amazon connection.	10s
ADMIN_PASS	optional	Simple password for the admin server.

Admin Server

Swagger Doc

Proxy Server

Swagger Doc