Skip to content

Supabase database webhook example upgrade #1386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Oct 8, 2024
Merged

Supabase database webhook example upgrade #1386

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
8df1778
Added overview for guides and examples section and split them all out
D-K-P Oct 3, 2024
45c27c1
New supabase guide wip
D-K-P Oct 3, 2024
6014359
Updated images and improved docs
D-K-P Oct 3, 2024
e5c8f8b
Trimmed the supabase prereqs
D-K-P Oct 4, 2024
ed3f8c6
Supabase guide wip
D-K-P Oct 4, 2024
7062b8a
more updates
D-K-P Oct 4, 2024
59300d9
Replaced old database webhook guide
D-K-P Oct 4, 2024
1a65b31
Created one intro page and removed snippets
D-K-P Oct 5, 2024
77eb41e
Updated guide sidebar titles
D-K-P Oct 5, 2024
a443794
Code updates
D-K-P Oct 6, 2024
d9fbd1c
More improvements
D-K-P Oct 6, 2024
bba83c9
Updates and added images
D-K-P Oct 6, 2024
a1db2b2
Compressed image
D-K-P Oct 6, 2024
ade508a
Merge branch 'main' into docs/supabase-ffmpeg-deepgram-example
D-K-P Oct 6, 2024
49163c6
Merge branch 'main' into docs/supabase-ffmpeg-deepgram-example
D-K-P Oct 8, 2024
db6e6d0
Updated guides descriptions and edge function basic
D-K-P Oct 8, 2024
0287e27
Removed bold
D-K-P Oct 8, 2024
95f9ff1
Updated redirects
D-K-P Oct 8, 2024
422e875
Merge branch 'main' into docs/supabase-ffmpeg-deepgram-example
D-K-P Oct 8, 2024
3424680
Fixed broken links
D-K-P Oct 8, 2024
db073ca
Updated intro
D-K-P Oct 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 0 additions & 20 deletions docs/guides/frameworks/introduction.mdx
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion docs/guides/frameworks/prisma.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Prisma setup guide"
sidebarTitle: "Prisma"
sidebarTitle: "Prisma setup guide"
description: "This guide will show you how to setup Prisma with Trigger.dev"
icon: "Triangle"
---
Expand Down
223 changes: 118 additions & 105 deletions docs/guides/frameworks/sequin.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Sequin database triggers"
sidebarTitle: "Sequin"
sidebarTitle: "Sequin database triggers"
description: "This guide will show you how to trigger tasks from database changes using Sequin"
icon: "database"
---
Expand All @@ -22,9 +22,11 @@ As long as you create an HTTP endpoint that Sequin can deliver webhooks to, you
You'll need the following to follow this guide:

- A Next.js project with [Trigger.dev](https://trigger.dev) installed
<Info>
If you don't have one already, follow [Trigger.dev's Next.js setup guide](/guides/frameworks/nextjs) to setup your project. You can return to this guide when you're ready to write your first Trigger.dev task.
</Info>
<Info>
If you don't have one already, follow [Trigger.dev's Next.js setup
guide](/guides/frameworks/nextjs) to setup your project. You can return to this guide when
you're ready to write your first Trigger.dev task.
</Info>
- A [Sequin](https://console.sequinstream.com/register) account
- A Postgres database (Sequin works with any Postgres database version 12 and up) with a `posts` table.

Expand All @@ -42,36 +44,36 @@ Start by creating a new Trigger.dev task that takes in a Sequin change event as
import { OpenAI } from "openai";
import { upsertEmbedding } from "../util";

const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});

export const createEmbeddingForPost = task({
id: "create-embedding-for-post",
run: async (payload: {
record: {
id: number;
title: string;
body: string;
author: string;
createdAt: string;
embedding: string | null;
},
metadata: {
table_schema: string,
table_name: string,
consumer: {
id: string;
name: string;
};
};
}) => {
// Create an embedding using the title and body of payload.record
const content = `${payload.record.title}\n\n${payload.record.body}`;
const embedding = (await openai.embeddings.create({
model: "text-embedding-ada-002",
input: content,
})).data[0].embedding;
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});

export const createEmbeddingForPost = task({
id: "create-embedding-for-post",
run: async (payload: {
record: {
id: number;
title: string;
body: string;
author: string;
createdAt: string;
embedding: string | null;
},
metadata: {
table_schema: string,
table_name: string,
consumer: {
id: string;
name: string;
};
};
}) => {
// Create an embedding using the title and body of payload.record
const content = `${payload.record.title}\n\n${payload.record.body}`;
const embedding = (await openai.embeddings.create({
model: "text-embedding-ada-002",
input: content,
})).data[0].embedding;

// Upsert the embedding in the database. See utils.ts for the implementation -> ->
await upsertEmbedding(embedding, payload.record.id);
Expand All @@ -82,51 +84,55 @@ Start by creating a new Trigger.dev task that takes in a Sequin change event as
embedding: JSON.stringify(embedding),
};
}

});

````

```ts utils.ts
import pg from "pg";

export async function upsertEmbedding(embedding: number[], id: number) {
const client = new pg.Client({
connectionString: process.env.DATABASE_URL,
});
```

```ts utils.ts
import pg from "pg";

export async function upsertEmbedding(embedding: number[], id: number) {
const client = new pg.Client({
connectionString: process.env.DATABASE_URL,
});
await client.connect();

try {
const query = `
INSERT INTO post_embeddings (id, embedding)
VALUES ($2, $1)
ON CONFLICT (id)
DO UPDATE SET embedding = $1
`;
const values = [JSON.stringify(embedding), id];

const result = await client.query(query, values);
console.log(`Updated record in database. Rows affected: ${result.rowCount}`);

return result.rowCount;
} catch (error) {
console.error("Error updating record in database:", error);
throw error;
} finally {
await client.end();
}
await client.connect();

try {
const query = `
INSERT INTO post_embeddings (id, embedding)
VALUES ($2, $1)
ON CONFLICT (id)
DO UPDATE SET embedding = $1
`;
const values = [JSON.stringify(embedding), id];

const result = await client.query(query, values);
console.log(`Updated record in database. Rows affected: ${result.rowCount}`);

return result.rowCount;
} catch (error) {
console.error("Error updating record in database:", error);
throw error;
} finally {
await client.end();
}
```
}
````

</CodeGroup>

This task takes in a Sequin record event, creates an embedding, and then upserts the embedding into a `post_embeddings` table.
This task takes in a Sequin record event, creates an embedding, and then upserts the embedding into a `post_embeddings` table.

</Step>
<Step title="Add the task to your Trigger.dev project">
Register the `create-embedding-for-post` task to your Trigger.dev cloud project by running the following command:

```bash
npx trigger.dev@latest dev
```
```bash
npx trigger.dev@latest dev
```

In the Trigger.dev dashboard, you should now see the `create-embedding-for-post` task:
In the Trigger.dev dashboard, you should now see the `create-embedding-for-post` task:

<Frame>
<img src="/images/sequin-register-task.png" alt="Task added" />
Expand All @@ -135,63 +141,69 @@ Start by creating a new Trigger.dev task that takes in a Sequin change event as
</Steps>

<Check>
You've successfully created a Trigger.dev task that will create an embedding for each post in your database. In the next step, you'll create an API endpoint that Sequin can deliver records to.
You've successfully created a Trigger.dev task that will create an embedding for each post in your
database. In the next step, you'll create an API endpoint that Sequin can deliver records to.
</Check>

## Setup API route

You'll now create an API endpoint that will receive posts from Sequin and then trigger the `create-embedding-for-post` task.

<Info>
This guide covers how to setup an API endpoint using the Next.js App Router. You can find examples for Next.js Server Actions and Pages Router in the [Trigger.dev documentation](https://trigger.dev/docs/guides/frameworks/nextjs).
This guide covers how to setup an API endpoint using the Next.js App Router. You can find examples
for Next.js Server Actions and Pages Router in the [Trigger.dev
documentation](https://trigger.dev/docs/guides/frameworks/nextjs).
</Info>

<Steps titleSize="h3">
<Step title="Create a route handler">
Add a route handler by creating a new `route.ts` file in a `/app/api/create-embedding-for-post` directory:

```ts app/api/create-embedding-for-post/route.ts
import type { createEmbeddingForPost } from "@/trigger/create-embedding-for-post";
import { tasks } from "@trigger.dev/sdk/v3";
import { NextResponse } from "next/server";

export async function POST(req: Request) {
const authHeader = req.headers.get('authorization');
if (!authHeader || authHeader !== `Bearer ${process.env.SEQUIN_WEBHOOK_SECRET}`) {
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
}
const payload = await req.json();
const handle = await tasks.trigger<typeof createEmbeddingForPost>(
"create-embedding-for-post",
payload
);
```ts app/api/create-embedding-for-post/route.ts
import type { createEmbeddingForPost } from "@/trigger/create-embedding-for-post";
import { tasks } from "@trigger.dev/sdk/v3";
import { NextResponse } from "next/server";

return NextResponse.json(handle);
export async function POST(req: Request) {
const authHeader = req.headers.get("authorization");
if (!authHeader || authHeader !== `Bearer ${process.env.SEQUIN_WEBHOOK_SECRET}`) {
return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
}
```
const payload = await req.json();
const handle = await tasks.trigger<typeof createEmbeddingForPost>(
"create-embedding-for-post",
payload
);

return NextResponse.json(handle);
}
```

This route handler will receive records from Sequin, parse them, and then trigger the `create-embedding-for-post` task.

This route handler will receive records from Sequin, parse them, and then trigger the `create-embedding-for-post` task.
</Step>
<Step title="Set secret keys">
You'll need to set four secret keys in a `.env.local` file:

```bash
SEQUIN_WEBHOOK_SECRET=your-secret-key
TRIGGER_SECRET_KEY=secret-from-trigger-dev
OPENAI_API_KEY=sk-proj-asdfasdfasdf
DATABASE_URL=postgresql://
```
```bash
SEQUIN_WEBHOOK_SECRET=your-secret-key
TRIGGER_SECRET_KEY=secret-from-trigger-dev
OPENAI_API_KEY=sk-proj-asdfasdfasdf
DATABASE_URL=postgresql://
```
Comment on lines +162 to +193
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Enhance security and error handling in the route handler.

The API route setup is clear and well-explained. However, consider the following improvements to enhance security and error handling:

  1. Use a constant-time comparison for the authorization header to prevent timing attacks.
  2. Add input validation for the payload.
  3. Implement proper error handling for the tasks.trigger call.

Here's an improved version of the route handler:

import type { createEmbeddingForPost } from "@/trigger/create-embedding-for-post";
import { tasks } from "@trigger.dev/sdk/v3";
import { NextResponse } from "next/server";
import { timingSafeEqual } from "crypto";

export async function POST(req: Request) {
  try {
    const authHeader = req.headers.get("authorization");
    if (!authHeader || !isValidAuthHeader(authHeader)) {
      return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
    }

    const payload = await req.json();
    if (!isValidPayload(payload)) {
      return NextResponse.json({ error: "Invalid payload" }, { status: 400 });
    }

    const handle = await tasks.trigger<typeof createEmbeddingForPost>(
      "create-embedding-for-post",
      payload
    );

    return NextResponse.json(handle);
  } catch (error) {
    console.error("Error in POST handler:", error);
    return NextResponse.json({ error: "Internal Server Error" }, { status: 500 });
  }
}

function isValidAuthHeader(header: string): boolean {
  const expectedHeader = `Bearer ${process.env.SEQUIN_WEBHOOK_SECRET}`;
  return timingSafeEqual(Buffer.from(header), Buffer.from(expectedHeader));
}

function isValidPayload(payload: any): boolean {
  // Implement payload validation logic here
  return true; // Placeholder
}

This version includes constant-time comparison for the authorization header, a placeholder for payload validation, and proper error handling. Remember to implement the isValidPayload function according to your specific payload structure.


The `SEQUIN_WEBHOOK_SECRET` ensures that only Sequin can access your API endpoint.
The `SEQUIN_WEBHOOK_SECRET` ensures that only Sequin can access your API endpoint.

The `TRIGGER_SECRET_KEY` is used to authenticate requests to Trigger.dev and can be found in the **API keys** tab of the Trigger.dev dashboard.
The `TRIGGER_SECRET_KEY` is used to authenticate requests to Trigger.dev and can be found in the **API keys** tab of the Trigger.dev dashboard.

The `OPENAI_API_KEY` and `DATABASE_URL` are used to create an embedding using OpenAI and connect to your database. Be sure to add these as [environment variables](https://trigger.dev/docs/deploy-environment-variables) in Trigger.dev as well.

The `OPENAI_API_KEY` and `DATABASE_URL` are used to create an embedding using OpenAI and connect to your database. Be sure to add these as [environment variables](https://trigger.dev/docs/deploy-environment-variables) in Trigger.dev as well.
</Step>
</Steps>

<Check>
You've successfully created an API endpoint that can receive record payloads from Sequin and trigger a Trigger.dev task. In the next step, you'll setup Sequin to trigger the endpoint.
You've successfully created an API endpoint that can receive record payloads from Sequin and
trigger a Trigger.dev task. In the next step, you'll setup Sequin to trigger the endpoint.
</Check>

## Create Sequin consumer
Expand Down Expand Up @@ -253,11 +265,10 @@ You'll now configure Sequin to send every row in your `posts` table to your Trig
</Frame>
7. Click the **Create Consumer** button.
</Step>

</Steps>

<Check>
Your Sequin consumer is now created and ready to send events to your API endpoint.
</Check>
<Check>Your Sequin consumer is now created and ready to send events to your API endpoint.</Check>

## Test end-to-end

Expand Down Expand Up @@ -301,10 +312,12 @@ You'll now configure Sequin to send every row in your `posts` table to your Trig
<img src="/images/sequin-final-run.png" alt="Task run" />
</Frame>
</Step>

</Steps>

<Check>
Every time a post is created or updated, Sequin will deliver the row payload to your API endpoint and Trigger.dev will run the `create-embedding-for-post` task.
Every time a post is created or updated, Sequin will deliver the row payload to your API endpoint
and Trigger.dev will run the `create-embedding-for-post` task.
</Check>

## Next steps
Expand All @@ -314,4 +327,4 @@ With Sequin and Trigger.dev, every post in your database will now have an embedd
From here, add error handling and deploy to production:

- Add [retries](/errors-retrying) to your Trigger.dev task to ensure that any errors are captured and logged.
- Deploy to [production](/guides/frameworks/nextjs#deploying-your-task-to-trigger-dev) and update your Sequin consumer to point to your production database and endpoint.
- Deploy to [production](/guides/frameworks/nextjs#deploying-your-task-to-trigger-dev) and update your Sequin consumer to point to your production database and endpoint.
1 change: 1 addition & 0 deletions docs/guides/frameworks/supabase-edge-functions-basic.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ This guide shows you how to set up and deploy a simple Supabase edge function ex
## Prerequisites

- Ensure you have the [Supabase CLI](https://supabase.com/docs/guides/cli/getting-started) installed
- Since Supabase CLI version 1.123.4, you must have [Docker Desktop installed](https://supabase.com/docs/guides/functions/deploy#deploy-your-edge-functions) to deploy Edge Functions
- Ensure TypeScript is installed
- [Create a Trigger.dev account](https://cloud.trigger.dev)
- [Create a new Trigger.dev project](/guides/dashboard/creating-a-project)
Expand Down
Loading