Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(wasm): add Cloudflare D1 Wasm wrapper #387

Merged
merged 3 commits into from
Dec 16, 2024
+1,046 −26
Merged

feat(wasm): add Cloudflare D1 Wasm wrapper #387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ea53ffb
add cfd1 fdw
burmecia Dec 15, 2024
aa38002
Merge branch 'main' into bo/feat/cfd1_fdw
burmecia Dec 16, 2024
ad66f5f
add cloudflare d1 fdw docs
burmecia Dec 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ members = [
exclude = [
"wasm-wrappers/fdw/cal_fdw",
"wasm-wrappers/fdw/calendly_fdw",
"wasm-wrappers/fdw/cfd1_fdw",
"wasm-wrappers/fdw/helloworld_fdw",
"wasm-wrappers/fdw/snowflake_fdw",
"wasm-wrappers/fdw/paddle_fdw",
Expand Down
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
| [Paddle](./wasm-wrappers/fdw/paddle_fdw) | A Wasm FDW for [Paddle](https://www.paddle.com/) | ✅ | ✅ |
| [Calendly](./wasm-wrappers/fdw/calendly_fdw) | A Wasm FDW for [Calendly](https://www.calendly.com/) | ✅ | ❌ |
| [Cal.com](./wasm-wrappers/fdw/cal_fdw) | A Wasm FDW for [Cal.com](https://www.cal.com/) | ✅ | ❌ |
| [Cloudflare D1](./wasm-wrappers/fdw/cfd1_fdw) | A Wasm FDW for [Cloudflare D1](https://developers.cloudflare.com/d1/) | ✅ | ✅ |

### Warning

Expand Down
312 changes: 312 additions & 0 deletions docs/catalog/cfd1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,312 @@
---
source:
documentation:
author: supabase
tags:
- wasm
- official
---

# Cloudflare D1

[Cloudflare D1](https://developers.cloudflare.com/d1/) is Cloudflare's managed, serverless database with SQLite's SQL semantics, built-in disaster recovery, and Worker and HTTP API access.

The Cloudflare D1 Wrapper is a WebAssembly(Wasm) foreign data wrapper which allows you to read data from Cloudflare D1 database for use within your Postgres database.

!!! warning

Restoring a logical backup of a database with a materialized view using a foreign table can fail. For this reason, either do not use foreign tables in materialized views or use them in databases with physical backups enabled.

## Supported Data Types

| Postgres Data Type | D1 Data Type |
| ------------------ | ------------ |
| bigint | integer |
| double precision | real |
| text | text |
| text | blob |

The D1 API uses JSON formatted data, please refer to [D1 API docs](https://developers.cloudflare.com/api/operations/cloudflare-d1-list-databases) for more details.

## Available Versions

| Version | Wasm Package URL | Checksum |
| ------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| 0.1.0 | `https://github.com/supabase/wrappers/releases/download/wasm_cfd1_fdw_v0.1.0/cfd1_fdw.wasm` | `tbd` |

## Preparation

Before you can query D1, you need to enable the Wrappers extension and store your credentials in Postgres.

### Enable Wrappers

Make sure the `wrappers` extension is installed on your database:

```sql
create extension if not exists wrappers with schema extensions;
```

### Enable the D1 Wrapper

Enable the Wasm foreign data wrapper:

```sql
create foreign data wrapper wasm_wrapper
handler wasm_fdw_handler
validator wasm_fdw_validator;
```

### Store your credentials (optional)

By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials.

```sql
-- Save your D1 API token in Vault and retrieve the `key_id`
insert into vault.secrets (name, secret)
values (
'cfd1',
'<D1 API token>' -- Cloudflare D1 API token
)
returning key_id;
```

### Connecting to D1

We need to provide Postgres with the credentials to access D1 and any additional options. We can do this using the `create server` command:

=== "With Vault"

```sql
create server cfd1_server
foreign data wrapper wasm_wrapper
options (
fdw_package_url 'https://github.com/supabase/wrappers/releases/download/wasm_cfd1_fdw_v0.1.0/cfd1_fdw.wasm',
fdw_package_name 'supabase:cfd1-fdw',
fdw_package_version '0.1.0',
fdw_package_checksum 'tbd',
api_url 'https://api.cloudflare.com/client/v4/accounts/<account_id>/d1/database', -- optional
account_id '<Account ID>',
database_id '<Database ID>',
api_token_id '<key_ID>' -- The Key ID from above.
);
```

=== "Without Vault"

```sql
create server cfd1_server
foreign data wrapper wasm_wrapper
options (
fdw_package_url 'https://github.com/supabase/wrappers/releases/download/wasm_cfd1_fdw_v0.1.0/cfd1_fdw.wasm',
fdw_package_name 'supabase:cfd1-fdw',
fdw_package_version '0.1.0',
fdw_package_checksum 'tbd',
api_url 'https://api.cloudflare.com/client/v4/accounts/<account_id>/d1/database', -- optional
account_id '<Account ID>',
database_id '<Database ID>',
api_token '<D1 API token>'
);
```

Note the `fdw_package_*` options are required, which specify the Wasm package metadata. You can get the available package version list from [above](#available-versions).

### Create a schema

We recommend creating a schema to hold all the foreign tables:

```sql
create schema if not exists cfd1;
```

## Options

The full list of foreign table options are below:

- `table` - Source table name in D1, required.

- This option can also be a subquery enclosed in parentheses, see below for examples.
- A pseudo-table name `_meta_databases` can be used to query databases.

- `rowid_column` - Primary key column name, optional for data scan, required for data modify.

## Entities

The D1 Wrapper supports data reads and writes from the Cloudflare D1 API.

### D1 Databases

This is an object representing a D1 database.

Ref: [D1 databases docs](https://developers.cloudflare.com/api/operations/cloudflare-d1-list-databases)

#### Operations

| Object | Select | Insert | Update | Delete | Truncate |
| ----------- | :----: | :----: | :----: | :----: | :------: |
| database | ✅ | ❌ | ❌ | ❌ | ❌ |

#### Usage

```sql
create foreign table cfd1.databases (
uuid text,
name text,
version text,
num_tables bigint,
file_size bigint,
created_at text,
_attrs jsonb
)
server cfd1_server
options (
table '_meta_databases'
);
```

#### Notes

- The `_attrs` meta column contains all database attributes in JSON format
- The table option must be `_meta_databases`
- Only column names listed above are allowed

### D1 Tables

This is an object representing a D1 table.

Ref: [D1 query docs](https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database)

#### Operations

| Object | Select | Insert | Update | Delete | Truncate |
| ----------- | :----: | :----: | :----: | :----: | :------: |
| table | ✅ | ✅ | ✅ | ✅ | ❌ |

#### Usage

```sql
create foreign table cfd1.mytable (
id bigint,
name text,
amount double precision,
metadata text,
_attrs jsonb
)
server cfd1_server
options (
table 'mytable',
rowid_column 'id'
);
```

#### Notes

- The `_attrs` meta column contains all attributes in JSON format
- Can use subquery in `table` option
- Requires `rowid_column` for data modification operations
- Supports query pushdown for `where`, `order by`, and `limit` clauses
- Column names, except `_attrs`, must match between D1 and foreign table
- Data types must be compatible according to type mapping table

## Query Pushdown Support

This FDW supports `where`, `order by` and `limit` clause pushdown.

## Examples

Below are some examples on how to use D1 foreign tables.

### Basic Example

This example will create a "foreign table" inside your Postgres database and query its data.

```sql
create foreign table cfd1.databases (
uuid text,
name text,
version text,
num_tables bigint,
file_size bigint,
created_at text,
_attrs jsonb
)
server cfd1_server
options (
table '_meta_databases'
);

-- query D1 databases
select * from cfd1.databases;
```

### Query A Table

Let's create a source table `test_table` in D1 web console and add some testing data.

| Column Name | Data Type |
| ----------- | --------- |
| id | integer |
| name | text |
| amount | real |
| metadata | blob |

This example will create a "foreign table" inside your Postgres database and query its data.

```sql
create foreign table cfd1.test_table (
id bigint,
name text,
amount double precision,
metadata text,
_attrs jsonb
)
server cfd1_server
options (
table 'test_table',
rowid_column 'id'
);

select * from cfd1.test_table;
```

### Table With Subquery

The `table` option can also be a subquery enclosed in parentheses.

```sql
create foreign table cfd1.test_table_subquery (
id bigint,
name text,
amount double precision,
metadata text,
_attrs jsonb
)
server cfd1_server
options (
table '(select * from test_table)'
);

select * from cfd1.test_table_subquery;
```

!!! note

The foreign table with subquery option cannot support data modification.

### Modify Data

This example will modify data in a "foreign table" inside your Postgres database, note that `rowid_column` table option is required for data modify.

```sql
-- insert new data
insert into cfd1.test_table(id, name, amount)
values (123, 'test name 123', 321.654);

-- update existing data
update cfd1.test_table
set name = 'new name', amount = null
where id = 123;

-- delete data
delete from cfd1.test_table where id = 123;
```

52 changes: 27 additions & 25 deletions docs/catalog/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,35 +7,37 @@ hide:

## Official

| Integration | Select | Insert | Update | Delete | Truncate | Push Down |
| ----------- | :----: | :----: | :----: | :----: | :------: | :-------: |
| Airtable | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Auth0 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| AWS Cognito | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| BigQuery | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| Cal.com | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Calendly | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| ClickHouse | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| Firebase | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Logflare | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Notion | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Paddle | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
| Redis | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| S3 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Snowflake | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| Stripe | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| SQL Server | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Integration | Select | Insert | Update | Delete | Truncate | Push Down |
| ------------- | :----: | :----: | :----: | :----: | :------: | :-------: |
| Airtable | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Auth0 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| AWS Cognito | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| BigQuery | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| Cal.com | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Calendly | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| ClickHouse | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| Cloudflare D1 | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| Firebase | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Logflare | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Notion | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Paddle | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
| Redis | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| S3 | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Snowflake | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| Stripe | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| SQL Server | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |

## Community

Wasm wrappers can be installed directly from GitHub or any external source.

See [Developing a Wasm Wrapper](../guides/create-wasm-wrapper.md) for instructions on how to build and develop your own.

| Integration | Developer | Docs | Source |
| ----------- | :------------------------------: | :----------------------------------: | :------------------------------------------------------------------------------------: |
| Cal.com | [Supabase](https://supabase.com) | [Link](cal.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/cal_fdw) |
| Calendly | [Supabase](https://supabase.com) | [Link](calendly.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/calendly_fdw) |
| Notion | [Supabase](https://supabase.com) | [Link](notion.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/notion_fdw) |
| Paddle | [Supabase](https://supabase.com) | [Link](paddle.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/paddle_fdw) |
| Snowflake | [Supabase](https://supabase.com) | [Link](snowflake.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/snowflake_fdw) |
| Integration | Developer | Docs | Source |
| :-----------: | :------------------------------: | :------------------: | :------------------------------------------------------------------------------------: |
| Cal.com | [Supabase](https://supabase.com) | [Link](cal.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/cal_fdw) |
| Calendly | [Supabase](https://supabase.com) | [Link](calendly.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/calendly_fdw) |
| Cloudflare D1 | [Supabase](https://supabase.com) | [Link](cfd1.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/cfd1_fdw) |
| Notion | [Supabase](https://supabase.com) | [Link](notion.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/notion_fdw) |
| Paddle | [Supabase](https://supabase.com) | [Link](paddle.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/paddle_fdw) |
| Snowflake | [Supabase](https://supabase.com) | [Link](snowflake.md) | [Link](https://github.com/supabase/wrappers/tree/main/wasm-wrappers/fdw/snowflake_fdw) |
Loading
Loading