Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add wasm fdw framework with snowflake and paddle fdw #280

Merged
merged 15 commits into from
Jun 27, 2024

Conversation

burmecia
Copy link
Member

@burmecia burmecia commented May 27, 2024

What kind of change does this PR introduce?

This PR is to add WASM foreign data wrapper prototype with 3 new FDWs developed using this new framework.

  • Snowflake FDW, supports data read and modify
  • Paddle FDW, supports data read only
  • HelloWorld FDW, a simple hello world FDW for demo purpose

All those 3 FDWs are WASM FDWs.

What is the current behavior?

In current framework, all FDWs needs to be compile into a single Postgres extension and then released to Supabase Postgres image and then deployed to Supabase platform. This architecture has many drawbacks:

  • Wrappers Release time is too long, a new FDW may need several weeks to be landed on Supabase platform
  • Wrappers extension size get bigger and bigger with more FDWs added in, but customers normally only need 1-2 FDWs, all the others are wasting resources
  • Customers cannot use their own FDW easily on Supabase platform, they needs to rely on Supabase team to include their FDWs into Wrappers and deployed to platform
  • Upgrade to new version of Wrappers isn't easy, customers need to upgrade their whole Postgres server and then can use the new version of Wrappers. This process is time consuming and error-prone, and also impossible to use older version after upgrade.

What is the new behavior?

The new architecture is based on WebAssembly (WASM), each FDW is complied into a WASM component package (Guest) and it is dynamically loaded and run by a new WASM FDW (Host). This consists of 3 parts:

  • Host: a new FDW using current Wrappers framework and embedding a WASM runtime, which run one or many guest FDWs
  • Guest: the actual FDW which does the actual data loading and transformation, compiled into a WASM component (.wasm file)
  • Registry: a WASM FDW registry based on Warg protocol, which serves as a FDW publish and distribution platform. It runs as a Docker compose cluster and has 3 components:
    • API server
    • Postgres Database
    • Frontend

WASM FDW Wrappers

Those 3 parts are quite similar as Docker Host, Container and Docker Hub in concept. Host FDW loads and runs guest FDW when user runs query against a foreign table, and the guest FDWs are published and distributed on FDW registry.

The guest FDW doesn't have to be published on registry, it can be uploaded to anywhere on the internet, as long as the host FDW can access it. The guest FDW can also be a local .wasm file, which is quite useful when developing a new FDW.

Pros:

  • Each guest FDW is a WASM file, size is small and can easily download and cached in local
  • Guest FDW can be individually loaded and run on demand
  • User can develop and use their own guest FDW without go through Supabase
  • Guest FDW runs in restricted WASM env, only can access outside through HTTP, no system call, no file or socket IO and etc.
  • User can post their own FDW to Registry (with Supabase authorization), which can be shared with other users

Cons:

  • Only support HTTP for guest FDW, so users can only develop API-based FDW
  • WASM env is restricted, many existing Rust libs cannot be compiled to wasm32-unknown-unknown target, especially those ones need to access system calls or IOs
  • Developing WASM using Rust may not very familiar for developers

Additional context

  • This PR is mainly to add initial prototype of WASM FDW, test cases and documents for the new FDW are not ready yet.
  • The FDW Registry code hasn't been included in this PR, I haven't decided to save it this repo or create a new repo, any ideas?

The Registry front page:
image

@burmecia burmecia marked this pull request as ready for review June 14, 2024 06:19
@burmecia burmecia added enhancement New feature or request wasm labels Jun 14, 2024
@burmecia burmecia requested a review from imor June 18, 2024 23:39
@burmecia burmecia merged commit 55f8814 into main Jun 27, 2024
2 checks passed
@burmecia burmecia deleted the bo/feat/wasm-fdw branch June 27, 2024 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request wasm
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants