feat: add wasm fdw framework with snowflake and paddle fdw #280
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
This PR is to add WASM foreign data wrapper prototype with 3 new FDWs developed using this new framework.
Snowflake FDW
, supports data read and modifyPaddle FDW
, supports data read onlyHelloWorld FDW
, a simple hello world FDW for demo purposeAll those 3 FDWs are WASM FDWs.
What is the current behavior?
In current framework, all FDWs needs to be compile into a single Postgres extension and then released to Supabase Postgres image and then deployed to Supabase platform. This architecture has many drawbacks:
What is the new behavior?
The new architecture is based on WebAssembly (WASM), each FDW is complied into a WASM component package (Guest) and it is dynamically loaded and run by a new WASM FDW (Host). This consists of 3 parts:
Host
: a new FDW using current Wrappers framework and embedding a WASM runtime, which run one or many guest FDWsGuest
: the actual FDW which does the actual data loading and transformation, compiled into a WASM component (.wasm file)Registry
: a WASM FDW registry based on Warg protocol, which serves as a FDW publish and distribution platform. It runs as a Docker compose cluster and has 3 components:Those 3 parts are quite similar as Docker Host, Container and Docker Hub in concept. Host FDW loads and runs guest FDW when user runs query against a foreign table, and the guest FDWs are published and distributed on FDW registry.
The guest FDW doesn't have to be published on registry, it can be uploaded to anywhere on the internet, as long as the host FDW can access it. The guest FDW can also be a local
.wasm
file, which is quite useful when developing a new FDW.Pros:
Cons:
wasm32-unknown-unknown
target, especially those ones need to access system calls or IOsAdditional context
The Registry front page: