Fix: Correct the profile of cache layer schema for Canner Persistence Store

[fredalai]

Description

For the Canner Enterprise product, the extension-store-canner pulls the result.json from Minio to merge into one file, it lacks the profile setting of the cache layer in the schema, so we need to fix it and let caching datasets functionality works.

Issue ticket number

closes #275

Screenshot for integrating with Canner Enterprise.

Step 1: Check the connection and deploy Data API to the Canner Enterprise product

Step 1-1

Opened a Canner Enterprise of PR testing env and create the w1 workspaces with the table:

Step 1-2.1

Use the In the VulcanSQL labs and create canner-w1 profiles with temporal PAT for connecting Canner Enterprise by extension-driver-canner through PG Wire Protocol:

Step 1-2.2

Enable caching for vulcan.yaml

cache:
  type: parquet
  folderPath: tmp
  loader: duckdb

Step 1-2.3

Apply my bugfix code for package.json of labs

{
  "name": "...",
  "dependencies": {
   ...
   "@vulcan-sql/extension-store-canner": "../../dist/packages/extension-store-canner",
  
  },
  ...
}

Step 1-3

Prepare Data APIs

Step 1-4

Add pkg-extension-store-canner in the Makefile and execute it
(Get the result.json)

Step 1-5

Upload them to Canner Enterprise, and see the uploaded artifacts in the vulcansql folder of w1:

Step 2: Check CannerPersistenceStore

Now we delete the SQLs and API schemas files, in the lasbs project, and update the vulcan.yaml for artifact options, extensions and profiles like below to use the Canner extension:

Step 2-1

....
artifact:
  provider: Canner # use Canner Persistence Store
  serializer: JSON
  # Path to build result
  filePath: fe54258b-da25-408b-a019-efb3b36f1b47 # root path

extensions:
  duckdb: "@vulcan-sql/extension-driver-duckdb"
  # Needed extensions for integrating with Canner Enterprise
  canner-driver: "@vulcan-sql/extension-driver-canner"
  canner-store: "@vulcan-sql/extension-store-canner"

profiles:
  # Specify the Canner profile reader to create profiles with different databases indicating to the workspaces of Canner Enterprise
  - type: Canner # use Canner Profile Reader
    options:
      path: fe54258b-da25-408b-a019-efb3b36f1b47 # root path

Step 2-2

Then set some environment variables we need:

export MINIO_ACCESS_KEY=<ACCESS-KEY>
export MINIO_BUCKET=<BUCKET>
export MINIO_PORT=9000
export MINIO_SECRET_KEY=<SECRETE-KEY>
export MINIO_SSL=false
export MINIO_URL=<IP>

export PROFILE_CANNER_DRIVER_HOST=<IP>
export PROFILE_CANNER_DRIVER_PASSWORD=<PAT>
export PROFILE_CANNER_DRIVER_USER=<USER>
export STORAGE_PROVIDER=<PROVIDER> # E.g., MINIO

Step 2-3.1

Finally, run vulcan serve to fetch the artifacts from Canner Enterprise, and generate Canner Enterprise Profiles settings:

Step 2-3.2

Open the API documentation URL, we will see the workspace SQL Name in the prefix for the w1 Data API:

Step 2-3.3

Send request to /api/w1/activity_logs?operation=LOGIN to query and you will see the result:

Done.

Additional Context

I will revert #273 in this PR as it was causing bugs with the Canner integration. It seems that this issue needs to be discussed in an Issue. cc @cyyeh

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
vulcan-sql-document	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 14, 2023 5:27am

[codecov-commenter]

Codecov Report

Patch coverage: 80.00% and no project coverage change.

Comparison is base (b3d73ed) 90.20% compared to head (235ccb9) 90.21%.
Report is 12 commits behind head on develop.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop     #276   +/-   ##
========================================
  Coverage    90.20%   90.21%           
========================================
  Files          350      350           
  Lines         5965     5967    +2     
  Branches       803      803           
========================================
+ Hits          5381     5383    +2     
+ Misses         424      423    -1     
- Partials       160      161    +1     

Flag	Coverage Δ
build	89.57% <ø> (-0.02%)	⬇️
catalog-server	100.00% <ø> (ø)
extension-authenticator-canner	78.37% <ø> (ø)
extension-store-canner	97.54% <75.00%> (-0.78%)	⬇️
integration-testing	90.27% <ø> (ø)
serve	87.17% <100.00%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...ckages/build/src/lib/schema-parser/schemaParser.ts	96.00% <ø> (-0.16%)	⬇️
...ges/serve/src/lib/catalog-router/catalogRouters.ts	26.19% <ø> (ø)
...on-store-canner/src/lib/canner/persistenceStore.ts	96.49% <75.00%> (-1.66%)	⬇️
...erve/src/lib/route/route-component/restfulRoute.ts	93.75% <100.00%> (+6.25%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kokokuo]

LGTM 👍