[EPIC] Ibis-project as the data source connector

[goldmedal]

Purpose

Previously, Wren Engine implemented the data source connector by ourselves. To support a new data source, we had to spend a lot of time solving various issues, such as type mapping, syntax support, or data serialization/deserialization. Ibis is a Python DataFrame library that can connect to and convert SQL for many data sources.

We plan to change Wren Engine to a Python project. The core features will be implemented in Rust, and the data flow will be controlled by a Python server.

Architecture

The architecture of the new Wren Engine is illustrated below, and it also demonstrates how to integrate with WrenAI.

In the meantime, I'm working on redesigning the model module in Rust and DataFusion. PR #570 is the first step in this effort. I will file another epic issue for this work later.

Because the work of redesigning the modeling module will take some time, the first version of the Ibis connector will transform WrenSQL using the original SQL plan API (referred to as dry-plan in MDLResourceV2).

Work Lists

There're some subtasks in this epic. We should file other tickets for them.

• Data Query API: Query the data source the result. @grieve54706 is working on it now.
  • [REBASE] Implement ibis server #574
  • Chore: integrate with ibis server WrenAI#306
  • Add column_dtypes to forcibly convert the type #594
• Dry Run API: Check if the query is valid and got the schema of the result set. Same as the original dry-run api in MDLResource.
  • Add dryRun parameter for query API #593
• Analysis API: analyze a SQL to get some important information for AI service. Invoke the original analysis API in AnalysisResource
• MDL Validation API: Provide a way to check if a MDL is valid. Same as the validation API in MDLResource.
  • Add validation API #605
• CI workflow and code quality check
  • Add test case for ibis #583

[goldmedal]

This epic is totally finished.