Re-organize gandiva related docs, and add an overall diagram for exte…

…rnal function integration.
apache · Nov 29, 2023 · 90968bd · 90968bd
1 parent b6ffbf3
commit 90968bd
Show file tree

Hide file tree

Showing 4 changed files with 179 additions and 124 deletions.
diff --git a/docs/source/cpp/gandiva.rst b/docs/source/cpp/gandiva.rst
@@ -40,122 +40,13 @@ pre-compiled into LLVM IR (intermediate representation).
 .. _LLVM: https://llvm.org/
 
 
-Building Expressions
-====================
+Expression, Projector and Filter
+================================
+To effectively utilize Gandiva, you will construct expression trees with ``TreeExprBuilder``, 
+including the creation of function nodes, if-else logic, and boolean expressions. 
+Subsequently, leverage ``Projector`` or ``Filter`` execution kernels to efficiently evaluate these expressions.
+See :doc:`./gandiva/expr_projector_filter` for more details. 
 
-Gandiva provides a general expression representation where expressions are
-represented by a tree of nodes. The expression trees are built using
-:class:`TreeExprBuilder`. The leaves of the expression tree are typically
-field references, created by :func:`TreeExprBuilder::MakeField`, and
-literal values, created by :func:`TreeExprBuilder::MakeLiteral`. Nodes
-can be combined into more complex expression trees using:
-
-* :func:`TreeExprBuilder::MakeFunction` to create a function
-  node. (You can call :func:`GetRegisteredFunctionSignatures` to 
-  get a list of valid function signatures.)
-* :func:`TreeExprBuilder::MakeIf` to create if-else logic.
-* :func:`TreeExprBuilder::MakeAnd` and :func:`TreeExprBuilder::MakeOr`
-  to create boolean expressions. (For "not", use the ``not(bool)`` function in ``MakeFunction``.)
-* :func:`TreeExprBuilder::MakeInExpressionInt32` and the other "in expression"
-  functions to create set membership tests.
-
-Each of these functions create new composite nodes, which contain the leaf nodes
-(literals and field references) or other composite nodes as children. By 
-composing these, you can create arbitrarily complex expression trees.
-
-Once an expression tree is built, they are wrapped in either :class:`Expression`
-or :class:`Condition`, depending on how they will be used.
-``Expression`` is used in projections while ``Condition`` is used in filters.
-
-As an example, here is how to create an Expression representing ``x + 3`` and a
-Condition representing ``x < 3``:
-
-.. literalinclude:: ../../../cpp/examples/arrow/gandiva_example.cc
-   :language: cpp
-   :start-after: (Doc section: Create expressions)
-   :end-before: (Doc section: Create expressions)
-   :dedent: 2
-
-
-Projectors and Filters
-======================
-
-Gandiva's two execution kernels are :class:`Projector` and
-:class:`Filter`. ``Projector`` consumes a record batch and projects
-into a new record batch. ``Filter`` consumes a record batch and produces a
-:class:`SelectionVector` containing the indices that matched the condition.
-
-For both ``Projector`` and ``Filter``, optimization of the expression IR happens
-when creating instances. They are compiled against a static schema, so the
-schema of the record batches must be known at this point.
-
-Continuing with the ``expression`` and ``condition`` created in the previous
-section, here is an example of creating a Projector and a Filter:
-
-.. literalinclude:: ../../../cpp/examples/arrow/gandiva_example.cc
-   :language: cpp
-   :start-after: (Doc section: Create projector and filter)
-   :end-before: (Doc section: Create projector and filter)
-   :dedent: 2
-
-Once a Projector or Filter is created, it can be evaluated on Arrow record batches.
-These execution kernels are single-threaded on their own, but are designed to be
-reused to process distinct record batches in parallel.
-
-Evaluating projections
-----------------------
-
-Execution is performed with :func:`Projector::Evaluate`. This outputs 
-a vector of arrays, which can be passed along with the output schema to
-:func:`arrow::RecordBatch::Make()`.
-
-.. literalinclude:: ../../../cpp/examples/arrow/gandiva_example.cc
-   :language: cpp
-   :start-after: (Doc section: Evaluate projection)
-   :end-before: (Doc section: Evaluate projection)
-   :dedent: 2
-
-Evaluating filters
-------------------
-
-:func:`Filter::Evaluate` produces :class:`SelectionVector`,
-a vector of row indices that matched the filter condition. The selection vector
-is a wrapper around an arrow integer array, parameterized by bitwidth. When 
-creating the selection vector (you must initialize it *before* passing to 
-``Evaluate()``), you must choose the bitwidth, which determines the max index 
-value it can hold, and the max number of slots, which determines how many indices
-it may contain. In general, the max number of slots should be set to your batch 
-size and the bitwidth the smallest integer size that can represent all integers 
-less than the batch size. For example, if your batch size is 100k, set the 
-maximum number of slots to 100k and the bitwidth to 32 (since 2^16 = 64k which 
-would be too small).
-
-Once ``Evaluate()`` has been run and the :class:`SelectionVector` is
-populated, use the :func:`SelectionVector::ToArray()` method to get
-the underlying array and then :func:`::arrow::compute::Take()` to materialize the
-output record batch.
-
-.. literalinclude:: ../../../cpp/examples/arrow/gandiva_example.cc
-   :language: cpp
-   :start-after: (Doc section: Evaluate filter)
-   :end-before: (Doc section: Evaluate filter)
-   :dedent: 2
-
-Evaluating projections and filters
-----------------------------------
-
-Finally, you can also project while apply a selection vector, with 
-:func:`Projector::Evaluate()`. To do so, first make sure to initialize the
-:class:`Projector` with :func:`SelectionVector::GetMode()` so that the projector
-compiles with the correct bitwidth. Then you can pass the 
-:class:`SelectionVector` into the :func:`Projector::Evaluate()` method.
-
-
-.. literalinclude:: ../../../cpp/examples/arrow/gandiva_example.cc
-   :language: cpp
-   :start-after: (Doc section: Evaluate filter and projection)
-   :end-before: (Doc section: Evaluate filter and projection)
-   :dedent: 2
 
 External Functions Development
 ==============================
@@ -166,4 +57,10 @@ looking to customize and enhance their computational solutions,
 Gandiva provides the opportunity to develop and register their own external 
 functions, thus allowing for a more tailored and flexible use of the Gandiva 
 environment.
-See :doc:`./gandiva/external_func` for more details. 
+See :doc:`./gandiva/external_func` for more details. 
+
+.. toctree::
+   :maxdepth: 2
+
+   gandiva/expr_projector_filter
+   gandiva/external_func
diff --git a/docs/source/cpp/gandiva/expr_projector_filter.rst b/docs/source/cpp/gandiva/expr_projector_filter.rst
@@ -0,0 +1,137 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+=========================================
+Gandiva Expression, Projector, and Filter
+=========================================
+
+Building Expressions
+====================
+
+Gandiva provides a general expression representation where expressions are
+represented by a tree of nodes. The expression trees are built using
+:class:`TreeExprBuilder`. The leaves of the expression tree are typically
+field references, created by :func:`TreeExprBuilder::MakeField`, and
+literal values, created by :func:`TreeExprBuilder::MakeLiteral`. Nodes
+can be combined into more complex expression trees using:
+
+* :func:`TreeExprBuilder::MakeFunction` to create a function
+  node. (You can call :func:`GetRegisteredFunctionSignatures` to 
+  get a list of valid function signatures.)
+* :func:`TreeExprBuilder::MakeIf` to create if-else logic.
+* :func:`TreeExprBuilder::MakeAnd` and :func:`TreeExprBuilder::MakeOr`
+  to create boolean expressions. (For "not", use the ``not(bool)`` function in ``MakeFunction``.)
+* :func:`TreeExprBuilder::MakeInExpressionInt32` and the other "in expression"
+  functions to create set membership tests.
+
+Each of these functions create new composite nodes, which contain the leaf nodes
+(literals and field references) or other composite nodes as children. By 
+composing these, you can create arbitrarily complex expression trees.
+
+Once an expression tree is built, they are wrapped in either :class:`Expression`
+or :class:`Condition`, depending on how they will be used.
+``Expression`` is used in projections while ``Condition`` is used in filters.
+
+As an example, here is how to create an Expression representing ``x + 3`` and a
+Condition representing ``x < 3``:
+
+.. literalinclude:: ../../../../cpp/examples/arrow/gandiva_example.cc
+   :language: cpp
+   :start-after: (Doc section: Create expressions)
+   :end-before: (Doc section: Create expressions)
+   :dedent: 2
+
+
+Projectors and Filters
+======================
+
+Gandiva's two execution kernels are :class:`Projector` and
+:class:`Filter`. ``Projector`` consumes a record batch and projects
+into a new record batch. ``Filter`` consumes a record batch and produces a
+:class:`SelectionVector` containing the indices that matched the condition.
+
+For both ``Projector`` and ``Filter``, optimization of the expression IR happens
+when creating instances. They are compiled against a static schema, so the
+schema of the record batches must be known at this point.
+
+Continuing with the ``expression`` and ``condition`` created in the previous
+section, here is an example of creating a Projector and a Filter:
+
+.. literalinclude:: ../../../../cpp/examples/arrow/gandiva_example.cc
+   :language: cpp
+   :start-after: (Doc section: Create projector and filter)
+   :end-before: (Doc section: Create projector and filter)
+   :dedent: 2
+
+Once a Projector or Filter is created, it can be evaluated on Arrow record batches.
+These execution kernels are single-threaded on their own, but are designed to be
+reused to process distinct record batches in parallel.
+
+Evaluating projections
+----------------------
+
+Execution is performed with :func:`Projector::Evaluate`. This outputs 
+a vector of arrays, which can be passed along with the output schema to
+:func:`arrow::RecordBatch::Make()`.
+
+.. literalinclude:: ../../../../cpp/examples/arrow/gandiva_example.cc
+   :language: cpp
+   :start-after: (Doc section: Evaluate projection)
+   :end-before: (Doc section: Evaluate projection)
+   :dedent: 2
+
+Evaluating filters
+------------------
+
+:func:`Filter::Evaluate` produces :class:`SelectionVector`,
+a vector of row indices that matched the filter condition. The selection vector
+is a wrapper around an arrow integer array, parameterized by bitwidth. When 
+creating the selection vector (you must initialize it *before* passing to 
+``Evaluate()``), you must choose the bitwidth, which determines the max index 
+value it can hold, and the max number of slots, which determines how many indices
+it may contain. In general, the max number of slots should be set to your batch 
+size and the bitwidth the smallest integer size that can represent all integers 
+less than the batch size. For example, if your batch size is 100k, set the 
+maximum number of slots to 100k and the bitwidth to 32 (since 2^16 = 64k which 
+would be too small).
+
+Once ``Evaluate()`` has been run and the :class:`SelectionVector` is
+populated, use the :func:`SelectionVector::ToArray()` method to get
+the underlying array and then :func:`::arrow::compute::Take()` to materialize the
+output record batch.
+
+.. literalinclude:: ../../../../cpp/examples/arrow/gandiva_example.cc
+   :language: cpp
+   :start-after: (Doc section: Evaluate filter)
+   :end-before: (Doc section: Evaluate filter)
+   :dedent: 2
+
+Evaluating projections and filters
+----------------------------------
+
+Finally, you can also project while apply a selection vector, with 
+:func:`Projector::Evaluate()`. To do so, first make sure to initialize the
+:class:`Projector` with :func:`SelectionVector::GetMode()` so that the projector
+compiles with the correct bitwidth. Then you can pass the 
+:class:`SelectionVector` into the :func:`Projector::Evaluate()` method.
+
+
+.. literalinclude:: ../../../../cpp/examples/arrow/gandiva_example.cc
+   :language: cpp
+   :start-after: (Doc section: Evaluate filter and projection)
+   :end-before: (Doc section: Evaluate filter and projection)
+   :dedent: 2
diff --git a/docs/source/cpp/gandiva/external_func.png b/docs/source/cpp/gandiva/external_func.png
diff --git a/docs/source/cpp/gandiva/external_func.rst b/docs/source/cpp/gandiva/external_func.rst
@@ -29,9 +29,9 @@ Overview of External Function Types in Gandiva
 
 Gandiva supports two primary types of external functions:
 
-* C Functions: Functions conforming to the C calling convention. Developers can implement functions in various languages (like C++, Rust, C, or Zig) and expose them as C functions for Gandiva.
+* C Functions: Functions conforming to the C calling convention. Developers can implement functions in various languages (like C++, Rust, C, or Zig) and expose them as C functions to Gandiva.
 
-* IR Functions: Functions implemented in LLVM's Intermediate Representation (IR). These can be written in multiple languages and then compiled into LLVM IR to be registered in Gandiva.
+* IR Functions: Functions implemented in LLVM Intermediate Representation (LLVM IR). These can be written in multiple languages and then compiled into LLVM IR to be registered in Gandiva.
 
 Choosing the Right Type of External Function for Your Needs
 ---------------------------------------------------------------
@@ -43,17 +43,20 @@ When integrating external functions into Gandiva, it's crucial to select the typ
     * **Broad Applicability:** They are generally a go-to choice for a wide range of use cases due to their compatibility and ease of integration.
 
 * IR Functions
+    * **Recommended Use Cases:** IR functions excel in handling straightforward tasks that do not require elaborate logic or dependence on sophisticated third-party libraries. Unlike C functions, IR functions have the advantage of being inlinable, which is particularly beneficial for simple operations where the invocation overhead constitutes a significant expense. Additionally, they are an ideal choice for projects that are already integrated with the LLVM toolchain.
     * **IR Compilation Requirement:** For IR functions, the entire implementation, including any third-party libraries used, must be compiled into LLVM IR. This might affect performance, especially if the dependent libraries are complex.
     * **Limitations in Capabilities:** Certain advanced features, such as using thread-local variables, are not supported in IR functions. This is due to the limitations of the current JIT (Just-In-Time) engine utilized internally by Gandiva.
-    * **Recommended Use Cases:** IR functions are best suited for simpler tasks that don't demand intricate logic or reliance on complex third-party libraries. They are also a good fit if your project already incorporates the LLVM toolchain.
+
+.. image:: ./external_func.png
+   :alt: External C functions and IR functions integrating with Gandiva
 
 External function registration
 =================================
 
 To make a function available to Gandiva, you need to register it as an external function, providing both a function's metadata and its implementation to Gandiva.
 
-Using the NativeFunction Class
-----------------------------------
+Metadata Registration Using the ``NativeFunction`` Class
+--------------------------------------------------------
 
 To register a function in Gandiva, use the ``gandiva::NativeFunction`` class. This class captures both the signature and metadata of the external function.
 
@@ -80,6 +83,8 @@ The ``NativeFunction`` class is used to define the metadata for an external func
   * Typically, this name follows the convention ``{base_name}`` + ``_{param1_type}`` + ``{param2_type}`` + ... + ``{paramN_type}``. For example, if the base name is ``add`` and the function takes two ``int32`` parameters and returns an ``int32``, the precompiled function name would be ``add_int32_int32``, but this convention is not mandatory as long as you can guarantee its uniqueness.
 * ``flags``: Optional flags for additional function attributes (default is 0). Please check out ``NativeFunction::kNeedsContext``, ``NativeFunction::kNeedsFunctionHolder``, and ``NativeFunction::kCanReturnErrors`` for more details.
 
+After the function is registered, its implementation needs to be provided via either a C function pointer or a LLVM IR function.
+
 External C functions
 ------------------------
 
@@ -91,7 +96,7 @@ C Function Signature
 Signature Mapping
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The following table lists the mapping between Gandiva external function signature types and the C function signature types:
+Not all Arrow data types are supported in Gandiva. The following table lists the mapping between Gandiva external function signature types and the C function signature types:
 
 +-------------------------------------+-------------------+
 | Gandiva type (arrow data type)      | C function type   |
@@ -126,6 +131,12 @@ The following table lists the mapping between Gandiva external function signatur
 +-------------------------------------+-------------------+
 | time32                              | int32_t           |
 +-------------------------------------+-------------------+
+| time64                              | int64_t           |
++-------------------------------------+-------------------+
+| interval_month                      | int32_t           |
++-------------------------------------+-------------------+
+| interval_day_time                   | int64_t           |
++-------------------------------------+-------------------+
 | utf8 (as parameter type)            | const char*,      |
 |                                     | uint32_t          |
 |                                     | [see next section]|
@@ -135,9 +146,19 @@ The following table lists the mapping between Gandiva external function signatur
 |                                     | uint32_t*         |
 |                                     | [see next section]|
 +-------------------------------------+-------------------+
+| binary (as parameter type)          | const char*,      |
+|                                     | uint32_t          |
+|                                     | [see next section]|
++-------------------------------------+-------------------+
+| utf8 (as return type)               | int64_t context,  |
+|                                     | const char*,      |
+|                                     | uint32_t*         |
+|                                     | [see next section]| 
++-------------------------------------+-------------------+
 
-Handling arrow::StringType (utf8 type)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Handling arrow::StringType (utf8 type) and arrow::BinaryType 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Both ``arrow::StringType`` and ``arrow::BinaryType`` are variable-length types. And they are handled similarly in external functions. Since ``arrow::StringType`` (utf8 type) is more commonly used, we will use it below as the example to explain how to handle variable-length types in external functions.
 
 Using ``arrow::StringType`` (also known as the ``utf8`` type) as function parameter or return value needs special handling in external functions. This section provides details on how to handle ``arrow::StringType``.