Rebase master #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

chenxinwei merged 68 commits into master from rebase_master

Feb 5, 2025

LICENSE

-Original file line number
+Diff line change
@@ Expand Up / @@ -273,6 +273,20 @@ from https://parquet.apache.org/ version 1.14.0 @@
     paimon-common/src/main/java/org/apache/paimon/data/variant/GenericVariant.java
     paimon-common/src/main/java/org/apache/paimon/data/variant/GenericVariantBuilder.java
     paimon-common/src/main/java/org/apache/paimon/data/variant/GenericVariantUtil.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/ParquetColumnVector.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/ParquetReadState.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/ParquetVectorUpdater.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/ParquetVectorUpdaterFactory.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/RowIndexGenerator.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/VectorizedColumnReader.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/VectorizedDeltaBinaryPackedReader.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/VectorizedDeltaByteArrayReader.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/VectorizedDeltaLengthByteArrayReader.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/VectorizedParquetRecordReader.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/VectorizedPlainValuesReader.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/VectorizedReaderBase.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/VectorizedRleValuesReader.java
+    paimon-format/src/main/java/org/apache/paimon/format/parquet/newReader/VectorizedValuesReader.java
     from https://spark.apache.org/ version 4.0.0-preview2
     MIT License
@@ Expand Down @@

README.md

-Original file line number
+Diff line change
@@ Expand Up @@
     ## Building
-    JDK 8/11 is required for building the project. Maven version >=3.3.1.
+    JDK 8/11 is required for building the project. Maven version >=3.6.3.
     - Run the `mvn clean install -DskipTests` command to build the project.
     - Run the `mvn spotless:apply` to format the project (both Java and Scala).
@@ Expand Down @@

docs/content/append-table/streaming.md

-Original file line number
+Diff line change
@@ Expand Up @@
     Flink, using it like a queue. The only difference is that its latency is in minutes. Its advantages are very low cost
     and the ability to push down filters and projection.
-    ## Automatic small file merging
+    ## Pre small files merging
+    Pre means that this compact occurs before committing files to the snapshot.
+    If Flink's checkpoint interval is short (for example, 30 seconds), each snapshot may produce lots of small changelog
+    files. Too many files may put a burden on the distributed storage cluster.
+    In order to compact small changelog files into large ones, you can set the table option `precommit-compact = true`.
+    Default value of this option is false, if true, it will add a compact coordinator and worker operator after the writer
+    operator, which copies changelog files into large ones.
+    ## Post small files merging
+    Post means that this compact occurs after committing files to the snapshot.
     In streaming writing job, without bucket definition, there is no compaction in writer, instead, will use
     `Compact Coordinator` to scan the small files and pass compaction task to `Compact Worker`. In streaming mode, if you
@@ Expand Down @@

docs/content/cdc-ingestion/kafka-cdc.md

-Original file line number
+Diff line change
@@ Expand Up @@
     ```bash
     <FLINK_HOME>/bin/flink run \
         /path/to/paimon-flink-action-{{< version >}}.jar \
-        kafka_sync_table
+        kafka_sync_table \
         --warehouse <warehouse-path> \
         --database <database-name> \
         --table <table-name> \
@@ Expand Down Expand Up @@
     ```bash
     <FLINK_HOME>/bin/flink run \
         /path/to/paimon-flink-action-{{< version >}}.jar \
-        kafka_sync_database
+        kafka_sync_database \
         --warehouse <warehouse-path> \
         --database <database-name> \
         [--table_mapping <table-name>=<paimon-table-name>] \
@@ Expand Down @@

docs/content/cdc-ingestion/mongo-cdc.md

-Original file line number
+Diff line change
@@ Expand Up @@
     ```bash
     <FLINK_HOME>/bin/flink run \
         /path/to/paimon-flink-action-{{< version >}}.jar \
-        mongodb_sync_table
+        mongodb_sync_table \
         --warehouse <warehouse-path> \
         --database <database-name> \
         --table <table-name> \
@@ Expand Down Expand Up @@
     ```bash
     <FLINK_HOME>/bin/flink run \
         /path/to/paimon-flink-action-{{< version >}}.jar \
-        mongodb_sync_database
+        mongodb_sync_database \
         --warehouse <warehouse-path> \
         --database <database-name> \
         [--table_prefix <paimon-table-prefix>] \
@@ Expand Down @@

docs/content/cdc-ingestion/postgres-cdc.md

-Original file line number
+Diff line change
@@ Expand Up @@
     ```bash
     <FLINK_HOME>/bin/flink run \
         /path/to/paimon-flink-action-{{< version >}}.jar \
-        postgres_sync_table
+        postgres_sync_table \
         --warehouse <warehouse_path> \
         --database <database_name> \
         --table <table_name> \
@@ Expand Down @@

docs/content/cdc-ingestion/pulsar-cdc.md

-Original file line number
+Diff line change
@@ Expand Up @@
     ```bash
     <FLINK_HOME>/bin/flink run \
         /path/to/paimon-flink-action-{{< version >}}.jar \
-        pulsar_sync_table
+        pulsar_sync_table \
         --warehouse <warehouse-path> \
         --database <database-name> \
         --table <table-name> \
@@ Expand Down Expand Up @@
     ```bash
     <FLINK_HOME>/bin/flink run \
         /path/to/paimon-flink-action-{{< version >}}.jar \
-        pulsar_sync_database
+        pulsar_sync_database \
         --warehouse <warehouse-path> \
         --database <database-name> \
         [--table_prefix <paimon-table-prefix>] \
@@ Expand Down @@

docs/content/concepts/spec/fileindex.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -69,15 +69,15 @@ File index file format. Put all column and offset in the header.
  
    ｜                BODY                 ｜

    ｜_____________________________________｜    _____________________

    *

    magic:                            8 bytes long, value is 1493475289347502L, BIT_ENDIAN

    version:                          4 bytes int, BIT_ENDIAN

    head length:                      4 bytes int, BIT_ENDIAN

    column number:                    4 bytes int, BIT_ENDIAN

    column x name:                    2 bytes short BIT_ENDIAN and Java modified-utf-8

    index number:                     4 bytes int (how many column items below), BIT_ENDIAN

    index name x:                     2 bytes short BIT_ENDIAN and Java modified-utf-8

    start pos:                        4 bytes int, BIT_ENDIAN

    length:                           4 bytes int, BIT_ENDIAN

    magic:                            8 bytes long, value is 1493475289347502L, BIG_ENDIAN

    version:                          4 bytes int, BIG_ENDIAN

    head length:                      4 bytes int, BIG_ENDIAN

    column number:                    4 bytes int, BIG_ENDIAN

    column x name:                    2 bytes short BIG_ENDIAN and Java modified-utf-8

    index number:                     4 bytes int (how many column items below), BIG_ENDIAN

    index name x:                     2 bytes short BIG_ENDIAN and Java modified-utf-8

    start pos:                        4 bytes int, BIG_ENDIAN

    length:                           4 bytes int, BIG_ENDIAN

    redundant length:                 4 bytes int (for compatibility with later versions, in this version, content is zero)

    redundant bytes:                  var bytes (for compatibility with later version, in this version, is empty)

    BODY:                             column index bytes + column index bytes + column index bytes + .......

    @@ -88,7 +88,7 @@ BODY:                             column index bytes + column index bytes + colu
  
    Define `'file-index.bloom-filter.columns'`.

    Content of bloom filter index is simple: 

    - numHashFunctions 4 bytes int, BIT_ENDIAN

    - numHashFunctions 4 bytes int, BIG_ENDIAN

    - bloom filter bytes

    This class use (64-bits) long hash. Store the num hash function (one integer) and bit set bytes only. Hash bytes type 

    @@ -135,7 +135,7 @@ offset:                        4 bytes int (when it is negative, it represents t
  
                                     and its position is the inverse of the negative value)

    </pre>

    Integer are all BIT_ENDIAN.

    Integer are all BIG_ENDIAN.

    ## Index: Bit-Slice Index Bitmap

docs/content/concepts/spec/tableindex.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -36,7 +36,7 @@ Its structure is very simple, only storing hash values in the file:
  
    HASH_VALUE | HASH_VALUE | HASH_VALUE | HASH_VALUE | ...

    HASH_VALUE is the hash value of the primary-key. 4 bytes, BIT_ENDIAN.

    HASH_VALUE is the hash value of the primary-key. 4 bytes, BIG_ENDIAN.

    ## Deletion Vectors

    @@ -49,9 +49,9 @@ The deletion file is a binary file, and the format is as follows:
  
    - First, record version by a byte. Current version is 1.

    - Then, record <size of serialized bin, serialized bin, checksum of serialized bin> in sequence.

    - Size and checksum are BIT_ENDIAN Integer.

    - Size and checksum are BIG_ENDIAN Integer.

    For each serialized bin:

    - First, record a const magic number by an int (BIT_ENDIAN). Current the magic number is 1581511376.

    - First, record a const magic number by an int (BIG_ENDIAN). Current the magic number is 1581511376.

    - Then, record serialized bitmap. Which is a [RoaringBitmap](https://github.com/RoaringBitmap/RoaringBitmap) (org.roaringbitmap.RoaringBitmap).

docs/content/concepts/system-tables.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -279,45 +279,45 @@ You can query all manifest files contained in the latest snapshot or the specifi
  
    SELECT * FROM my_table$manifests;

    /*

    +--------------------------------+-------------+------------------+-------------------+---------------+

    |                      file_name |   file_size |  num_added_files | num_deleted_files |     schema_id |

    +--------------------------------+-------------+------------------+-------------------+---------------+

    | manifest-f4dcab43-ef6b-4713... |        12365|               40 |                 0 |             0 |

    | manifest-f4dcab43-ef6b-4713... |        1648 |                1 |                 0 |             0 |

    +--------------------------------+-------------+------------------+-------------------+---------------+

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    |                      file_name |   file_size |  num_added_files | num_deleted_files |     schema_id | min_partition_stats | max_partition_stats |

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    | manifest-f4dcab43-ef6b-4713... |        12365|               40 |                 0 |             0 |      {20230315, 00} |      {20230315, 20} |

    | manifest-f4dcab43-ef6b-4713... |        1648 |                1 |                 0 |             0 |      {20230115, 00} |      {20230316, 23} |

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    2 rows in set

    */

    -- You can also query the manifest with specified snapshot

    SELECT * FROM my_table$manifests /*+ OPTIONS('scan.snapshot-id'='1') */;

    /*

    +--------------------------------+-------------+------------------+-------------------+---------------+

    |                      file_name |   file_size |  num_added_files | num_deleted_files |     schema_id |

    +--------------------------------+-------------+------------------+-------------------+---------------+

    | manifest-f4dcab43-ef6b-4713... |        12365|               40 |                 0 |             0 |

    +--------------------------------+-------------+------------------+-------------------+---------------+

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    |                      file_name |   file_size |  num_added_files | num_deleted_files |     schema_id | min_partition_stats | max_partition_stats |

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    | manifest-f4dcab43-ef6b-4713... |        12365|               40 |                 0 |             0 |      {20230315, 00} |      {20230315, 20} |

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    1 rows in set

    */

    - You can also query the manifest with specified tagName

    SELECT * FROM my_table$manifests /*+ OPTIONS('scan.tag-name'='tag1') */;

    /*

    +--------------------------------+-------------+------------------+-------------------+---------------+

    |                      file_name |   file_size |  num_added_files | num_deleted_files |     schema_id |

    +--------------------------------+-------------+------------------+-------------------+---------------+

    | manifest-f4dcab43-ef6b-4713... |        12365|               40 |                 0 |             0 |

    +--------------------------------+-------------+------------------+-------------------+---------------+

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    |                      file_name |   file_size |  num_added_files | num_deleted_files |     schema_id | min_partition_stats | max_partition_stats |

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    | manifest-f4dcab43-ef6b-4713... |        12365|               40 |                 0 |             0 |      {20230315, 00} |      {20230315, 20} |

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    1 rows in set

    */

    - You can also query the manifest with specified timestamp in unix milliseconds

    SELECT * FROM my_table$manifests /*+ OPTIONS('scan.timestamp-millis'='1678883047356') */;

    /*

    +--------------------------------+-------------+------------------+-------------------+---------------+

    |                      file_name |   file_size |  num_added_files | num_deleted_files |     schema_id |

    +--------------------------------+-------------+------------------+-------------------+---------------+

    | manifest-f4dcab43-ef6b-4713... |        12365|               40 |                 0 |             0 |

    +--------------------------------+-------------+------------------+-------------------+---------------+

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    |                      file_name |   file_size |  num_added_files | num_deleted_files |     schema_id | min_partition_stats | max_partition_stats |

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    | manifest-f4dcab43-ef6b-4713... |        12365|               40 |                 0 |             0 |      {20230315, 00} |      {20230315, 20} |

    +--------------------------------+-------------+------------------+-------------------+---------------+---------------------+---------------------+

    1 rows in set

    */

    ```

docs/content/engines/doris.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -28,7 +28,7 @@ under the License.
  
    This documentation is a guide for using Paimon in Doris.

    > More details can be found in [Apache Doris Website](https://doris.apache.org/docs/lakehouse/datalake-analytics/paimon/)

    > More details can be found in [Apache Doris Website](https://doris.apache.org/docs/dev/lakehouse/catalogs/paimon-catalog)

    ## Version

    @@ -65,9 +65,21 @@ CREATE CATALOG `paimon_hms` PROPERTIES (
  
        "hive.metastore.uris" = "thrift://172.21.0.44:7004",

        "hadoop.username" = "hadoop"

    );

    -- Integrate with Aliyun DLF

    CREATE CATALOG paimon_dlf PROPERTIES (

        'type' = 'paimon',

        'paimon.catalog.type' = 'dlf',

        'warehouse' = 'oss://paimon-bucket/paimonoss/',

        'dlf.proxy.mode' = 'DLF_ONLY',

        'dlf.uid' = 'xxxxx',

        'dlf.region' = 'cn-beijing',

        'dlf.access_key' = 'ak',

        'dlf.secret_key' = 'sk'

    );

    ```

    See [Apache Doris Website](https://doris.apache.org/docs/lakehouse/datalake-analytics/paimon/) for more examples.

    See [Apache Doris Website](https://doris.apache.org/docs/dev/lakehouse/catalogs/paimon-catalog) for more examples.

    ## Access Paimon Catalog

docs/content/engines/overview.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -33,7 +33,7 @@ under the License. @@
     |                                      Flink                                      |  1.15 - 1.20  |     ✅      |      ✅      |      ✅       |  ✅(1.17+)   |        ✅        |       ✅        |        ✅        |    ✅(1.17+)     |     ❌      |      ✅      |
     |                                      Spark                                      |   3.2 - 3.5   |     ✅      |      ✅      |      ✅       |      ✅      |      ✅(3.3+)    |    ✅(3.3+)     |        ✅        |        ✅        |     ✅      |   ✅(3.3+)   |
     |                                      Hive                                       |   2.1 - 3.1   |     ✅      |      ✅      |      ✅       |      ❌      |        ❌        |       ❌        |        ❌        |        ❌        |     ❌      |      ✅      |
-    |                                      Trino                                      |   420 - 439   |     ✅      |   ✅(427+)   |   ✅(427+)    |   ✅(427+)   |        ❌        |       ❌        |        ❌        |        ❌        |     ❌      |      ✅      |
+    |                                      Trino                                      |   420 - 440   |     ✅      |   ✅(427+)   |   ✅(427+)    |   ✅(427+)   |        ❌        |       ❌        |        ❌        |        ❌        |     ❌      |      ✅      |
     |                                     Presto                                      | 0.236 - 0.280 |     ✅      |      ❌      |      ✅       |      ✅      |        ❌        |       ❌        |        ❌        |        ❌        |     ❌      |      ❌      |
     | [StarRocks](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/) |     3.1+      |     ✅      |      ❌      |      ❌       |      ❌      |        ❌        |       ❌        |        ❌        |        ❌        |     ❌      |      ✅      |
     | [Doris](https://doris.apache.org/docs/lakehouse/datalake-analytics/paimon)      |    2.0.6+     |     ✅      |      ❌      |      ❌       |      ❌      |        ❌        |       ❌        |        ❌        |        ❌        |     ❌      |      ✅      |
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebase master #6

Uh oh!

Diff view

Diff view

Uh oh!

There are no files selected for viewing

Uh oh!

Uh oh!