Skip to content

schema_force_view_type configuration not working for CREATE EXTERNAL TABLE #14909

@2010YOUY01

Description

@2010YOUY01

Describe the bug

schema_force_view_type configuration said:

datafusion.execution.parquet.schema_force_view_types | true | (reading) If true, parquet reader will read columns of Utf8/Utf8Large with Utf8View, and Binary/BinaryLarge with BinaryView.

Reference: https://datafusion.apache.org/user-guide/configs.html

I tried the following script within datafusion-cli

set datafusion.execution.parquet.schema_force_view_types = true;

CREATE EXTERNAL TABLE IF NOT EXISTS lineitem (
        l_orderkey BIGINT,
        l_partkey BIGINT,
        l_suppkey BIGINT,
        l_linenumber INTEGER,
        l_quantity DECIMAL(15, 2),
        l_extendedprice DECIMAL(15, 2),
        l_discount DECIMAL(15, 2),
        l_tax DECIMAL(15, 2),
        l_returnflag VARCHAR,
        l_linestatus VARCHAR,
        l_shipdate DATE,
        l_commitdate DATE,
        l_receiptdate DATE,
        l_shipinstruct VARCHAR,
        l_shipmode VARCHAR,
        l_comment VARCHAR
) STORED AS parquet
LOCATION '/Users/yongting/Code/datafusion/benchmarks/data/tpch_sf10/lineitem';

select arrow_typeof(l_comment) from lineitem limit 1;

The data type is Utf8 instead Utf8View

+----------------------------------+
| arrow_typeof(lineitem.l_comment) |
+----------------------------------+
| Utf8                             |
+----------------------------------+

To Reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions