Skip to content

Provide option to disable auto crop feature in DumpCommand output #1983

@asfimport

Description

@asfimport

Problem

When using the dump command in parquet-tools, the output will sometimes be truncated based on the width of your console, especially on smaller displays.

Example:

row group 0
--------------------------------------------------------------------------------
id:          INT32 SNAPPY DO:0 FPO:4 SZ:44668/920538/20.61 VC:7240100  [more]...
name:        BINARY SNAPPY DO:0 FPO:44672 SZ:89464018/1031768430/11.53 [more]...
event_time:  INT64 SNAPPY DO:0 FPO:89508690 SZ:43600235/57923935/1.33 VC:7240100 [more]...

    id TV=7240100 RL=0 DL=0 DS: 2 DE:PLAIN_DICTIONARY
    ----------------------------------------------------------------------------
    page 0:                      DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLA [more]... SZ:33291

This is especially annoying if you pipe the output to a file as the truncation remains in place.

Proposed fix

Provide the flag --disable-crop for the dump command. Truncation is enabled by default and will only be disabled when this flag is provided,

This will output the full content to standard out, for example:

row group 0
--------------------------------------------------------------------------------
id:          INT32 SNAPPY DO:0 FPO:4 SZ:44668/920538/20.61 VC:7240100 ENC:BIT_PACKED,PLAIN_DICTIONARY
name:        BINARY SNAPPY DO:0 FPO:44672 SZ:89464018/1031768430/11.53 VC:7240100 ENC:PLAIN,BIT_PACKED
event_time:  INT64 SNAPPY DO:0 FPO:89508690 SZ:43600235/57923935/1.33 VC:7240100 ENC:PLAIN,BIT_PACKED,RLE

    id TV=7240100 RL=0 DL=0 DS: 2 DE:PLAIN_DICTIONARY
    ----------------------------------------------------------------------------
    page 0:                      DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN_DICTIONARY ST:[min: 0, max: 1, num_nulls: 0] SZ:33291 VC:262146
    page 1:                      DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN_DICTIONARY ST:[min: 0, max: 1, num_nulls: 0] SZ:33291 VC:262145

Reporter: Daniel Harper

PRs and other links:

Note: This issue was originally created as PARQUET-668. Please see the migration documentation for further details.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions