feat: Add utility to print summary of a vector #11859

mbasmanova · 2024-12-14T02:36:42Z

Summary:
Introduce VectorPrinter::summarizeToText helper function to generate human-friendly summary of a vector.

The summary shows the overall hierarchy of the vector and annotates each node with data type, number of rows, encoding, and size in memory.

Dictionary nodes include number of unique indices. Array and Map nodes specify number of empty arrays / maps as well as min/max/avg sizes of the arrays / maps.

ROW(4) 8 rows ROW 528B
   INTEGER 8 rows FLAT 32B
   ARRAY 8 rows ARRAY 288B
         Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3]
      BIGINT 12 rows FLAT 128B
   MAP 8 rows DICTIONARY 192B
         Stats: 0 nulls, 4 unique
      MAP 4 rows MAP 160B
            Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2]
         INTEGER 8 rows FLAT 32B
         REAL 8 rows FLAT 32B
   VARCHAR 8 rows CONSTANT 16B

The summary optionally includes unique node IDs to allow for easy referencing.

0 ROW(4) 8 rows ROW 528B
   0.0 INTEGER 8 rows FLAT 32B
   0.1 ARRAY 8 rows ARRAY 288B
         Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3]
      0.1.0 BIGINT 12 rows FLAT 128B
   0.2 MAP 8 rows DICTIONARY 192B
         Stats: 0 nulls, 4 unique
      0.2.0 MAP 4 rows MAP 160B
            Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2]
         0.2.0.0 INTEGER 8 rows FLAT 32B
         0.2.0.1 REAL 8 rows FLAT 32B
   0.3 VARCHAR 8 rows CONSTANT 16B

The number of RowVector chidren is limited to 5, but can be increased by specifying options.maxChildren:

ROW(4) 8 rows ROW 528B
   INTEGER 8 rows FLAT 32B
   ARRAY 8 rows ARRAY 288B
         Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3]
      BIGINT 12 rows FLAT 128B
   ...2 more

Differential Revision: D67229321

facebook-github-bot · 2024-12-14T02:36:52Z

This pull request was exported from Phabricator. Differential Revision: D67229321

netlify · 2024-12-14T02:36:59Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`74bf4d6`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/676015b9d963020008c04eab

xiaoxmeng

@mbasmanova LGTM. Thanks for adding this!

Summary: Introduce VectorPrinter::summarizeToText helper function to generate human-friendly summary of a vector. The summary shows the overall hierarchy of the vector and annotates each node with data type, number of rows, encoding, and size in memory. Dictionary nodes include number of unique indices. Array and Map nodes specify number of empty arrays / maps as well as min/max/avg sizes of the arrays / maps. ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] INTEGER 8 rows FLAT 32B REAL 8 rows FLAT 32B VARCHAR 8 rows CONSTANT 16B ``` The summary optionally includes unique node IDs to allow for easy referencing. ``` 0 ROW(4) 8 rows ROW 528B 0.0 INTEGER 8 rows FLAT 32B 0.1 ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] 0.1.0 BIGINT 12 rows FLAT 128B 0.2 MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique 0.2.0 MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] 0.2.0.0 INTEGER 8 rows FLAT 32B 0.2.0.1 REAL 8 rows FLAT 32B 0.3 VARCHAR 8 rows CONSTANT 16B ``` The number of RowVector chidren is limited to 5, but can be increased by specifying options.maxChildren: ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B ...2 more ``` Reviewed By: xiaoxmeng Differential Revision: D67229321

facebook-github-bot · 2024-12-14T08:29:31Z

This pull request was exported from Phabricator. Differential Revision: D67229321

Summary: Introduce VectorPrinter::summarizeToText helper function to generate human-friendly summary of a vector. The summary shows the overall hierarchy of the vector and annotates each node with data type, number of rows, encoding, and size in memory. Dictionary nodes include number of unique indices. Array and Map nodes specify number of empty arrays / maps as well as min/max/avg sizes of the arrays / maps. ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] INTEGER 8 rows FLAT 32B REAL 8 rows FLAT 32B VARCHAR 8 rows CONSTANT 16B ``` The summary optionally includes unique node IDs to allow for easy referencing. ``` 0 ROW(4) 8 rows ROW 528B 0.0 INTEGER 8 rows FLAT 32B 0.1 ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] 0.1.0 BIGINT 12 rows FLAT 128B 0.2 MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique 0.2.0 MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] 0.2.0.0 INTEGER 8 rows FLAT 32B 0.2.0.1 REAL 8 rows FLAT 32B 0.3 VARCHAR 8 rows CONSTANT 16B ``` The number of RowVector chidren is limited to 5, but can be increased by specifying options.maxChildren: ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B ...2 more ``` Reviewed By: xiaoxmeng Differential Revision: D67229321

facebook-github-bot · 2024-12-14T08:53:35Z

This pull request was exported from Phabricator. Differential Revision: D67229321

Summary: Introduce VectorPrinter::summarizeToText helper function to generate human-friendly summary of a vector. The summary shows the overall hierarchy of the vector and annotates each node with data type, number of rows, encoding, and size in memory. Dictionary nodes include number of unique indices. Array and Map nodes specify number of empty arrays / maps as well as min/max/avg sizes of the arrays / maps. ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] INTEGER 8 rows FLAT 32B REAL 8 rows FLAT 32B VARCHAR 8 rows CONSTANT 16B ``` The summary optionally includes unique node IDs to allow for easy referencing. ``` 0 ROW(4) 8 rows ROW 528B 0.0 INTEGER 8 rows FLAT 32B 0.1 ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] 0.1.0 BIGINT 12 rows FLAT 128B 0.2 MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique 0.2.0 MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] 0.2.0.0 INTEGER 8 rows FLAT 32B 0.2.0.1 REAL 8 rows FLAT 32B 0.3 VARCHAR 8 rows CONSTANT 16B ``` The number of RowVector chidren is limited to 5, but can be increased by specifying options.maxChildren: ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B ...2 more ``` Reviewed By: xiaoxmeng Differential Revision: D67229321

facebook-github-bot · 2024-12-16T10:55:04Z

This pull request was exported from Phabricator. Differential Revision: D67229321

Summary: Introduce VectorPrinter::summarizeToText helper function to generate human-friendly summary of a vector. The summary shows the overall hierarchy of the vector and annotates each node with data type, number of rows, encoding, and size in memory. Dictionary nodes include number of unique indices. Array and Map nodes specify number of empty arrays / maps as well as min/max/avg sizes of the arrays / maps. ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] INTEGER 8 rows FLAT 32B REAL 8 rows FLAT 32B VARCHAR 8 rows CONSTANT 16B ``` The summary optionally includes unique node IDs to allow for easy referencing. ``` 0 ROW(4) 8 rows ROW 528B 0.0 INTEGER 8 rows FLAT 32B 0.1 ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] 0.1.0 BIGINT 12 rows FLAT 128B 0.2 MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique 0.2.0 MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] 0.2.0.0 INTEGER 8 rows FLAT 32B 0.2.0.1 REAL 8 rows FLAT 32B 0.3 VARCHAR 8 rows CONSTANT 16B ``` The number of RowVector chidren is limited to 5, but can be increased by specifying options.maxChildren: ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B ...2 more ``` Reviewed By: xiaoxmeng Differential Revision: D67229321

facebook-github-bot · 2024-12-16T11:08:16Z

This pull request was exported from Phabricator. Differential Revision: D67229321

Summary: Introduce VectorPrinter::summarizeToText helper function to generate human-friendly summary of a vector. The summary shows the overall hierarchy of the vector and annotates each node with data type, number of rows, encoding, and size in memory. Dictionary nodes include number of unique indices. Array and Map nodes specify number of empty arrays / maps as well as min/max/avg sizes of the arrays / maps. ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] INTEGER 8 rows FLAT 32B REAL 8 rows FLAT 32B VARCHAR 8 rows CONSTANT 16B ``` The summary optionally includes unique node IDs to allow for easy referencing. ``` 0 ROW(4) 8 rows ROW 528B 0.0 INTEGER 8 rows FLAT 32B 0.1 ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] 0.1.0 BIGINT 12 rows FLAT 128B 0.2 MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique 0.2.0 MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] 0.2.0.0 INTEGER 8 rows FLAT 32B 0.2.0.1 REAL 8 rows FLAT 32B 0.3 VARCHAR 8 rows CONSTANT 16B ``` The number of RowVector chidren is limited to 5, but can be increased by specifying options.maxChildren: ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B ...2 more ``` Reviewed By: xiaoxmeng Differential Revision: D67229321

facebook-github-bot · 2024-12-16T11:19:30Z

This pull request was exported from Phabricator. Differential Revision: D67229321

mbasmanova · 2024-12-16T11:55:17Z

macos-14 CI failure seems unrelated: #11871

Summary: Introduce VectorPrinter::summarizeToText helper function to generate human-friendly summary of a vector. The summary shows the overall hierarchy of the vector and annotates each node with data type, number of rows, encoding, and size in memory. Dictionary nodes include number of unique indices. Array and Map nodes specify number of empty arrays / maps as well as min/max/avg sizes of the arrays / maps. ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] INTEGER 8 rows FLAT 32B REAL 8 rows FLAT 32B VARCHAR 8 rows CONSTANT 16B ``` The summary optionally includes unique node IDs to allow for easy referencing. ``` 0 ROW(4) 8 rows ROW 528B 0.0 INTEGER 8 rows FLAT 32B 0.1 ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] 0.1.0 BIGINT 12 rows FLAT 128B 0.2 MAP 8 rows DICTIONARY 192B Stats: 0 nulls, 4 unique 0.2.0 MAP 4 rows MAP 160B Stats: 0 nulls, 1 empty, sizes: [1...4, avg 2] 0.2.0.0 INTEGER 8 rows FLAT 32B 0.2.0.1 REAL 8 rows FLAT 32B 0.3 VARCHAR 8 rows CONSTANT 16B ``` The number of RowVector chidren is limited to 5, but can be increased by specifying options.maxChildren: ``` ROW(4) 8 rows ROW 528B INTEGER 8 rows FLAT 32B ARRAY 8 rows ARRAY 288B Stats: 3 nulls, 1 empty, sizes: [2...4, avg 3] BIGINT 12 rows FLAT 128B ...2 more ``` Reviewed By: xiaoxmeng Differential Revision: D67229321

facebook-github-bot · 2024-12-16T11:58:01Z

This pull request was exported from Phabricator. Differential Revision: D67229321

facebook-github-bot · 2024-12-16T17:34:14Z

This pull request has been merged in 2d31862.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 14, 2024

facebook-github-bot added the fb-exported label Dec 14, 2024

mbasmanova requested review from xiaoxmeng and pedroerp December 14, 2024 02:37

xiaoxmeng approved these changes Dec 14, 2024

View reviewed changes

mbasmanova force-pushed the export-D67229321 branch from 59ef4a5 to cda995a Compare December 14, 2024 08:29

mbasmanova requested review from assignUser and majetideepak as code owners December 14, 2024 08:29

mbasmanova force-pushed the export-D67229321 branch from cda995a to 3c52409 Compare December 14, 2024 08:53

mbasmanova force-pushed the export-D67229321 branch from 3c52409 to 13c0328 Compare December 16, 2024 10:54

mbasmanova force-pushed the export-D67229321 branch from 13c0328 to a327772 Compare December 16, 2024 11:07

mbasmanova force-pushed the export-D67229321 branch from a327772 to 358eb7b Compare December 16, 2024 11:19

mbasmanova force-pushed the export-D67229321 branch from 358eb7b to 74bf4d6 Compare December 16, 2024 11:57

facebook-github-bot closed this in 2d31862 Dec 16, 2024

facebook-github-bot added the Merged label Dec 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add utility to print summary of a vector #11859

feat: Add utility to print summary of a vector #11859

mbasmanova commented Dec 14, 2024

facebook-github-bot commented Dec 14, 2024

netlify bot commented Dec 14, 2024 •

edited

Loading

xiaoxmeng left a comment

facebook-github-bot commented Dec 14, 2024

facebook-github-bot commented Dec 14, 2024

facebook-github-bot commented Dec 16, 2024

facebook-github-bot commented Dec 16, 2024

facebook-github-bot commented Dec 16, 2024

mbasmanova commented Dec 16, 2024 •

edited

Loading

facebook-github-bot commented Dec 16, 2024

facebook-github-bot commented Dec 16, 2024

feat: Add utility to print summary of a vector #11859

feat: Add utility to print summary of a vector #11859

Conversation

mbasmanova commented Dec 14, 2024

facebook-github-bot commented Dec 14, 2024

netlify bot commented Dec 14, 2024 • edited Loading

✅ Deploy Preview for meta-velox canceled.

xiaoxmeng left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Dec 14, 2024

facebook-github-bot commented Dec 14, 2024

facebook-github-bot commented Dec 16, 2024

facebook-github-bot commented Dec 16, 2024

facebook-github-bot commented Dec 16, 2024

mbasmanova commented Dec 16, 2024 • edited Loading

facebook-github-bot commented Dec 16, 2024

facebook-github-bot commented Dec 16, 2024

netlify bot commented Dec 14, 2024 •

edited

Loading

mbasmanova commented Dec 16, 2024 •

edited

Loading