Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add single line description of ExecutionPlan (#2216) #2217

Merged
merged 3 commits into from
Apr 13, 2022
Merged

Add single line description of ExecutionPlan (#2216) #2217

merged 3 commits into from
Apr 13, 2022

Conversation

tustvold
Copy link
Contributor

Which issue does this PR close?

Closes #2216

Rationale for this change

See ticket

What changes are included in this PR?

Adds the ability to get a single-line summary of the root of an ExecutionPlan

Are there any user-facing changes?

No

@tustvold tustvold requested a review from alamb April 12, 2022 16:25
@github-actions github-actions bot added the datafusion Changes in the datafusion crate label Apr 12, 2022
@alamb
Copy link
Contributor

alamb commented Apr 13, 2022

Looks useful to me 👍

@tustvold @alamb
Update datafusion/core/src/physical_plan/display.rs
cb76484 
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@yjshen
Copy link
Member

yjshen commented Apr 13, 2022 •
edited
Loading

Another physical plan description optimization would be to limit each physical plan's max line width. This would be especially useful for the ParquetExec node:

ProjectionExec: expr=[ss_sold_date_sk@0 as ss_sold_date_sk, ss_customer_sk@1 as ss_customer_sk]
      FilterExec: ss_sold_date_sk@0 IS NOT NULL
        ParquetExec: limit=None, partitions=[/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00000-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00023-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00012-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00001-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00024-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00030-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00008-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00056-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00010-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00078-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00028-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00016-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00017-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00002-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00021-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00027-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00032-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00031-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00034-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00014-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00005-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00026-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00019-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00011-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00013-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00089-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00020-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00029-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00004-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00015-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00007-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00018-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00067-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00022-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00009-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00006-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00045-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00033-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00003-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00025-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00035-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00096-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00062-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00068-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00066-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00072-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00077-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00086-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00061-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00063-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00052-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00082-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00075-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00087-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00059-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00070-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00053-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00050-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00084-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00085-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00049-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00080-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00038-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00046-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00074-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00043-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00076-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00093-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00058-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00039-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00090-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00036-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00083-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00051-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00079-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00092-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00055-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00081-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00088-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00071-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00044-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00042-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00069-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00048-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00054-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00095-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00065-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00047-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00094-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00037-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00073-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00064-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00057-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00041-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00060-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00091-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00040-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet, /Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00097-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet], predicate=true, projection=[ss_sold_date_sk, ss_customer_sk]

It would be wonderful if I could reduce this to:

....
.....
ParquetExec: limit=None, predicate=true, projection=[ss_sold_date_sk, ss_customer_sk], partitions=[100 items ... ]

Actually, I was thinking of this when I first saw the title.

@tustvold
Copy link
Contributor Author

Perhaps #2079 might help with that 😅

@tustvold tustvold merged commit 7a6317a into apache:master Apr 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alamb alamb alamb approved these changes

@yjshen yjshen yjshen approved these changes

Assignees
No one assigned
Labels
datafusion Changes in the datafusion crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Render Single Line Description of ExecutionPlan
3 participants
@tustvold @alamb @yjshen