Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Generated documentation for AST expression support #3172

Closed
jlowe opened this issue Aug 9, 2021 · 1 comment · Fixed by #3259
Closed

[DOC] Generated documentation for AST expression support #3172

jlowe opened this issue Aug 9, 2021 · 1 comment · Fixed by #3259
Assignees
Labels
documentation Improvements or additions to documentation P0 Must have for release

Comments

@jlowe
Copy link
Member

jlowe commented Aug 9, 2021

With the addition of AST expression support in #3165, we need the ability to generate documentation describing what is supported by an AST expression. We will also have to consider how best to present this in a way that doesn't confuse users with respect to normal GPU expression support.

@jlowe jlowe added documentation Improvements or additions to documentation ? - Needs Triage Need team to review and classify labels Aug 9, 2021
@revans2
Copy link
Collaborator

revans2 commented Aug 9, 2021

We already have a section for each expression saying if it works as a part of lambda or not. Right now nothing works with lambda. But AST is not a one to one mapping to lambda. For some lambda operations, like ArrayTransform, we can do it without any AST support. But we could also do it with AST support. Where as others, like ArraySort, I don't see any good way to do it fully without AST support.

To me it feels like we almost want to replace the Lambda section with an AST section, and then have a way in the docs to explain what operations, expressions or variable require AST support. This could then be expanded in the future to possibly include some hybrid AST/Non-AST support.

So for example with Add we could update it to look like

Expression SQL Functions(s) Description Notes Context Param/Output BOOLEAN BYTE SHORT INT LONG FLOAT DOUBLE DATE TIMESTAMP STRING DECIMAL NULL BINARY CALENDAR ARRAY MAP STRUCT UDT
Add `+` Addition None project lhs S S S S S S PS
max DECIMAL precision of 18
NS
rhs S S S S S S PS
max DECIMAL precision of 18
NS
result S S S S S S PS
max DECIMAL precision of 18
NS
AST lhs NS NS S S S S NS NS
rhs NS NS S S S S NS NS
result NS NS S S S S NS NS

And for join when we do support parameters for SparkPlan execs we could have something more like

Executor Description Notes Param BOOLEAN BYTE SHORT INT LONG FLOAT DOUBLE DATE TIMESTAMP STRING DECIMAL NULL BINARY CALENDAR ARRAY MAP STRUCT UDT
BroadcastHashJoinExec Implementation of join using broadcast data None conditional (AST param) S

@Salonijain27 Salonijain27 added P0 Must have for release and removed ? - Needs Triage Need team to review and classify labels Aug 10, 2021
@revans2 revans2 self-assigned this Aug 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation P0 Must have for release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants