Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operation output check is too restrictive #931

Closed
wajda opened this issue Aug 2, 2021 · 1 comment
Closed

Operation output check is too restrictive #931

wajda opened this issue Aug 2, 2021 · 1 comment
Assignees
Milestone

Comments

@wajda
Copy link
Contributor

wajda commented Aug 2, 2021

Causes AbsaOSS/spline-spark-agent#264

The following check constraint in ExecutionPlanPersistentModelBuilder doesn't assume existence of outputs schema with no attributes ([]).

      case (Some(schemaAgnosticTermOp), Some(schemaAwareAnyOp)) =>
        throw new InconsistentEntityException(s"" +
          s"At least one operation defines output schema [#${schemaAwareAnyOp.id}], " +
          s"while some terminal input operations lack of it [#${schemaAgnosticTermOp.id}]")

But there are some specific cases when this can be valid. For instance, when a data frame is created by selecting literals from an empty data frame.

SELECT 42 as x

Technically the execution plan could start from a SELECT operation, but someone would argue that it would not be 100% correct representation of an execution plan. A SELECT operation projects something on something else, so it has to be preceded by another (synthetic) operation with Nil output, and we want to preserve that relation.

@wajda
Copy link
Contributor Author

wajda commented Aug 5, 2021

The fix requires changing in a Producer model to distinguish between an empty schema and an undefined schema.
Remediation will be implemented in the agent AbsaOSS/spline-spark-agent#264 instead and this issue will be moved to 1.0.0

@wajda wajda modified the milestones: 0.6.2, 1.0.0 Aug 5, 2021
@wajda wajda modified the milestones: 1.0.0, 0.7.0 Aug 11, 2021
wajda added a commit that referenced this issue Aug 11, 2021
wajda added a commit that referenced this issue Aug 11, 2021
wajda added a commit that referenced this issue Aug 12, 2021
* spline #931 [DRAFT] Operation output check is too tight

* spline #931 fix a typo + full name of type aliases

* spline #931 Update rest-api-doc-generator version to 1.0.4

* spline #931 + doc

* spline #931 minor optimization
@wajda wajda closed this as completed Aug 16, 2021
@wajda wajda changed the title Operation output check is too tight Operation output check is too restrictive Aug 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

1 participant