Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Externalize pipeline architecture #20

Merged
merged 19 commits into from
Apr 14, 2019
Merged

Externalize pipeline architecture #20

merged 19 commits into from
Apr 14, 2019

Conversation

Callidon
Copy link
Owner

@Callidon Callidon commented Apr 10, 2019

Implements the functionalities discusses in #15
WARNING: this PR introduces major changes to almost every class in the framework, so it will likely break all existing forks.

The changes are:

  • All the pipeline-related operations are now abstracted using PipelineEngine.
  • Subclasses of PipelineEngine are used to execute every operation in the main query engine, which removes the strong coupling to rxjs.
  • RxjsPipeline is a subclass of PipelineEngine which implements the pipeline operations using rxjs. It corresponds to the engine behaviour before this PR. RxjsPipeline is used as the default pipeline implementation.
  • VectorPiepline is a subclass of PipelineEngine which follows the vectorization paradigm: all intermediate results are materialized in main memory. This pipeline impl. is more efficient CPU-wise, but consume more memory.
  • Added a configurable test suite to test all subclasses of PipelineEngine.
  • Update the README with a short tutorial on these new functionalities.

@Callidon Callidon merged commit bf4c602 into master Apr 14, 2019
@Callidon Callidon deleted the external-pipeline branch April 14, 2019 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant