Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ISSUES-863 fix predicate not work for asterisk #2929

Conversation

zhang2014
Copy link
Contributor

#863 #2015
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

@zhang2014 zhang2014 force-pushed the fix/support_asterisk_for_predcate_optimize branch from 33e073a to 32cc611 Compare August 24, 2018 02:05
@alexey-milovidov
Copy link
Member

There are failed tests, example:
https://travis-ci.org/yandex/ClickHouse/jobs/419930263#L9155

@zhang2014
Copy link
Contributor Author

I turned on enable_optimize_predicate_expression and was checking each test.

#2015 (comment)

@zhang2014 zhang2014 force-pushed the fix/support_asterisk_for_predcate_optimize branch from 6ffa8e0 to bf517d5 Compare August 24, 2018 04:11
@zhang2014
Copy link
Contributor Author

zhang2014 commented Aug 24, 2018

All tests passed. need to turn off enable_optimize_predicate_expression by default ?

@alexey-milovidov
Copy link
Member

Yes, just in case. We will first enable manually on our servers, then we will enable by default.

@alexey-milovidov alexey-milovidov merged commit 5637b30 into ClickHouse:master Aug 24, 2018
@zhang2014
Copy link
Contributor Author

Yes, just in case. We will first enable manually on our servers, then we will enable by default.

BTW, i wanna tell a digression, compile and compile_expressions open on your servers? How did it work?

@alexey-milovidov
Copy link
Member

alexey-milovidov commented Aug 27, 2018

We enable compile setting on our servers, but not compile_expressions.

compile is pretty mature and stable feature.

When this setting is enabled, ClickHouse will generate specialized code for first step of aggregation (GROUP BY). See SpecializedAggregator.cpp, Compiler and Aggregator::compileIfPossible. It generates tiny file with C++ code that contains explicit template instantiation, then executes clickhouse clang to compile it to .so binary, dlopens it and calls a function from that library.

For this feature to work, you have to build ClickHouse package with ClickHouse C++ headers. If you run ./release script, it will do that for you. Also you can install ClickHouse headers manually (see copy_headers.sh).

You have to build ClickHouse with embedded compiler (ENABLE_EMBEDDED_COMPILER=1 CMake option). To build with embedded compiler, you have to use LLVM, Clang and LLD libraries. You can use that libraries from the submodule (USE_INTERNAL_LLVM_LIBRARY=1) or from the OS packages.

If you use LLVM from the submodule, it will take much more time to compile (few times more than ClickHouse without LLVM). If you want to use LLVM from OS packages, you have to install liblld-6.0-dev libclang-6.0-dev liblld-6.0. It's not recommended to use OS packages, because to link with libraries, C++ ABI should match: if you use something different than gcc and libstdc++, you will get linkage errors.

compile_expressions is new and experimental.

It uses LLVM (directly through LLVM IRBuilder) to generate code to transform expressions with compositions of simple functions (arithmetic, logic, comparison, if...) to a single function that will execute in one pass.

Currently it is not used in production, because some additional work remains. For example, a cache of compiled modules is needed. @alesapin is doing this task right now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp-optimizers Query optimizations
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants