Plugins in Clickhouse to improve flexibility

[prog8]

I have a feeling that Clickhouse could be more interesting to extend if it is possible to extend it without touching the main codebase. One could simply implement a plugin. I can imagine a couple places where plugins could be used:

• functions as plugins (math, date, strings, ...)
  Some functions which implement complex logic can be done using existing math, date, string functions but sometimes they might be optimized if a custom function is just a separate plugin.
• aggregate functions as plugins
• plugins implementing more external dictionaries
  At the moment only a limited number of external data sources can be used. What if one can prepare a new external data source?
• plugins implementing different compression strategies
  General purpose compression algorithms are good but they can be sometimes improved when one knows domain characteristics ... or simply one could use different compression algorithm without changing Clickhouse codebase
• custom backup methods (S3, HDFS, ... Clickhouse incremental backups on distributed storage #1052)
• security (AD integration, document level security, ...)
• plugins can be implemented in any language (C, C++, Go, ...). It just has to be a shared library

I mentioned only a couple use cases. I think each of them could be possibly done in separate tasks (make security filters pluggable, make functions pluggable, ...). I'm just sharing the idea and wanted to get a feedback if you think it makes sense. Could Clickouse be extensible in an easier way?

[prog8]

I've just realized that this is already possible to attach some plugins (Compiler.h) but this is just not documented in https://clickhouse.yandex/docs/en/. Is it only used to automatically compile queries so they can be inlined and speeded up? As far as I understand at the moment only aggregating functions can be compiled but they cannot be loaded from the external .so attached by a user (I can see that SharedLibrary which loads actual .so object is instantiated only after compilation). @alexey-milovidov is it possible to attach external .so which will be loaded and which will register aggregate function by calling factory.registerFunction(?

I have a feeling that we could have a new configuration section (for instance in config.xml in which one can specify a list of external libraries to load) ... or maybe this is already possible and I missed it during a quick scanning of source code.

[theseusyang]

I've interested in this. @alexey-milovidov can you lead this development or make a roadmap plan in 2018 Q1 ?

we will join the development on these plugins extending.

[theseusyang]

@prog8 We have implemented the integration ClickHouse with HDFS, next week We will push codes into github.

[alexey-milovidov]

We don't consider using shared libraries for implementation of plugins due to maintainability issues.
In fact, most of the usages of shared libraries were removed from ClickHouse, examples:

• compiled queries via runtime template instantiation - deprecated after Batch aggregator (experimental) #6435 in Remove Compiler (runtime template instantiation) because we've win over it's performance. #6646 (comiled LLVM expressions still exist);
• interaction with ODBC sources is moved to separate process, clickhouse-odbc-bridge: CLICKHOUSE-3878: ODBC-Bridge tool implementation #2828
• library dictionary source: still exists but will be separated to bridge process similar to clickhouse-odbc-bridge: Library dictionary source is harmful #9502
• CatBoost models: still exists but it's Ok as the shared library does not depend on ClickHouse interfaces and is self-contained.

We don't want to use shared libraries for plugins.
But we consider other possible ways for implementation of plugins.