feature/cli (#941)

* Bump ts and fix some polymorphic code. and by fix I mean lie to compiler. (#927)

* http basic auth built in (#921)

* cluster auth settings in cluster configuration

* compose request decorator with cluster auth headers

* update docs

* simplify auth headers merging

* lift returned value to Promise to simplify

* missing tests

* move docs

* simplify test construction

* cli refactor (#924)

* main menu for cli

* utils for commander

* create server as a function with explicit dependencies

* util for loading file, soon will talk to commander

* explicit default value

* explicit dependencie on timekeeper getter

* we always post process loaded files so enforce that intypes

* create express app in function with explicit dependencies

* implement commands

* be careful to what "this" is bound

* use relative path because sources endpoint is placed on server-root directory not a file system root

* add required option for time attribute for json files

* columns from json files are loaded later so we can't check for time column at the start

* add verbose option for config verification

* refactor a little

* handle version read failure

* pass command to server so we can recover from errors

* lint fixes

* native data cubes dimensions are loaded after introspection so we can't check for dimension on cube creation

* auth is optional

* pass parsed cluster credentials to sources config

* add "Basic" prefix to encoded credentials header

* handle errors when loading config file

* remove config.ts file

* remove old comments

* "generic" parse credentials

* remove port from introspection

* runTurnilo util

* add few tests (#931)

* add few tests

* match npm scripts with new commands to check CI

* fix deploy workflows with new commands

* more documentation (#936)

* more documentation

* Update src/server/cli/options.ts

Co-authored-by: Marcin Kuthan <marcin.kuthan@allegro.pl>

* Update src/server/cli.ts

Co-authored-by: Marcin Kuthan <marcin.kuthan@allegro.pl>

* Update README.md

Co-authored-by: Marcin Kuthan <marcin.kuthan@allegro.pl>

* Update README.md

Co-authored-by: Piotr Szczepanik <piter75@gmail.com>

* update more docs

Co-authored-by: Marcin Kuthan <marcin.kuthan@allegro.pl>
Co-authored-by: Piotr Szczepanik <piter75@gmail.com>

* remove legacy tests (#937)

* remove legacy tests

* keep mocha setup file

* log on stdout only new config and exit process after succesful introspection (#939)

Co-authored-by: Marcin Kuthan <marcin.kuthan@allegro.pl>
Co-authored-by: Piotr Szczepanik <piter75@gmail.com>

.github/workflows/deploy-branch.yml

@@ -48,7 +48,7 @@ jobs:
             --quiet \
             --port 9090 \
             --cpu 1 --memory 1G --max-instances 1 --concurrency 80 \
-            --args="--examples"
+            --args="run-examples"
 
 
       - name: Get app URL

.github/workflows/deploy-master.yml

@@ -43,7 +43,7 @@ jobs:
             --quiet \
             --port 9090 \
             --cpu 1 --memory 1G --max-instances 1 --concurrency 80 \
-            --args="--examples"
+            --args="run-examples"
 
       - name: Delete previous Docker image(s)
         run: |

README.md

@@ -72,18 +72,18 @@ Install Turnilo distribution using [npm](https://www.npmjs.com/).
 npm install -g turnilo
 ```
 
-Start off by running an example with Covid-19 and Wikipedia datasets
+Start off by running Turnilo with example datasets
 and open [http://localhost:9090/](http://localhost:9090/).
 
 ```
-turnilo --examples
+turnilo run-examples
 ```
 
-Or connect to the existing Druid broker using `--druid` command line option.
+Use `connect-druid` command to connect to the existing Druid broker.
 Turnilo will automatically introspect your Druid broker and figure out available datasets.
 
 ```
-turnilo --druid http[s]://druid-broker-hostname[:port]
+turnilo connect-druid http[s]://druid-broker-hostname[:port]
 ```
 
 ## Documentation
@@ -107,7 +107,7 @@ npm run build
 
 ### Run project
 
-Run Covid-19 and Wikipedia examples.
+Run example datasets.
 
 ```
 npm run start:examples
@@ -116,20 +116,20 @@ npm run start:examples
 Connect to the existing Druid broker.
 
 ```
-npm run start -- --druid http[s]://druid-broker-hostname[:port]
+npm run start -- connect-druid http[s]://druid-broker-hostname[:port]
 ```
 
 Connect to the existing Druid broker using your config file.
 
 ```
-npm run start -- --config path/to/config.yml
+npm run start -- run-config path/to/config.yml
 ```
 
 ### Run project in developer mode
 
 Every change in frontend code would recompile project and reload page.
 
-Run Covid-19 and Wikipedia examples.
+Run example datasets.
 
 ```
 npm run start:dev:examples
@@ -138,13 +138,13 @@ npm run start:dev:examples
 Connect to the existing Druid broker.
 
 ```
-npm run start:dev -- --druid http[s]://druid-broker-hostname[:port]
+npm run start:dev -- connect-druid http[s]://druid-broker-hostname[:port]
 ```
 
 Connect to the existing Druid broker using your config file.
 
 ```
-npm run start:dev -- --config path/to/config.yml
+npm run start:dev -- run-config path/to/config.yml
 ```
 
 

bin/turnilo

@@ -19,6 +19,6 @@
 
 var path = require('path');
 var fs   = require('fs');
-var lib = path.join(path.dirname(fs.realpathSync(__filename)), '../build/server/www.js');
+var lib = path.join(path.dirname(fs.realpathSync(__filename)), '../build/server/cli.js');
 
 require(lib);

docs/configuration-cluster.md

@@ -6,7 +6,7 @@ layout: page
 
 ## Overview
 
-It is easy to start using Turnilo with Druid by pointing it at your Druid cluster: `turnilo --druid broker_host:broker_port`
+It is easy to start using Turnilo with Druid by pointing it at your Druid cluster: `turnilo connect-druid broker_host:broker_port`
 Turnilo will automatically introspect your Druid cluster and figure out available datasets.
 
 Turnilo can be configured with a *config* YAML file. While you could write one from scratch it is recommended to let
@@ -15,13 +15,13 @@ Turnilo give you a head start by using it to generate a config file for you usin
 Run:
 
 ```bash
-turnilo --druid broker_host:broker_port --print-config --with-comments > config.yaml
+turnilo introspect-druid broker_host:broker_port --verbose > config.yaml
 ```
 
 This will cause Turnilo to go through its normal startup and introspection routine and then dump the internally generated
 config (complete with comments) into the provided file.
 
-You can now run `turnilo --config config.yaml` to run Turnilo with your config.
+You can now run `turnilo run-config config.yaml` to run Turnilo with your config.
 
 The next step is to open the generated config file in your favourite text editor and configure Turnilo to your liking.
 Below we will go through a typical configuration flow. At any point you can save the config and re-launch Turnilo to load
@@ -120,6 +120,25 @@ The url address (http[s]://hostname[:port]) of the cluster. If no port, 80 is as
 
 The host (hostname:port) of the cluster, http protocol is assumed. Deprecated, use **url** field
 
+**auth**
+
+The cluster authorization strategy.
+
+* Http Basic authorization
+
+Strategy will add `Authorization` header to each request to cluster and encode passed username and password with base64.
+
+```yaml
+auth:
+  type: "http-basic"
+  username: Aladdin
+  password: OpenSesame
+```
+
+This would result in all Druid request having added headers
+
+![](assets/images/basic-auth-headers.png)
+
 **version** (string)
 
 The explicit version to use for this cluster.

docs/example/request-decoration/config.yaml

@@ -9,6 +9,7 @@ clusters:
     requestDecorator:
       path: './druid-request-decorator.js'
       options:
-        username: Aladdin
-        password: OpenSesame
+        base: Pancakes
+        extras:
+          - Blueberries
 

docs/example/request-decoration/druid-request-decorator.js

@@ -5,20 +5,13 @@ exports.version = 1;
 //   * options: the options field from the requestDecorator property
 //   * cluster: Cluster - the cluster object
 exports.druidRequestDecoratorFactory = function (logger, params) {
-  const options = params.options;
-  const username = options.username; // pretend we store the username and password
-  const password = options.password; // in the config
 
-  if (!username) {
-    throw new Error("must have username");
-  }
-  if (!password) {
-    throw new Error("must have password");
-  }
+  const options = params.options;
+  const extras = options.extras.join(", ");
 
-  logger.log("Decorator init for username: " + username);
+  const like = `${options.base} with ${extras}`;
 
-  const auth = "Basic " + Buffer.from(`${username}:${password}`).toString("base64");
+  logger.log("Decorator created with options:", { options });
 
   // decoratorRequest: DecoratorRequest - is an object that has the following keys:
   //   * method: string - the method that is used (POST or GET)
@@ -27,8 +20,7 @@ exports.druidRequestDecoratorFactory = function (logger, params) {
   return function (decoratorRequest) {
     const decoration = {
       headers: {
-        "Authorization": auth,
-        "X-I-Like": "Koalas",
+        "X-I-Like": like,
       },
     };
 

docs/extending-turnilo.md

@@ -28,10 +28,10 @@ to your `druidRequestDecoratorFactory` under `options` key in second parameter.
 druidRequestDecorator: 
     path: './druid-request-decorator.js'
     options:
-        keyA: valueA
-        keyB:
-          - firstElement
-          - secondElement
+        base: Pancakes
+        extras:
+          - maple-syrup
+          - blueberries
 ```
 
 The contract is that your module should export a function `druidRequestDecoratorFactory` that has to return a decorator.
@@ -46,15 +46,14 @@ exports.version = 1;
 
 exports.druidRequestDecoratorFactory = function (logger, params) {
   const options = params.options;
-  const username = options.username;
-  const password = options.password;
+  const extras = options.extras.join(", ");
 
-  const auth = "Basic " + Buffer.from(`${username}:${password}`).toString("base64");
+  const like = `${options.base} with ${extras}`;
 
   return function () {
     return {
       headers: {
-        "Authorization": auth
+        "X-I-Like": auth
       },
     };
   };
@@ -63,9 +62,7 @@ exports.druidRequestDecoratorFactory = function (logger, params) {
 
 You can find this example with additional comments and example config in the [example](example/request-decoration) folder.
 
-This would result in all Druid requests being tagged as:
-
-![decoration example](example/request-decoration/result.png)
+Please note that your object will be merged with [Cluster Authorization](configuration-cluster.md) headers.
 
 ## Query decorator