Web console: Improve the handling of extreme data (funky datasources,…

… longs) (apache#10641)

* better API escape

* fix escaping issue, bigints

* update licenses

* fix align

* do not show Query with SQL if no SQL

* add prettify script

* update dev readme

* add ordering to the datasource list

* add ordering to supervisor table

licenses.yaml

@@ -4751,7 +4751,7 @@ license_category: binary
 module: web-console
 license_name: Apache License version 2.0
 copyright: Imply Data
-version: 0.10.4
+version: 0.10.5
 
 ---
 
@@ -4915,6 +4915,16 @@ license_file_path: licenses/bin/js-tokens.MIT
 
 ---
 
+name: "json-bigint-native"
+license_category: binary
+module: web-console
+license_name: MIT License
+copyright: Vadim Ogievetsky, Andrey Sidorov
+version: 1.0.0
+license_file_path: licenses/bin/json-bigint-native.MIT
+
+---
+
 name: "lodash.debounce"
 license_category: binary
 module: web-console

licenses/bin/json-bigint-native.MIT

@@ -0,0 +1,20 @@
+The MIT License (MIT)
+
+Copyright (c) 2020 Vadim Ogievetsky, Andrey Sidorov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

web-console/README.md

@@ -19,13 +19,15 @@
 
 # Apache Druid web console
 
-This is the unified Druid web console that servers as a data management layer for Druid.
+This is the Druid web console that servers as a data management interface for Druid.
 
-## How to watch and run for development
+## Developing the console
+
+### Getting started
 
 1. You need to be within the `web-console` directory
 2. Install the modules with `npm install`
-3. Run `npm run compile` to compile the scss files
+3. Run `npm run compile` to compile the scss files (this usually needs to be done only once)
 4. Run `npm start` will start in development mode and will proxy druid requests to `localhost:8888`
 
 
@@ -36,40 +38,31 @@ To try the console in (say) coordinator mode you could run it as such:
 
 `druid_host=localhost:8081 npm start`
 
-## Description of the directory structure
+### Developing
 
-A lot of the directory structure was created to preserve the existing console structure as much as possible.
+You should use a TypeScript friendly IDE (such as [WebStorm](https://www.jetbrains.com/webstorm/), or [VS Code](https://code.visualstudio.com/)) to develop the web console.
 
-As part of this repo:
+The console relies on [tslint](https://palantir.github.io/tslint/), [sass-lint](https://github.com/sasstools/sass-lint), and [prettier](https://prettier.io/) to enforce the code style.
 
-- `assets/` - The images (and other assets) used within the console
-- `e2e-tests/` - End-to-end tests for the console
-- `lib/` - A place where some overrides to the react-table stylus files live, this is outside of the normal SCSS build system.
-- `public/` - The compiled destination of the file powering this console
-- `script/` - Some helper bash scripts for running this console
-- `src/` - This directory (together with `lib`) constitutes all the source code for this console
+If you are going to do any non-trivial development you should set up file watchers in your IDE to automatically fix your code as you type.
 
-## List of non SQL data reading APIs used
+If you do not set up auto file watchers then even a trivial change such as a typo fix might draw the ire of the code style enforcement (it might require some lines to be re-wrapped).
+If you find yourself in that position you should run on or more of:
 
-```
-GET /status
-GET /druid/indexer/v1/supervisor?full
-POST /druid/indexer/v1/worker
-GET /druid/indexer/v1/workers
-GET /druid/indexer/v1/tasks
-GET /druid/coordinator/v1/loadqueue?simple
-GET /druid/coordinator/v1/config
-GET /druid/coordinator/v1/metadata/datasources?includeUnused
-GET /druid/coordinator/v1/rules
-GET /druid/coordinator/v1/config/compaction
-GET /druid/coordinator/v1/tiers
-```
+- `npm run tslint-fix`
+- `npm run sasslint-fix`
+- `npm run prettify`
+
+To get your code into an acceptable state.
 
-## Updating the list of license files
+### Updating the list of license files
 
-From the web-console directory run `script/licenses`
+If you change the dependencies of the console in any way please run `script/licenses` (from the web-console directory).
+It will analyze the changes and update the `../licenses` file as needed.
 
-## Running End-to-End Tests
+Please be conscious of not introducing dependencies on packages with Apache incompatible licenses. 
+
+### Running end-to-end tests
 
 From the web-console directory:
 
@@ -81,21 +74,47 @@ From the web-console directory:
 If you already have a druid cluster running on the standard ports, the steps to build/start/stop a druid cluster can
 be skipped.
 
-### Debugging
-
-#### Screenshots
+#### Screenshots for debugging
 
 `e2e-tests/util/debug.ts:saveScreenshotIfError()` is used to save a screenshot of the web console
 when the test fails. For example, if `e2e-tests/tutorial-batch.spec.ts` fails, it will create
 `load-data-from-local-disk-error-screenshot.png`.
 
-#### Disabling Headless Mode
+#### Disabling headless mode
 
 Disabling headless mode while running the tests can be helpful. This can be done via the `DRUID_E2E_TEST_HEADLESS`
 environment variable, which defaults to `true`.
 
-#### Running Against Alternate Web Console
+#### Running against alternate web console
 
 The environment variable `DRUID_E2E_TEST_UNIFIED_CONSOLE_PORT` can be used to target a web console running on a
 non-default port (i.e., not port `8888`). For example, this environment variable can be used to target the
-development mode of the web console (started via `npm start`), which runs on port `18081`.
+development mode of the web console (started via `npm start`), which runs on port `18081`.
+
+
+## Description of the directory structure
+
+As part of this directory:
+
+- `assets/` - The images (and other assets) used within the console
+- `e2e-tests/` - End-to-end tests for the console
+- `lib/` - A place where some overrides to the react-table stylus files live, this is outside of the normal SCSS build system.
+- `public/` - The compiled destination for the files powering this console
+- `script/` - Some helper bash scripts for running this console
+- `src/` - This directory (together with `lib`) constitutes all the source code for this console
+
+## List of non SQL data reading APIs used
+
+```
+GET /status
+GET /druid/indexer/v1/supervisor?full
+POST /druid/indexer/v1/worker
+GET /druid/indexer/v1/workers
+GET /druid/indexer/v1/tasks
+GET /druid/coordinator/v1/loadqueue?simple
+GET /druid/coordinator/v1/config
+GET /druid/coordinator/v1/metadata/datasources?includeUnused
+GET /druid/coordinator/v1/rules
+GET /druid/coordinator/v1/config/compaction
+GET /druid/coordinator/v1/tiers
+```

web-console/package.json

@@ -51,6 +51,7 @@
     "sasslint-fix": "npm run sasslint -- --fix",
     "sasslint-changed-only": "git diff --diff-filter=ACMR --name-only | grep -E \\.scss$ | xargs ./node_modules/.bin/stylelint --config sasslint.json",
     "sasslint-fix-changed-only": "npm run sasslint-changed-only -- --fix",
+    "prettify": "prettier --write '{src,e2e-tests}/**/*.{ts,tsx,scss}'",
     "generate-licenses-file": "license-checker --production --json --out licenses.json",
     "check-licenses": "license-checker --production --onlyAllow 'Apache-1.1;Apache-2.0;BSD-2-Clause;BSD-3-Clause;0BSD;MIT;CC0-1.0' --summary",
     "start": "webpack-dev-server --hot --open"
@@ -68,11 +69,12 @@
     "d3-axis": "^1.0.12",
     "d3-scale": "^3.2.0",
     "d3-selection": "^1.4.0",
-    "druid-query-toolkit": "^0.10.4",
+    "druid-query-toolkit": "^0.10.5",
     "file-saver": "^2.0.2",
     "fontsource-open-sans": "^3.0.9",
     "has-own-prop": "^2.0.0",
     "hjson": "^3.2.1",
+    "json-bigint-native": "^1.0.0",
     "lodash.debounce": "^4.0.8",
     "lodash.escape": "^4.0.1",
     "memoize-one": "^5.1.1",

web-console/src/components/auto-form/auto-form.tsx

@@ -56,6 +56,7 @@ export interface Field<M> {
   defined?: Functor<M, boolean>;
   required?: Functor<M, boolean>;
   hideInMore?: Functor<M, boolean>;
+  valueAdjustment?: (value: any) => any;
   adjustment?: (model: M) => M;
   issueWithValue?: (value: any) => string | undefined;
 }
@@ -156,6 +157,10 @@ export class AutoForm<T extends Record<string, any>> extends React.PureComponent
     const { model } = this.props;
     if (!model) return;
 
+    if (field.valueAdjustment) {
+      newValue = field.valueAdjustment(newValue);
+    }
+
     let newModel: T;
     if (typeof newValue === 'undefined') {
       if (typeof field.emptyValue === 'undefined') {

web-console/src/components/segment-timeline/segment-timeline.tsx

@@ -270,7 +270,9 @@ ORDER BY "start" DESC`;
             intervals = (await Promise.all(
               datasources.map(async datasource => {
                 const intervalMap = (await Api.instance.get(
-                  `/druid/coordinator/v1/datasources/${datasource}/intervals?simple`,
+                  `/druid/coordinator/v1/datasources/${Api.encodePath(
+                    datasource,
+                  )}/intervals?simple`,
                 )).data;
 
                 return Object.keys(intervalMap)

web-console/src/components/supervisor-statistics-table/supervisor-statistics-table.tsx

@@ -60,7 +60,7 @@ export const SupervisorStatisticsTable = React.memo(function SupervisorStatistic
   props: SupervisorStatisticsTableProps,
 ) {
   const { supervisorId } = props;
-  const endpoint = `/druid/indexer/v1/supervisor/${supervisorId}/stats`;
+  const endpoint = `/druid/indexer/v1/supervisor/${Api.encodePath(supervisorId)}/stats`;
 
   const [supervisorStatisticsState] = useQueryManager<null, SupervisorStatisticsTableRow[]>({
     processQuery: async () => {

web-console/src/dialogs/retention-dialog/retention-dialog.tsx

@@ -47,7 +47,7 @@ export const RetentionDialog = React.memo(function RetentionDialog(props: Retent
   const [historyQueryState] = useQueryManager<string, any[]>({
     processQuery: async datasource => {
       const historyResp = await Api.instance.get(
-        `/druid/coordinator/v1/rules/${datasource}/history`,
+        `/druid/coordinator/v1/rules/${Api.encodePath(datasource)}/history`,
       );
       return historyResp.data;
     },

web-console/src/dialogs/segments-table-action-dialog/segment-table-action-dialog.tsx

@@ -19,12 +19,13 @@
 import React, { useState } from 'react';
 
 import { ShowJson } from '../../components';
+import { Api } from '../../singletons';
 import { BasicAction } from '../../utils/basic-action';
 import { SideButtonMetaData, TableActionDialog } from '../table-action-dialog/table-action-dialog';
 
 interface SegmentTableActionDialogProps {
-  segmentId?: string;
-  datasourceId?: string;
+  segmentId: string;
+  datasourceId: string;
   actions: BasicAction[];
   onClose: () => void;
 }
@@ -53,7 +54,9 @@ export const SegmentTableActionDialog = React.memo(function SegmentTableActionDi
     >
       {activeTab === 'metadata' && (
         <ShowJson
-          endpoint={`/druid/coordinator/v1/metadata/datasources/${datasourceId}/segments/${segmentId}`}
+          endpoint={`/druid/coordinator/v1/metadata/datasources/${Api.encodePath(
+            datasourceId,
+          )}/segments/${Api.encodePath(segmentId)}`}
           downloadFilename={`Segment-metadata-${segmentId}.json`}
         />
       )}

web-console/src/dialogs/supervisor-table-action-dialog/supervisor-table-action-dialog.tsx

@@ -21,6 +21,7 @@ import React, { useState } from 'react';
 import { ShowJson } from '../../components';
 import { ShowHistory } from '../../components/show-history/show-history';
 import { SupervisorStatisticsTable } from '../../components/supervisor-statistics-table/supervisor-statistics-table';
+import { Api } from '../../singletons';
 import { deepGet } from '../../utils';
 import { BasicAction } from '../../utils/basic-action';
 import { SideButtonMetaData, TableActionDialog } from '../table-action-dialog/table-action-dialog';
@@ -64,6 +65,7 @@ export const SupervisorTableActionDialog = React.memo(function SupervisorTableAc
     },
   ];
 
+  const supervisorEndpointBase = `/druid/indexer/v1/supervisor/${Api.encodePath(supervisorId)}`;
   return (
     <TableActionDialog
       sideButtonMetadata={supervisorTableSideButtonMetadata}
@@ -73,7 +75,7 @@ export const SupervisorTableActionDialog = React.memo(function SupervisorTableAc
     >
       {activeTab === 'status' && (
         <ShowJson
-          endpoint={`/druid/indexer/v1/supervisor/${supervisorId}/status`}
+          endpoint={`${supervisorEndpointBase}/status`}
           transform={x => deepGet(x, 'payload')}
           downloadFilename={`supervisor-status-${supervisorId}.json`}
         />
@@ -86,13 +88,13 @@ export const SupervisorTableActionDialog = React.memo(function SupervisorTableAc
       )}
       {activeTab === 'payload' && (
         <ShowJson
-          endpoint={`/druid/indexer/v1/supervisor/${supervisorId}`}
+          endpoint={supervisorEndpointBase}
           downloadFilename={`supervisor-payload-${supervisorId}.json`}
         />
       )}
       {activeTab === 'history' && (
         <ShowHistory
-          endpoint={`/druid/indexer/v1/supervisor/${supervisorId}/history`}
+          endpoint={`${supervisorEndpointBase}/history`}
           downloadFilename={`supervisor-history-${supervisorId}.json`}
         />
       )}

web-console/src/dialogs/task-table-action-dialog/task-table-action-dialog.tsx

@@ -19,6 +19,7 @@
 import React, { useState } from 'react';
 
 import { ShowJson, ShowLog } from '../../components';
+import { Api } from '../../singletons';
 import { deepGet } from '../../utils';
 import { BasicAction } from '../../utils/basic-action';
 import { SideButtonMetaData, TableActionDialog } from '../table-action-dialog/table-action-dialog';
@@ -63,6 +64,7 @@ export const TaskTableActionDialog = React.memo(function TaskTableActionDialog(
     },
   ];
 
+  const taskEndpointBase = `/druid/indexer/v1/task/${Api.encodePath(taskId)}`;
   return (
     <TableActionDialog
       sideButtonMetadata={taskTableSideButtonMetadata}
@@ -72,29 +74,29 @@ export const TaskTableActionDialog = React.memo(function TaskTableActionDialog(
     >
       {activeTab === 'status' && (
         <ShowJson
-          endpoint={`/druid/indexer/v1/task/${taskId}/status`}
+          endpoint={`${taskEndpointBase}/status`}
           transform={x => deepGet(x, 'status')}
           downloadFilename={`task-status-${taskId}.json`}
         />
       )}
       {activeTab === 'payload' && (
         <ShowJson
-          endpoint={`/druid/indexer/v1/task/${taskId}`}
+          endpoint={taskEndpointBase}
           transform={x => deepGet(x, 'payload')}
           downloadFilename={`task-payload-${taskId}.json`}
         />
       )}
       {activeTab === 'reports' && (
         <ShowJson
-          endpoint={`/druid/indexer/v1/task/${taskId}/reports`}
+          endpoint={`${taskEndpointBase}/reports`}
           transform={x => deepGet(x, 'ingestionStatsAndErrors.payload')}
           downloadFilename={`task-reports-${taskId}.json`}
         />
       )}
       {activeTab === 'log' && (
         <ShowLog
           status={status}
-          endpoint={`/druid/indexer/v1/task/${taskId}/log`}
+          endpoint={`${taskEndpointBase}/log`}
           downloadFilename={`task-log-${taskId}.log`}
           tailOffset={16000}
         />