[ES|QL] Comment parsing and pretty-printing (#192173)

## Summary TL;DR - Adds ability to parse out comments from source to AST. - Adds ability for every AST node to have *decoration*—comments, which can be attached from left, top, and right from the node. - Implements routine which attached comments to AST nodes. - In `BasicPrettyPrinter` adds support only for *left* and *right* comment printing, as the basic printer prints only on one line. - In `WrappingPrettyPrinter` adds support for all comment printing for all AST nodes. - Introduces a `Query` object and `query` AST node, which represent thole query—the root node, list of commands. - The ES|QL AST example plugin now displays the pretty-printed text version. ### Comments This PR introduced an optional `formatting` field for all AST nodes. In the `formatting` field one can specify comment decorations from different sides of a node. When parsing, once can now specify the `{ withComments: true }` option, which will collect all comments from the source while parsing using the `collectDecorations` routine. It will then also call the `attachDecorations`, which walks the AST and assigns each comment to some AST node. Further, traversal and pretty-print API have been updated to work with comments: - The `Walker` has been updated to be able to walk all comments from the AST. - The `BasicPrettyPrinter` adds support only for *left* and *right* inline comment printing, as the basic printer prints only on one line. - The `WrappingPrettyPrinter` adds support for all comment printing for all AST nodes. It switches to line-break printing mode if it detects there are comments with line breaks (those could be multi-line comments, or single line comments—single line comments are always followed by a line break). It also correctly inserts punctuation, when an AST node is surrounded by comments. ### Parsing utils All parsing utils have been moved to the `/parser` sub-folder. Files in the `/parser` folder have been renamed as per Kibana convention to reflect what is inside the file. For example, the `EsqlErrorListener` class is in a file named `esql_error_listener.ts`. A `Query` class and `ESQLAstQueryExpression` AST nodes have been introduced. They represent the result of a full query parse. (Before that, the AST root was just an array of command nodes, now the AST root is represented by the `ESQLAstQueryExpression` node.) ### Builder I have started the implementation of the `Builder` static class in the `/builder` folder. It is simply a collection of stateless AST node factories—functions which construct AST nodes. Some of the `Builder` methods are already used by the parser, more will follow. We will also use the `Builder` in upcoming [*Mutation API*](#191812). ### ES|QL Example Plugin This PR sets up Storybook and implements few Storybook stories for the ES|QL AST example plugin, run it with: ``` yarn storybook esql_ast_inspector ``` This PR updates the *ES|QL AST Explorer* example plugin. Start Kibana with example plugins enabled: ``` yarn start --run-examples ``` And navigate to [`/app/esql_ast_inspector`](http://localhost:5601/app/esql_ast_inspector) to see the new example plugin UI. ![esql-ast-explorer](https://github.com/user-attachments/assets/8ded91ea-1b60-4514-8cf5-c8a4066a3a12) ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co> (cherry picked from commit 2217337)
elastic · Sep 26, 2024 · e5f98db · e5f98db
1 parent 86dbb85
commit e5f98db
Show file tree

Hide file tree

Showing 87 changed files with 5,330 additions and 607 deletions.
diff --git a/examples/esql_ast_inspector/.storybook/main.js b/examples/esql_ast_inspector/.storybook/main.js
@@ -0,0 +1,10 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+module.exports = require('@kbn/storybook').defaultConfig;
diff --git a/examples/esql_ast_inspector/public/app.tsx b/examples/esql_ast_inspector/public/app.tsx
@@ -7,84 +7,21 @@
  * License v3.0 only", or the "Server Side Public License, v 1".
  */
 
-import React, { useRef, useState } from 'react';
-import {
-  EuiPage,
-  EuiPageBody,
-  EuiPageSection,
-  EuiPageHeader,
-  EuiSpacer,
-  EuiForm,
-  EuiTextArea,
-  EuiFormRow,
-  EuiButton,
-} from '@elastic/eui';
+import * as React from 'react';
+import { EuiPage, EuiPageBody, EuiPageSection, EuiPageHeader, EuiSpacer } from '@elastic/eui';
 import { EuiProvider } from '@elastic/eui';
+import { EsqlInspector } from './components/esql_inspector';
 
-import type { CoreStart } from '@kbn/core/public';
-
-import { EditorError, ESQLAst, getAstAndSyntaxErrors } from '@kbn/esql-ast';
-import { CodeEditor } from '@kbn/code-editor';
-import type { StartDependencies } from './plugin';
-
-export const App = (props: { core: CoreStart; plugins: StartDependencies }) => {
-  const [currentErrors, setErrors] = useState<EditorError[]>([]);
-  const [currentQuery, setQuery] = useState(
-    'from index1 | eval var0 = round(numberField, 2) | stats by stringField'
-  );
-
-  const inputRef = useRef<HTMLTextAreaElement | null>(null);
-
-  const [ast, setAST] = useState<ESQLAst>(getAstAndSyntaxErrors(currentQuery).ast);
-
-  const parseQuery = (query: string) => {
-    const { ast: _ast, errors } = getAstAndSyntaxErrors(query);
-    setErrors(errors);
-    setAST(_ast);
-  };
-
+export const App = () => {
   return (
     <EuiProvider>
       <EuiPage>
-        <EuiPageBody style={{ maxWidth: 800, margin: '0 auto' }}>
+        <EuiPageBody style={{ maxWidth: 1200, margin: '0 auto' }}>
           <EuiPageHeader paddingSize="s" bottomBorder={true} pageTitle="ES|QL AST Inspector" />
           <EuiPageSection paddingSize="s">
             <p>This app gives you the AST for a particular ES|QL query.</p>
-
-            <EuiSpacer />
-
-            <EuiForm>
-              <EuiFormRow
-                fullWidth
-                label="Query"
-                isInvalid={Boolean(currentErrors.length)}
-                error={currentErrors.map((error) => error.message)}
-              >
-                <EuiTextArea
-                  inputRef={(node) => {
-                    inputRef.current = node;
-                  }}
-                  isInvalid={Boolean(currentErrors.length)}
-                  fullWidth
-                  value={currentQuery}
-                  onChange={(e) => setQuery(e.target.value)}
-                  css={{
-                    height: '5em',
-                  }}
-                />
-              </EuiFormRow>
-              <EuiFormRow fullWidth>
-                <EuiButton fullWidth onClick={() => parseQuery(inputRef.current?.value ?? '')}>
-                  Parse
-                </EuiButton>
-              </EuiFormRow>
-            </EuiForm>
             <EuiSpacer />
-            <CodeEditor
-              allowFullScreen={true}
-              languageId={'json'}
-              value={JSON.stringify(ast, null, 2)}
-            />
+            <EsqlInspector />
           </EuiPageSection>
         </EuiPageBody>
       </EuiPage>

diff --git a/examples/esql_ast_inspector/public/components/annotations/annotations.stories.tsx b/examples/esql_ast_inspector/public/components/annotations/annotations.stories.tsx
@@ -0,0 +1,28 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+import * as React from 'react';
+import { Annotations } from './annotations';
+
+export default {
+  title: '<Annotations>',
+  parameters: {},
+};
+
+export const Default = () => (
+  <Annotations
+    value={'FROM index | LIMIT 10 | SORT some_field'}
+    annotations={[
+      [0, 4, (text) => <span style={{ color: 'red' }}>{text}</span>],
+      [5, 10, (text) => <span style={{ color: 'blue' }}>{text}</span>],
+      [13, 18, (text) => <span style={{ color: 'red' }}>{text}</span>],
+      [19, 21, (text) => <span style={{ color: 'green' }}>{text}</span>],
+    ]}
+  />
+);
diff --git a/examples/esql_ast_inspector/public/components/annotations/annotations.tsx b/examples/esql_ast_inspector/public/components/annotations/annotations.tsx
@@ -0,0 +1,43 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+import * as React from 'react';
+import type { Annotation } from './types';
+
+export interface AnnotationsProps {
+  value: string;
+  annotations?: Annotation[];
+}
+
+export const Annotations: React.FC<AnnotationsProps> = (props) => {
+  const { value, annotations = [] } = props;
+  const annotationNodes: React.ReactNode[] = [];
+
+  let pos = 0;
+
+  for (const [start, end, render] of annotations) {
+    if (start > pos) {
+      const text = value.slice(pos, start);
+
+      annotationNodes.push(<span>{text}</span>);
+    }
+
+    const text = value.slice(start, end);
+
+    pos = end;
+    annotationNodes.push(render(text));
+  }
+
+  if (pos < value.length) {
+    const text = value.slice(pos);
+    annotationNodes.push(<span>{text}</span>);
+  }
+
+  return React.createElement('span', {}, ...annotationNodes);
+};
diff --git a/examples/esql_ast_inspector/public/components/annotations/index.ts b/examples/esql_ast_inspector/public/components/annotations/index.ts
@@ -0,0 +1,11 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+export { Annotations, type AnnotationsProps } from './annotations';
+export type { Annotation } from './types';
diff --git a/...es/kbn-esql-ast/src/ast_position_utils.ts → ...or/public/components/annotations/types.ts b/...es/kbn-esql-ast/src/ast_position_utils.ts → ...or/public/components/annotations/types.ts
@@ -7,19 +7,10 @@
  * License v3.0 only", or the "Server Side Public License, v 1".
  */
 
-import type { Token } from 'antlr4';
+import * as React from 'react';
 
-export function getPosition(
-  token: Pick<Token, 'start' | 'stop'> | null,
-  lastToken?: Pick<Token, 'stop'> | undefined
-) {
-  if (!token || token.start < 0) {
-    return { min: 0, max: 0 };
-  }
-  const endFirstToken = token.stop > -1 ? Math.max(token.stop + 1, token.start) : undefined;
-  const endLastToken = lastToken?.stop;
-  return {
-    min: token.start,
-    max: endLastToken ?? endFirstToken ?? Infinity,
-  };
-}
+export type Annotation = [
+  start: number,
+  end: number,
+  annotation: (text: string) => React.ReactNode
+];
diff --git a/examples/esql_ast_inspector/public/components/esql_editor/esql_editor.tsx b/examples/esql_ast_inspector/public/components/esql_editor/esql_editor.tsx
@@ -0,0 +1,91 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+import * as React from 'react';
+import { css } from '@emotion/react';
+import { Annotations, type Annotation } from '../annotations';
+import { FlexibleInput } from '../flexible_input/flexible_input';
+
+const blockCss = css({
+  display: 'inline-block',
+  position: 'relative',
+  width: '100%',
+  fontSize: '18px',
+  lineHeight: '1.3',
+  fontFamily:
+    "'SF Mono', SFMono-Regular, ui-monospace, 'DejaVu Sans Mono', Menlo, Consolas, monospace",
+});
+
+const backdropCss = css({
+  display: 'inline-block',
+  position: 'absolute',
+  left: 0,
+  width: '100%',
+  pointerEvents: 'all',
+  userSelect: 'none',
+  whiteSpace: 'pre',
+  color: 'rgba(255, 255, 255, 0.01)',
+});
+
+const inputCss = css({
+  display: 'inline-block',
+  color: 'rgba(255, 255, 255, 0.01)',
+  caretColor: '#07f',
+});
+
+const overlayCss = css({
+  display: 'inline-block',
+  position: 'absolute',
+  left: 0,
+  width: '100%',
+  pointerEvents: 'none',
+  userSelect: 'none',
+  whiteSpace: 'pre',
+});
+
+export interface EsqlEditorProps {
+  src: string;
+  backdrops?: Annotation[][];
+  highlight?: Annotation[];
+  onChange: (src: string) => void;
+}
+
+export const EsqlEditor: React.FC<EsqlEditorProps> = (props) => {
+  const { src, highlight, onChange } = props;
+
+  const backdrops: React.ReactNode[] = [];
+
+  if (props.backdrops) {
+    for (let i = 0; i < props.backdrops.length; i++) {
+      const backdrop = props.backdrops[i];
+
+      backdrops.push(
+        <div key={i} css={backdropCss}>
+          <Annotations value={src} annotations={backdrop} />
+        </div>
+      );
+    }
+  }
+
+  const overlay = !!highlight && (
+    <div css={overlayCss}>
+      <Annotations value={src} annotations={highlight} />
+    </div>
+  );
+
+  return (
+    <div css={blockCss}>
+      {backdrops}
+      <div css={inputCss}>
+        <FlexibleInput multiline value={src} onChange={(e) => onChange(e.target.value)} />
+      </div>
+      {overlay}
+    </div>
+  );
+};