[Console] Use ES specification for autocomplete definitions #159241

yuliacech · 2023-06-07T16:56:29Z

Summary

This PR adds a new package kbn-generate-console-definitions that will eventually replace the package kbn-spec-to-console. It also adds a new command to use the script in the new package. The new command can be used as following:

node scripts/generate_console_definitions.js --source <PATH_TO_ES_SPECIFICATION_REPO> where PATH_TO_ES_SPECIFICATION_FOLDER is the absolute path to the root of the ES specification repo, for example /Users/yulia/elastic/elasticsearch-specification. This command will generate autocomplete definitions in the folder KIBANA_ROOT/src/plugins/console/server/lib/json/generated.
Optionally --dest parameter can be passed to generate definitions in a different folder, relative to KIBANA_ROOT.

Basic script functionality was implemented in this PR:

Create the folder if doesn't exist yet
Remove all files in the folder before generating definitions
Load the specification schema and parse each endpoint
Create a file for each endpoint with the endpoint name, methods, patterns and doc urls.

Functionality that will be added in follow up PRs:

Url paramaters
Request body parameters
Availability property
Unit test for script functions

How to test

Checkout ES specification repo
Run the command with node scripts/generate_console_definitions.js --source <ES_SPECIFICATION_REPO> --emptyDest where <ES_SPECIFICATION_REPO> is the absolute path to the root of the ES specification repo
Check the changes to the generated files in the folder /KIBANA_REPO/src/plugins/console/server/lib/spec_definitions/json/generated and make sure they have a correct endpoint name, patterns, methods and doc links. We are not generating any url params, request body params or availability property for now.
Change the constant in the file KIBANA_REPO/src/plugins/console/common/constants/autocomplete_definitions.ts to a non-existent folder. Run the script node scripts/generate_console_definitions.js --source <ES_SPECIFICATION_REPO> and check that the folder has been created successfully
Re-run the command without --emptyDest flag targeting a folder that already contain some files. Check that the script fails and doesn't silently remove existing files
Run the help command node scripts/generate_console_definitions.js --help and check if the help message makes sense

…efinitions generation

…fix'

…se it in the new script. Fix the patterns with leading forward slash and doc urls.

…fix'

yuliacech · 2023-06-14T16:51:09Z

I will hold off on merging this until after the 8.9 FF.

…'t exist and to clean the folder if it already contains files

…fix'

yuliacech · 2023-06-15T16:07:43Z

packages/kbn-generate-console-definitions/src/generate_console_definitions.ts

+import fs from 'fs';
+import Path, { join } from 'path';
+import { ToolingLog } from '@kbn/tooling-log';
+


This file is mostly WIP and probably will change a lot in follow up PRs. But still feel free to review as usual

elasticmachine · 2023-06-15T16:08:34Z

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

alisonelizabeth

Nice job @yuliacech 🎉 This looks like a great start. I followed your testing instructions and everything worked as expected. I reviewed a handful of the generated autocomplete definitions and the definitions appeared correct. The only thing I noticed is the script seemed to remove the new line at the end of each file, which previously existed.

alisonelizabeth · 2023-06-15T19:09:39Z

packages/kbn-generate-console-definitions/package.json

+  "private": true,
+  "version": "1.0.0",
+  "license": "SSPL-1.0 OR Elastic License 2.0"
+}


[nit] add new line

packages/kbn-generate-console-definitions/src/cli.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

yuliacech · 2023-06-21T09:44:02Z

Thanks a lot for your review, @alisonelizabeth!
I added back missing new lines (had them disabled in my diffs view) in a90111b

alisonelizabeth

Thanks for addressing my feedback! Reviewed the commit a90111b and lgtm.

kibana-ci · 2023-06-23T19:05:26Z

💚 Build Succeeded

Buildkite Build
Commit: 927cb73

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`console`	204	205	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/generate-console-definitions`	-	1	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`console`	418.5KB	418.5KB	+31.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`console`	27.2KB	27.4KB	+179.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/generate-console-definitions`	-	1	+1

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	14	16	+2
`securitySolution`	416	420	+4
total			+6

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	15	17	+2
`securitySolution`	497	501	+4
total			+6

History

💛 Build #136853 was flaky a90111b
💛 Build #135699 was flaky 7804ccf
💚 Build #135602 succeeded a237de0
💔 Build #135563 failed 2340672
💛 Build #135295 was flaky 1b63e1e
💔 Build #134898 failed 52051aa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

## Summary Fixes #160528 Follow up to #159241 This PR adds logic for query parameters to the new script generating Console autocomplete definitions from ES specification. The logic is explained in details in the [file](https://github.com/elastic/kibana/pull/160515/files#diff-b6853462c38db4e237dbb3cdec9d9f6659aa3fdc5f96a6193f2c4bac1439db43) but here is a short version: - Currently, the autocomplete engine only works with url params of 2 types: boolean (`__flag__`) and a list of options (for example `['all', 'open', 'hidden']`). The script will convert all of the url params from the specification into this format: a boolean or a list. If there are no set options for a parameter, but a default value is known, then it will be converted into a list with 1 option, for example `['random']` so that the autocomplete engine will display it as a single suggestion. We also need to convert any numbers to strings, because they won't be displayed otherwise. - Endpoints in the specification have a property `request` which in turn has 2 properties describing url params: `attachedBehaviours` and `query`. Both object contain an array of `property`'s each describing a url param. `property` is configured with either a built in type (`string`, `number`, `boolean`) or defined type, for example `ExpandWildcards`. By finding the type `ExpandWildcards` in the specification, we can convert this type to a list of options `['open', 'all', 'none', 'hidden']`. ### How to test Similar to #159241, you need to re-generenate the definitions and see if local changes to definitions files make sense. 1. Checkout the ES specification [repo](https://github.com/elastic/elasticsearch-specification) 2. Run the command `node scripts/generate_console_definitions.js --source <ES_SPECIFICATION_REPO> --emptyDest` 3. Check the changes in the folder `KIBANA_REPO/src/plugins/console/server/lib/spec_definitions/json/generated` #### Intended changes to the definitions files - Most of endpoints have 4 default url params that previously were not in the definitions files but added to all endpoints in this [file](https://github.com/elastic/kibana/blob/main/src/plugins/console/public/lib/autocomplete/url_params.js). These params are configured in the interface `CommonQueryParameters` in the specification (see this [file](https://github.com/elastic/elasticsearch-specification/blob/main/specification/_spec_utils/behaviors.ts)). <details> The interface in the specification ```js export interface CommonQueryParameters { error_trace?: boolean filter_path?: string | string[] human?: boolean pretty?: boolean } ``` The converted url params ```json "error_trace": "__flag__", "filter_path": [], "human": "__flag__", "pretty": "__flag__", ``` </details> - Previously existing `url_components` property in the definitions is deleted and this change will be addressed separately. (not sure it is currently working but added a task to the meta issue) - Previously numbers were configured as `0` or `0.0` but that is not currently displayed in suggestions. Instead, the new script converts numbers to strings and if any default value is present, it will be displayed as a suggestion. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…63301) ## Summary This PR uses the new script to generate autocomplete definitions for Dev Tools Console from the ES specification repo. #### Definitions changes - New property `availability` is added to filter out endpoints that are not available in Serverless - Some endpoints' query parameters have more details now, for example common query params are now defined in definitions ```json "url_params": { "error_trace": "__flag__", "filter_path": [], "human": "__flag__", "pretty": "__flag__" }, ``` - Url components in few endpoints are removed, but those were added to overrides files in #163096 - Documentation links contain `{branch}` instead of `master` (fix for that added in #159241) #### Script changes - The logic for generating `availability` for endpoint has been updated based on the feedback from the Clients team. Details added to the script file. - Added a few "safe guards" to the spots in the script where an unexpected type of data might be coming from the ES specification schema #### Console changes - Fixed the autocomplete request on Serverless (we might need a proper fix for that, details in #163318) Also updates to readme files both in Console and the new script. I will remove the old script in a separate PR. ## Screenshots "ILM" autocomplete suggestions displayed on stateful <img width="583" alt="Screenshot 2023-08-07 at 17 47 48" src="https://github.com/elastic/kibana/assets/6585477/641a48b0-fb1a-4d3b-a8c9-99eab8795510"> "ILM" autocomplete suggestions not displayed on serverless <img width="572" alt="Screenshot 2023-08-07 at 17 35 16" src="https://github.com/elastic/kibana/assets/6585477/a1ee5468-eb9f-4f52-81d5-c661b06f8ceb"> ## How to test - Start Kibana on stateful (`yarn start`) and check that autocomplete suggestions are working as before (no changes) - Start Kibana on serverless (`yarn start --serverless`) and check that autocomplete suggestions are not displayed for endpoints blocked/internal on serverless. ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

yuliacech added 2 commits June 7, 2023 12:23

[Console] Create a new package to generate autocomplete definitions

2cc1e5f

[Console] Add Kibana a script command and implementation basics for d…

bd76003

…efinitions generation

yuliacech added Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more release_note:skip Skip the PR/issue when compiling release notes v8.9.0 Feature:Console Dev Tools Console Feature labels Jun 7, 2023

kibanamachine and others added 6 commits June 7, 2023 17:03

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

9b58966

…fix'

[CI] Auto-commit changed files from 'node scripts/generate codeowners'

fdcddb6

[Console] Remove the new folder

e6754ee

[Console] Add options for source and dest folders

52051aa

[Console] Export a constant for autocomplete definitions folder and u…

e0b3c18

…se it in the new script. Fix the patterns with leading forward slash and doc urls.

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

1b63e1e

…fix'

yuliacech added v8.10.0 and removed v8.9.0 labels Jun 14, 2023

yuliacech and others added 4 commits June 15, 2023 13:23

[Console] Add functionality to create the definitions folder if doesn…

751ac76

…'t exist and to clean the folder if it already contains files

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

2340672

…fix'

Merge branch 'main' into console/new_definitions

a237de0

[Console] Add description and usage to the command

7804ccf

yuliacech commented Jun 15, 2023

View reviewed changes

yuliacech marked this pull request as ready for review June 15, 2023 16:08

yuliacech requested a review from a team as a code owner June 15, 2023 16:08

alisonelizabeth reviewed Jun 15, 2023

View reviewed changes

yuliacech and others added 5 commits June 21, 2023 11:08

Update packages/kbn-generate-console-definitions/src/cli.ts

5b66edd

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

Update packages/kbn-generate-console-definitions/src/cli.ts

ad59531

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

Update packages/kbn-generate-console-definitions/src/cli.ts

70045a0

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

Merge branch 'main' into console/new_definitions

ddc250f

[Console] Add back a new line at the end of the definitions files

a90111b

yuliacech force-pushed the console/new_definitions branch from 94e3e07 to a90111b Compare June 21, 2023 09:43

yuliacech mentioned this pull request Jun 21, 2023

[Console] Filter autocomplete suggestions for unavailable endpoints in Serverless #160160

Closed

alisonelizabeth approved these changes Jun 22, 2023

View reviewed changes

yuliacech and others added 2 commits June 23, 2023 20:03

Merge branch 'main' into console/new_definitions

aa781d5

[Console] Fix the new line

927cb73

yuliacech enabled auto-merge (squash) June 23, 2023 18:04

yuliacech merged commit 0a7ee08 into elastic:main Jun 23, 2023

kibanamachine added the backport:skip This commit does not require backporting label Jun 23, 2023

This was referenced Jun 26, 2023

[Console] Implement the logic for query params #160528

Closed

[Console] Implement the logic for body params #160533

Open

[Console] Update the logic for variables #160539

Closed

[Console] Add logic for query params #160515

Merged

yuliacech mentioned this pull request Jul 11, 2023

[Console] Implement logic to generate body params #161684

Closed

yuliacech mentioned this pull request Aug 7, 2023

[Console] Generate autocomplete definitions from ES specification #163301

Merged

8 tasks

yuliacech deleted the console/new_definitions branch February 15, 2024 12:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Console] Use ES specification for autocomplete definitions #159241

[Console] Use ES specification for autocomplete definitions #159241

yuliacech commented Jun 7, 2023 •

edited

Loading

yuliacech commented Jun 14, 2023

yuliacech Jun 15, 2023

elasticmachine commented Jun 15, 2023

alisonelizabeth left a comment •

edited

Loading

alisonelizabeth Jun 15, 2023

yuliacech commented Jun 21, 2023

alisonelizabeth left a comment

kibana-ci commented Jun 23, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

[Console] Use ES specification for autocomplete definitions #159241

[Console] Use ES specification for autocomplete definitions #159241

Conversation

yuliacech commented Jun 7, 2023 • edited Loading

Summary

How to test

yuliacech commented Jun 14, 2023

yuliacech Jun 15, 2023

Choose a reason for hiding this comment

elasticmachine commented Jun 15, 2023

alisonelizabeth left a comment • edited Loading

Choose a reason for hiding this comment

alisonelizabeth Jun 15, 2023

Choose a reason for hiding this comment

yuliacech commented Jun 21, 2023

alisonelizabeth left a comment

Choose a reason for hiding this comment

kibana-ci commented Jun 23, 2023

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

yuliacech commented Jun 7, 2023 •

edited

Loading

alisonelizabeth left a comment •

edited

Loading