[ES|QL] AST traversal API

[drewdaemon]

Finding things in an AST (and possibly changing them) requires a tree-traversal algorithm. Developers working with ES|QL queries shouldn't have to worry about implementing this themselves. Instead, we should provide a nice API.

Ideas

const nodes = findAll(ast, 'function', ['==', '!=']);
...

Or maybe

const matcher = (node, ctx) => ctx.depth == 1
  && isFunctionNode(node) 
  && ['==', '!='].includes(node.name);

const nodes = findAll(ast, matcher);

...

Or for manipulation, something like

const negateComparisons = (ast) => {
  walk(ast, (node) => {
    if (isFunctionNode(node) && node.name === '==') {
      node.name = '!=';
    }
  });
}
...

Could also provide typed shortcut methods like

const walker = new Walker(ast);
const functions = walker.findAllFunctions();
const constants = walker.findAllConstants();
const source = walker.findSource()

Each of those could return typed AST nodes, removing the need for consumers to run their own type checks.

Or we could implement a visitor like recast's

visit(ast, {
  onCommand: (node, ctx) => {
    messages.push(...validateCommand(node, ctx));
  },
  onFunction: ...
});

Some comparables

• Acorn AST walker
• JSCodeshift
• AST Explorer (play with lots of different JS ASTs and code mod technologies)

Some potential adoption points

• [ES|QL] Recognize transformational commands with ast parsing #184291
• [ES|QL] [Discover] Creating where clause filters from the table, sidebar and table row viewer #181399
• Validation engine

• kibana/packages/kbn-esql-ast/src/ast_helpers.ts

  Lines 180 to 201 in b3a12c1

  	function walkFunctionStructure(
  	args: ESQLAstItem[],
  	initialLocation: ESQLLocation,
  	prop: 'min' | 'max',
  	getNextItemIndex: (arg: ESQLAstItem[]) => number
  	) {
  	let nextArg: ESQLAstItem | undefined = args[getNextItemIndex(args)];
  	const location = { ...initialLocation };
  	while (Array.isArray(nextArg) || nextArg) {
  	if (Array.isArray(nextArg)) {
  	nextArg = nextArg[getNextItemIndex(nextArg)];
  	} else {
  	location[prop] = Math[prop](location[prop], nextArg.location[prop]);
  	if (nextArg.type === 'function') {
  	nextArg = nextArg.args[getNextItemIndex(nextArg.args)];
  	} else {
  	nextArg = undefined;
  	}
  	}
  	}
  	return location[prop];
  	}

[elasticmachine]

Pinging @elastic/kibana-esql (Team:ESQL)

[drewdaemon]

It would be great to use these walker APIs in our own validation engine as well.

[mattkime]

I agree that this looks very useful, but I think its something that might live inside a more opinionated service that takes the whole ESQL string and provides getters and mutators.

[drewdaemon]

@mattkime I think I get what you're saying, but I think a couple examples of how you would use the opinionated service to accomplish a given task would be helpful here. One recent task we had was adding a new where clause if it doesn't exist, or edit it if it does. How would you accomplish this with the service you're envisioning? What would the getters and mutators look like?

[vadimkibana]

There is an initial implementation of the Walker now available, it is missing:

• Traversal of all node types
  • For example, option nodes
• The context object

[stratoula]

@vadimkibana also a Readme (documentation) when this is more mature would be very nice too.

[drewdaemon]

We may also want to consider giving the user more control over the traversal. Recast does this with this.traverse(), this.abort(), and return false;

[vadimkibana]

Acorn AST walker for inspiration: https://github.com/acornjs/acorn/tree/master/acorn-walk