Persistent banner to ./bin/elasticsearch on the terminal

[albertzaharovits]

As part of the "Security ON by default" project we ought to do some security auto-configuration when a v8 ES node is started for the first time, and from the terminal (the idea is that ./bin/elasticsearch starts a node with Security enabled without you looking into the yml file at all).

The output of the auto-configuration can be displayed using a yet-to-be-implemented cmd line utility, but also, as a convenience to the user, when the node first starts, as part of the node's usual output (which are the logs). This PR addresses this latter requirement, by showing the auto-configuration information (only a static mock for now, but will be generated) as the last output to the terminal (following the logs), in a persistent, banner-like fashion.
In this way, the information is clearly visible, irrespective of the log output stream, and without breaking any of the log lines.

In order to achieve the cursor movement, the PR uses ANSI VT 100 escape sequences, which should be recognized by all terminal emulators (I have tested with terminal and iTerm2 on MacOS, but I have qualms on Windows).

Here is a short video I made of the ./bin/elasticsearch output.

@elastic/es-core-infra I would appreciate if someone can help us validate this approach (we must also validate on windows, etc). If everything looks reasonable to you, I will proceed to polish this PR to have a static banner, that we will then later make dynamically generated.

CC @jkakavas @BigPandaToo

[dakrone]

As someone that runs a terminal with $TERM=dumb, I would appreciate if this could check the $TERM variable and only attempt terminal escaping if the terminal type supports it. Otherwise I end up with this: https://gist.github.com/dakrone/afbcf7ae866a4731ef1398a222ee82d9

[williamrandolph]

It looks like the skeleton of the approach here can be described as having a small standalone program that will display the contents of a file at the bottom of a console with the standard input to that program scrolling by above it. When the Elasticsearch startup script has certain parameters, the script's standard output is redirected to this standalone program.

On Windows, we invoke the file bin\elasticsearch.bat rather than bin\elasticsearch. In Windows batch, we have a very different syntax for conditionals. I'm not sure if the redirection model supports what we're doing here, and there's no process substitution. It will take some experimenting to see if we can actually get the standalone script correctly hooked up to the output of the Elasticsearch process.

On top of that, automated testing for this kind of interactive-only output is difficult. There's only one place I know of where we try to do it, and in that case we use the TCL expect program. We haven't been able to install this on our Windows servers (#49340).

The most "portable" thing to do would be to implement the banner behavior fully in Java. This would mean that we would have to design and implement a wrapper for all our current log messages and build in some "banner display" capability. (Gradle's output handling seems to be done entirely in Java code.) This seems like it would be a pretty major project, and it's hard to think of any other use cases that would justify it.

I'm still working on finding out what happens when we use the Java part of the banner tool on Windows.

[williamrandolph]

My naive addition to the batch script looks like this:

ECHO.!KEYSTORE_PASSWORD!| %JAVA% %ES_JAVA_OPTS% -Delasticsearch ^
  -Des.path.home="%ES_HOME%" -Des.path.conf="%ES_PATH_CONF%" ^
  -Des.distribution.flavor="%ES_DISTRIBUTION_FLAVOR%" ^
  -Des.distribution.type="%ES_DISTRIBUTION_TYPE%" ^
  -Des.bundled_jdk="%ES_BUNDLED_JDK%" ^
  -cp "%ES_CLASSPATH%" "org.elasticsearch.bootstrap.Elasticsearch" !newparams! ^
  | %JAVA% %XSHARE% -cp "%ES_CLASSPATH%" "org.elasticsearch.tools.launchers.OutputBottomBanner" ^
  "EOF" "bin/banner.txt"

"bin/banner.txt" is a text file with the contents of the banner.

I ran bin/elasticsearch.bat from Powershell and the output looked like this:

So not only are we not deleting and rewriting the banner; we're also garbling our output with the Windows pipe. I'd have to do some research to find out if there's a magic Windows incantation to correctly hook up stdin and stdout for this case.

[albertzaharovits]

Thanks for the input @williamrandolph and @dakrone !

In line with your investigations, I think we should conceptualize of the banner thing as a best effort, helpful thing.
It should be skipped if the attached output is not a terminal (bash -t), or if the terminal, according to terminfo, doesn't support escape sequences. I could also imagine that we end up using different escape sequences (maybe for the Terminal app in Windows). It is also perfectly legit that we don't show any banner.

My plan was to investigate how gradle itself does it, especially how it handles the multiple OS/platforms. But until now I've not been able to import the gradle project in IntelliJ in order to more easily explore its code base.

After we unravel these mysteries, I'm going to update the banner to show the HTTPS cert fingerprint, add tests, polish and raise a proper non-draft PR.

[elasticsearchmachine]

Hi @albertzaharovits, I've created a changelog YAML for you.

[albertzaharovits]

We discussed this in many, many forums.
This is a brief and rough summary in order to send this to rest for now.

The algorithm to display a banner is to output the banner contents after every line or couple of lines of regular output, and then, before outputing further lines of the regular output, use ANSI escape sequences to clear up the banner, and so forth for the banner lifetime duration.

The regular output of the ES process are the logs produced by log4j2. Consequently, in order to implement the algorithm just described, we would need to intercept the lines that log4j2 produces to its print stream, detect line boundaries, wait for pauses in the log line flow and insert the banner, and then clearing it up when log lines start comming in again, and so on.

So we need to hijack log4j2 print stream, not ideal, and I suspect it will entail many discussions, if technically possible.
RIP