Add Kibana Dev Container (elastic#188887)

## Summary
- Closes elastic/kibana-operations#101

This PR adds a [Dev Container](https://containers.dev/) to utilize for
Kibana local development in an isolated environment. The original
intention was to create a local environment for FIPS development because
setting up Kibana in FIPS mode is complicated and has the potential to
break the user's OS. However, it has been altered to allow for general
development if an engineer chooses as well. The idea is for this be a
cost efficient replacement for
[kibana-remote-dev](https://github.com/elastic/kibana-remote-dev)
eventually.

### Testing
- In VS Code you should be able to use the `Dev Containers: Clone GitHub
Pull Request in Named Container Volume...` command from the Command
Palette (F1) to easily test this PR.
- See the
[docs](https://github.com/elastic/kibana/blob/bd125fc230f06848d62a564dc88d8abfb4f13e88/dev_docs/getting_started/setting_up_a_development_env.mdx#using-the-kibana-dev-container-optional)
for additional information on setting up the Dev Container.


### Bazel
I tried many different solutions to copy the local Bazel cache into the
container to speed up bootstrap, but it either would break Bazel or
didn't provide any meaningful boost in performance. I opted to forgo
keeping it in this PR due to the complexity and since we're planning to
phase out Bazel in the future anyways.

.devcontainer/.env.template

@@ -0,0 +1,4 @@
+# /bin/bash or /bin/zsh (oh-my-zsh is installed by default as well)
+SHELL=/bin/bash
+# Switch to 1 to enable FIPS environment, any other value to disable
+FIPS=0

.devcontainer/Dockerfile

@@ -0,0 +1,69 @@
+FROM mcr.microsoft.com/devcontainers/base:ubuntu-22.04
+
+ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8
+ENV HOME=/home/vscode
+ENV NVM_DIR=${HOME}/nvm
+ENV NVM_VERSION=v0.39.1
+ENV KBN_DIR=/workspaces/kibana
+ENV OPENSSL_PATH=${HOME}/openssl
+# Only specific versions are FIPS certified.
+ENV OPENSSL_VERSION='3.0.8'
+
+RUN apt-get update && apt-get install -y curl git zsh locales docker.io perl make gcc xvfb
+
+RUN locale-gen en_US.UTF-8
+
+# Oh My Zsh setup
+RUN if [ ! -d "$HOME/.oh-my-zsh" ]; then \
+  sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"; \
+  fi && \
+  ZSH_CUSTOM=${ZSH_CUSTOM:-~/.oh-my-zsh/custom} && \
+  if [ ! -d "$ZSH_CUSTOM/plugins/zsh-autosuggestions" ]; then \
+  git clone https://github.com/zsh-users/zsh-autosuggestions $ZSH_CUSTOM/plugins/zsh-autosuggestions; \
+  fi && \
+  sed -i 's/plugins=(git)/plugins=(git ssh-agent npm docker zsh-autosuggestions)/' /home/vscode/.zshrc
+
+# Docker-in-Docker setup
+RUN usermod -aG docker vscode
+
+# FIPS setup
+# https://github.com/openssl/openssl/blob/openssl-3.0/README-FIPS.md
+# https://www.openssl.org/docs/man3.0/man7/fips_module.html
+WORKDIR ${HOME}
+
+RUN set -e ; \
+  mkdir -p "${OPENSSL_PATH}"; \
+  curl --retry 8 -S -L -O "https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz" ; \
+  curl --retry 8 -S -L -O "https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz.sha256" ; \
+  echo "$(cat openssl-${OPENSSL_VERSION}.tar.gz.sha256) openssl-${OPENSSL_VERSION}.tar.gz" | sha256sum -c ; \
+  tar -zxf "openssl-${OPENSSL_VERSION}.tar.gz" ; \
+  rm -rf openssl-${OPENSSL_VERSION}.tar* ; \
+  cd "${OPENSSL_PATH}-${OPENSSL_VERSION}" ; \
+  ./Configure --prefix="${OPENSSL_PATH}" --openssldir="${OPENSSL_PATH}/ssl" --libdir="${OPENSSL_PATH}/lib" shared -Wl,-rpath,${OPENSSL_PATH}/lib enable-fips; \
+  make -j $(nproc) > /dev/null ; \
+  make install > /dev/null ; \
+  rm -rf  "${OPENSSL_PATH}-${OPENSSL_VERSION}" ; \
+  chown -R 1000:1000 "${OPENSSL_PATH}";
+
+WORKDIR ${KBN_DIR}
+
+# Node and NVM setup
+COPY .node-version /tmp/
+RUN mkdir -p $NVM_DIR && \
+  curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/${NVM_VERSION}/install.sh | bash && \
+  . "$NVM_DIR/nvm.sh" && \
+  NODE_VERSION=$(cat /tmp/.node-version) && \
+  nvm install ${NODE_VERSION} && \
+  nvm use ${NODE_VERSION} && \
+  nvm alias default ${NODE_VERSION} && \
+  npm install -g yarn && \
+  echo "source $NVM_DIR/nvm.sh" >> ${HOME}/.bashrc && \
+  echo "source $NVM_DIR/nvm.sh" >> ${HOME}/.zshrc  && \
+  chown -R 1000:1000 "${HOME}/.npm"
+
+# Reload the env everytime a new shell is opened incase the .env file changed.
+RUN echo "source $KBN_DIR/.devcontainer/scripts/env.sh" >> ${HOME}/.bashrc && \
+  echo "source $KBN_DIR/.devcontainer/scripts/env.sh" >> ${HOME}/.zshrc
+
+# This is for documentation. Ports are exposed via devcontainer.json
+EXPOSE 9200 5601 9229 9230 9231

.devcontainer/README.md

@@ -0,0 +1 @@
+See the [dev docs](https://github.com/elastic/kibana/blob/main/dev_docs/getting_started/setting_up_a_development_env.mdx#using-the-kibana-dev-container-optional) for information on using the Kibana Dev Container.

.devcontainer/config/nodejs.cnf

@@ -0,0 +1,28 @@
+##########################################################################
+##                                                                      ##
+## This OpenSSL config is only loaded when running Kibana in FIPS mode. ##
+##                                                                      ##
+## See:                                                                 ##
+## https://github.com/openssl/openssl/blob/openssl-3.0/README-FIPS.md   ##
+## https://www.openssl.org/docs/man3.0/man7/fips_module.html            ##
+##                                                                      ##
+##########################################################################
+
+nodejs_conf = nodejs_init
+.include /home/vscode/openssl/ssl/fipsmodule.cnf
+
+[nodejs_init]
+providers = provider_sect
+alg_section = algorithm_sect
+
+[provider_sect]
+default = default_sect
+# The fips section name should match the section name inside the
+# included fipsmodule.cnf.
+fips = fips_sect
+
+[default_sect]
+activate = 1
+
+[algorithm_sect]
+default_properties = fips=yes 

.devcontainer/devcontainer.json

@@ -0,0 +1,41 @@
+{
+  "name": "Kibana",
+  "build": {
+    "dockerfile": "Dockerfile",
+    "context": ".."
+  },
+  "customizations": {
+    "vscode": {
+      "extensions": [
+        "dbaeumer.vscode-eslint",
+        "ms-azuretools.vscode-docker",
+        "editorconfig.editorconfig",
+        "timonwong.shellcheck",
+        "eamodio.gitlens",
+        "github.vscode-pull-request-github"
+      ]
+    }
+  },
+  "forwardPorts": [
+    9200,
+    5601,
+    9229,
+    9230,
+    9231
+  ],
+  "postStartCommand": "/workspaces/kibana/.devcontainer/scripts/post_start.sh",
+  "remoteUser": "vscode",
+  "features": {
+    "ghcr.io/devcontainers/features/docker-in-docker:2": {
+      "version": "latest",
+      "dockerDashComposeVersion": "latest"
+    },
+    "ghcr.io/devcontainers/features/github-cli:1": {
+      "installDirectlyFromGitHubRelease": true,
+      "version": "latest"
+    },
+    "ghcr.io/kreemer/features/chrometesting:1": {
+      "version": "stable"
+    }
+  }
+}

.devcontainer/scripts/env.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+ENV_PATH="${KBN_DIR}/.devcontainer/.env"
+KBN_CONFIG_FILE="${KBN_DIR}/config/kibana.dev.yml"
+
+setup_fips() {
+  if [ ! -f "$KBN_CONFIG_FILE" ]; then
+    touch "$KBN_CONFIG_FILE"
+  fi
+
+  if [ -n "$FIPS" ] && [ "$FIPS" = "1" ]; then
+    sed -i '/xpack.security.experimental.fipsMode.enabled:/ {s/.*/xpack.security.experimental.fipsMode.enabled: true/; t}; $a\xpack.security.experimental.fipsMode.enabled: true' "$KBN_CONFIG_FILE"
+
+    # Patch node_modules so we can start Kibana in dev mode
+    sed -i 's/hashType = hashType || '\''md5'\'';/hashType = hashType || '\''sha1'\'';/g' "${KBN_DIR}/node_modules/file-loader/node_modules/loader-utils/lib/getHashDigest.js"
+    sed -i 's/const hash = createHash("md4");/const hash = createHash("sha1");/g' "${KBN_DIR}/node_modules/webpack/lib/ModuleFilenameHelpers.js"
+    sed -i 's/contentHash: createHash("md4")/contentHash: createHash("sha1")/g' "${KBN_DIR}/node_modules/webpack/lib/SourceMapDevToolPlugin.js"
+
+    export OPENSSL_MODULES="$OPENSSL_PATH/lib/ossl-modules"
+    export NODE_OPTIONS="--enable-fips --openssl-config=$KBN_DIR/.devcontainer/config/nodejs.cnf"
+    echo "FIPS mode enabled"
+    echo "If manually bootstrapping in FIPS mode use: NODE_OPTIONS='' yarn kbn bootstrap"
+  else
+    sed -i '/xpack.security.experimental.fipsMode.enabled:/ {s/.*/xpack.security.experimental.fipsMode.enabled: false/; t}; $a\xpack.security.experimental.fipsMode.enabled: false' "$KBN_CONFIG_FILE"
+  fi
+}
+
+setup_shell() {
+  if [ -n "$SHELL" ] && [ -x "$SHELL" ]; then
+    current_shell=$(ps -p $$ -o comm=)
+    desired_shell=$(basename "$SHELL")
+
+    if [ "$current_shell" != "$desired_shell" ]; then
+      sudo chsh -s "$SHELL" vscode
+      exec "$SHELL"
+    fi
+  else
+    echo "Shell is not set or not executable, using bash"
+  fi
+}
+
+if [ -f "$ENV_PATH" ]; then
+  source "$ENV_PATH"
+  setup_fips
+  setup_shell
+else
+  echo ".env file not found, using default values"
+fi

.devcontainer/scripts/post_start.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+# If FIPS mode is enabled, there can be issues installing some dependencies due to invalid algorithms.
+# So override the NODE_OPTIONS environment variable to disable FIPS mode.
+NODE_OPTIONS='' yarn kbn bootstrap
+
+Xvfb :99 -screen 0 1920x1080x24 &
+export DISPLAY=:99

.github/CODEOWNERS

@@ -1279,6 +1279,7 @@ x-pack/test/observability_ai_assistant_functional @elastic/obs-ai-assistant
 /kbn_pm/ @elastic/kibana-operations
 /x-pack/dev-tools @elastic/kibana-operations
 /catalog-info.yaml @elastic/kibana-operations @elastic/kibana-tech-leads
+/.devcontainer/ @elastic/kibana-operations
 
 # Appex QA
 /src/dev/code_coverage @elastic/appex-qa

.gitignore

@@ -142,10 +142,13 @@ x-pack/test/security_api_integration/plugins/audit_log/audit.log
 .ftr
 role_users.json
 
+
+.devcontainer/.env
+
 # Ignore temporary files in oas_docs
 output/kibana.serverless.tmp1.yaml
 output/kibana.serverless.tmp2.yaml
 output/kibana.tmp1.yaml
 output/kibana.tmp2.yaml
 output/kibana.new.yaml
-output/kibana.serverless.new.yaml
+output/kibana.serverless.new.yaml

dev_docs/getting_started/setting_up_a_development_env.mdx

@@ -3,8 +3,8 @@ id: kibDevTutorialSetupDevEnv
 slug: /kibana-dev-docs/getting-started/setup-dev-env
 title: Set up a Development Environment
 description: Learn how to setup a development environment for contributing to the Kibana repository
-date: 2022-07-07
-tags: ['kibana', 'onboarding', 'dev', 'architecture', 'setup']
+date: 2024-08-09
+tags: ['kibana', 'onboarding', 'dev', 'architecture', 'setup', 'devcontainer']
 ---
 
 Setting up a development environment is pretty easy.
@@ -92,3 +92,28 @@ node scripts/register_git_hook
 ```
 
 After the script completes the pre-commit hook will be created within the file `.git/hooks/pre-commit`. If you choose to not install it, don’t worry, we still run a quick CI check to provide feedback earliest as we can about the same checks.
+
+## Using the Kibana Dev Container (optional)
+
+Kibana also supports using a [dev container](https://containers.dev/) which can integrate with various editors and tools [(supported tools)](https://containers.dev/supporting). The dev container provides a consistent development environment across different machines and setups which is based on Ubuntu Jammy (22.04). The only prerequisite is having [Docker](https://www.docker.com/) installed locally. VS Code is the recommended editor and will be used for these instructions because it is the most mature, but it is not required.
+
+### Setting up the Dev Container
+
+1. Make a copy of `.devcontainer/.env.template` and rename it to `.devcontainer/.env`. Edit any values you're interested in.
+1. There are three options for mounting the Kibana repo into the container:
+    - **Local Filesystem**: Clone the repo locally, or use an existing copy, and open it in VS Code. When prompted, select "Reopen in Dev Container". This uses a bind mount, allowing the container to access and modify files directly on your local filesystem. Your git credentials should be automatically mounted in the container as well. Note that Bazel will create symlinks and a cache inside the container file system. So, if switching to working on your local filesystem afterwards, you will need to bootstrap again.
+    - **Docker Repo Volume**: Use the `Dev Containers: Clone Repository in Named Container Volume...` command from the Command Palette (`F1`). This clones the repo into a Docker volume, isolating it from your local filesystem. You will need to configure your git credentials manually in this isolated environment.
+    - **Docker PR Volume**: Use the `Dev Containers: Clone GitHub Pull Request in Named Container Volume...` command from the Command Palette (`F1`). This is the same as the previous option, but can be useful for testing a PR in insolation of your local filesystem.
+1. VS Code will then build the container, this will take a few minutes the first time, but subsequent builds will utilize Docker caching and be much faster.
+1. Once the container is built and started, it will automatically run `yarn kbn bootstrap`.
+1. You should see the Kibana repo and your terminal will be inside the container. You can develop as normal now, including running `yarn es` from inside the container.
+
+### Customizing the Dev Container
+Installing any extra extensions or making adjustments to the OS environment inside the container will not have an effect on your local OS or VS Code installation. Editing the `devcontainer.json` or `.devcontainer/Dockerfile` should be reserved for changes to all dev environments.
+
+### FIPS Mode
+
+The dev container is pre-configured to run Kibana in FIPS mode if needed. Simply change the `.env` file to `FIPS=1` and reopen your terminal. There should be a log message in your terminal which indicates `FIPS mode enabled`.
+
+### Troubleshooting
+- Sometimes when rebuilding the container, there will be an error message that it failed. Usually hitting retry will fix this, and is only related to VS Code trying to reconnect to the container too quickly.

renovate.json

@@ -2,7 +2,7 @@
   "$schema": "https://docs.renovatebot.com/renovate-schema.json",
   "extends": ["config:recommended", "helpers:pinGitHubActionDigests", "helpers:pinGitHubActionDigestsToSemver"],
   "ignorePaths": ["**/__fixtures__/**", "**/fixtures/**"],
-  "enabledManagers": ["npm", "github-actions", "custom.regex"],
+  "enabledManagers": ["npm", "github-actions", "custom.regex", "devcontainer"],
   "baseBranches": ["main", "7.17"],
   "prConcurrentLimit": 0,
   "prHourlyLimit": 0,
@@ -20,6 +20,14 @@
       "matchDepPatterns": [".*"],
       "enabled": false
     },
+    {
+      "groupName": "devcontainer",
+      "reviewers": ["team:kibana-operations"],
+      "matchBaseBranches": ["main"],
+      "labels": ["Team:Operations", "release_note:skip", "backport:current-major"],
+      "enabled": true,
+      "matchManagers": ["devcontainer"]
+    },
     {
       "groupName": "chainguard",
       "matchPackageNames": ["docker.elastic.co/wolfi/chainguard-base"],