Merge zero-config-dev branch (#3791)

* [DNM] Zero-config C code to inject environment variables for Java and Node.js (#3514) * Remove sending a metric as part of auto-instrumentation (#3482) * refactor the code to inject allowed env vars * latest after pair programming * latest after review and pairing * only inject env vars for java and nodejs programs. Apply env vars from different files for java and nodejs * remove docker interactive and tty flags * wip * use arch var to set the name of the file to copy * make tests pass on arm64 * try to support arm * build for the proper platform * fix package build * review with Jason, test for env vars overrides * Make ALLOWED_ENV_VARS a single string defined through the preprocessor * Update instrumentation/tests/java/Dockerfile Co-authored-by: Jeff Cheng <83052155+jeffreyc-splunk@users.noreply.github.com> * load preload file same way as java test * move file locations to macros * use n functions to handle strings when possible --------- Co-authored-by: Jeff Cheng <83052155+jeffreyc-splunk@users.noreply.github.com> * Add systemd sample file to splunk-otel-auto-instrumention (#3506) * Add systemd sample file to splunk-otel-auto-instrumention * Update manual instructions * Include default /etc/splunk/zeroconfig_java.conf in deb/rpm packages * Update tests for new libsplunk.so * Run auto instrumentation workflow for the zero-config-dev branch * Include splunk-otel-js in auto instrumentation deb/rpm (#3540) * Add systemd sample file to splunk-otel-auto-instrumention * Update tests for new libsplunk.so * Include splunk-otel-js in auto instrumentation deb/rpm * Include default /etc/splunk/zeroconfig_node.conf in deb/rpm packages * Add tests for express instrumentation (#3566) * Update installer script for systemd auto instrumentation (#3536) * Rename zeroconfig config files (#3682) * Update ZC docs and tests for manual installation/configuration (#3700) * Update docs * Update splunk-otel-js to v2.4.2 * Combine sample systemd config files * Install splunk-otel-js globally for tests * Update deps installation for node tests * Update splunk-otel-js to v2.4.4 * Update linux-manual.md * Update test for centos/oraclelinux 7 arm64 --------- Co-authored-by: Anna U <104845867+aurbiztondo-splunk@users.noreply.github.com> * Update otlp endpoint for linux installer script (#3761) * Update otlp endpoint for linux installer script * remove 0.0.0.0 * Add debian bookworm for instrumentation tests * Update CHANGELOG.md --------- Co-authored-by: Antoine Toulme <antoine@lunar-ocean.com> Co-authored-by: Anna U <104845867+aurbiztondo-splunk@users.noreply.github.com>
signalfx · Oct 18, 2023 · b6e72c1 · b6e72c1
1 parent 5258139
commit b6e72c1
Show file tree

Hide file tree

Showing 81 changed files with 2,254 additions and 2,893 deletions.
diff --git a/.github/workflows/auto-instrumentation.yml b/.github/workflows/auto-instrumentation.yml
@@ -8,43 +8,51 @@ on:
     paths:
       - '.github/workflows/auto-instrumentation.yml'
       - 'instrumentation/**'
+      - 'internal/buildscripts/packaging/tests/helpers/**'
+      - 'internal/buildscripts/packaging/tests/instrumentation/**'
+      - 'internal/buildscripts/packaging/tests/requirements.txt'
+      - '!**.md'
 
 concurrency:
   group: auto-instrumentation-${{ github.event.pull_request.number || github.ref }}
   cancel-in-progress: true
 
+env:
+  PYTHON_VERSION: '3.11'
+  PIP_VERSION: '22.0.4'
+  REQUIREMENTS_PATH: "internal/buildscripts/packaging/tests/requirements.txt"
+  GO_VERSION: 1.20.10
+
 jobs:
-  libsplunk:
+  cross-compile:
     # Use 20.04.5 until https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/16450 is resolved
     runs-on: ubuntu-20.04
     strategy:
       matrix:
-        ARCH: [ "amd64", "arm64" ]
+        SYS_BINARIES: [ "binaries-linux_amd64", "binaries-linux_arm64" ]
     steps:
       - name: Check out the codebase.
         uses: actions/checkout@v4
 
-      - name: Set up QEMU
-        if: ${{ matrix.ARCH == 'arm64' }}
-        uses: docker/setup-qemu-action@v3
+      - name: Set up Go
+        uses: actions/setup-go@v4
         with:
-          platforms: arm64
-          image: tonistiigi/binfmt:qemu-v7.0.0
+          go-version: ${{ env.GO_VERSION }}
+          cache-dependency-path: '**/go.sum'
 
-      - name: Build libsplunk.so
-        run: make -C instrumentation dist ARCH=${{ matrix.ARCH }}
+      - name: Build Collector
+        run: |
+          make ${{ matrix.SYS_BINARIES }}
 
       - name: Upload artifact
         uses: actions/upload-artifact@v3
         with:
-          name: libsplunk-${{ matrix.ARCH }}
-          path: ./instrumentation/dist/libsplunk_${{ matrix.ARCH }}.so
+          name: ${{ matrix.SYS_BINARIES }}
+          path: |
+            ./bin/*
 
   build-package:
-    name: build-package
-    # Use 20.04.5 until https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/16450 is resolved
     runs-on: ubuntu-20.04
-    needs: [libsplunk]
     strategy:
       matrix:
         SYS_PACKAGE: [ "deb", "rpm" ]
@@ -55,86 +63,124 @@ jobs:
         with:
           fetch-depth: 0
 
-      - name: Downloading libsplunk-${{ matrix.ARCH }}
-        uses: actions/download-artifact@v3
+      - uses: docker/setup-qemu-action@v2
+        if: ${{ matrix.ARCH != 'amd64' }}
         with:
-          name: libsplunk-${{ matrix.ARCH }}
-          path: ./instrumentation/dist
+          platforms: ${{ matrix.ARCH }}
+          image: tonistiigi/binfmt:qemu-v7.0.0
 
-      - name: Build ${{ matrix.SYS_PACKAGE }} ${{ matrix.ARCH }} package
-        run: |
-          gitRef=(${GITHUB_REF//// })
-          if [[ "${gitRef[1]}" = "tags" ]] ;
-          then
-            echo "TAG_NAME=${gitRef[2]}" >> $GITHUB_ENV
-          else
-            echo "TAG_NAME=" >> $GITHUB_ENV
-          fi
-          make -C instrumentation ${{ matrix.SYS_PACKAGE }}-package SKIP_COMPILE=true VERSION="${TAG_NAME:-}" ARCH="${{ matrix.ARCH }}"
+      - name: Run tests
+        run: make -C instrumentation tests ARCH=${{ matrix.ARCH }}
+
+      - name: Build ${{ matrix.ARCH }} ${{ matrix.SYS_PACKAGE }} package
+        run: make -C instrumentation/ ${{ matrix.SYS_PACKAGE }}-package ARCH="${{ matrix.ARCH }}"
 
       - name: Upload artifact
         uses: actions/upload-artifact@v3
         with:
           name: splunk-otel-auto-instrumentation-${{ matrix.ARCH }}-${{ matrix.SYS_PACKAGE }}
-          path: ./instrumentation/dist/splunk-otel-auto-instrumentation*.${{ matrix.SYS_PACKAGE }}
+          path: ./instrumentation/dist/*.${{ matrix.SYS_PACKAGE }}
 
-  test-deb-package:
-    name: test-deb-package
-    # Use 20.04.5 until https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/16450 is resolved
+  test-package-matrix:
     runs-on: ubuntu-20.04
-    needs: [build-package]
-    strategy:
-      matrix:
-        DISTRO: [ "debian:8", "debian:9", "debian:10", "ubuntu:16.04", "ubuntu:18.04", "ubuntu:20.04" ]
-        ARCH: [ "amd64", "arm64" ]
-        exclude:
-          - DISTRO: debian:8
-            ARCH: arm64
     steps:
-      - name: Check out the codebase.
+      - name: Check out code
         uses: actions/checkout@v4
 
-      - name: Downloading splunk-otel-auto-instrumentation-${{ matrix.ARCH }}-deb
-        uses: actions/download-artifact@v3
-        with:
-          name: splunk-otel-auto-instrumentation-${{ matrix.ARCH }}-deb
-          path: ./instrumentation/dist
-
-      - name: Set up QEMU
-        if: ${{ matrix.ARCH == 'arm64' }}
-        uses: docker/setup-qemu-action@v3
-        with:
-          platforms: arm64
-          image: tonistiigi/binfmt:qemu-v7.0.0
-
-      - name: Test
-        run: docker run --platform linux/${{ matrix.ARCH }} --rm -v $(pwd):/repo -w /repo ${{ matrix.DISTRO }} /repo/instrumentation/packaging/fpm/test.sh deb ${{ matrix.ARCH }}
-
-  test-rpm-package:
-    name: test-rpm-package
+      - name: Get matrix
+        id: get-matrix
+        run: |
+          # create test matrix for distro and arch
+          dockerfiles=$(find internal/buildscripts/packaging/tests/instrumentation/images/ -name "Dockerfile.*" | cut -d '.' -f2- | sort -u)
+          if [ -z "$dockerfiles" ]; then
+            echo "Failed to get dockerfiles from internal/buildscripts/packaging/tests/instrumentation/images!" >&2
+            exit 1
+          fi
+          distro=$(for d in $dockerfiles; do echo -n "\"$d\","; done)
+          arch="\"amd64\", \"arm64\""
+          matrix="{\"DISTRO\": [${distro%,}], \"ARCH\": [${arch}]}"
+          echo "$matrix" | jq
+          echo "matrix=${matrix}" >> $GITHUB_OUTPUT
+    outputs:
+      matrix: ${{ steps.get-matrix.outputs.matrix }}
+
+  test-package:
     # Use 20.04.5 until https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/16450 is resolved
-    runs-on: ubuntu-20.04
-    needs: [build-package]
+    runs-on: ${{ fromJSON('["ubuntu-20.04", "ubuntu-22.04"]')[matrix.DISTRO == 'amazonlinux-2023'] }}
+    timeout-minutes: 60
+    needs: [cross-compile, build-package, test-package-matrix]
     strategy:
-      matrix:
-        DISTRO: [ "centos:7", "centos:8", "amazonlinux:2", "opensuse/leap:42", "opensuse/leap:15", "oraclelinux:7", "oraclelinux:8" ]
-        ARCH: [ "amd64", "arm64" ]
+      matrix: ${{ fromJSON(needs.test-package-matrix.outputs.matrix) }}
+      fail-fast: false
     steps:
       - name: Check out the codebase.
         uses: actions/checkout@v4
 
-      - name: Downloading splunk-otel-auto-instrumentation-${{ matrix.ARCH }}-rpm
-        uses: actions/download-artifact@v3
+      - name: Get package type for ${{ matrix.DISTRO }}
+        run: |
+          for pkg in "deb" "rpm"; do
+            if [[ -f "internal/buildscripts/packaging/tests/instrumentation/images/${pkg}/Dockerfile.${{ matrix.DISTRO }}" ]]; then
+              echo "SYS_PACKAGE=${pkg}" >> $GITHUB_ENV
+              exit 0
+            fi
+          done
+          echo "Unknown distro '${{ matrix.DISTRO }}'!"
+          exit 1
+
+      - uses: actions/download-artifact@v3
+        with:
+          name: binaries-linux_${{ matrix.ARCH }}
+          path: ./bin
+
+      - uses: actions/download-artifact@v3
         with:
-          name: splunk-otel-auto-instrumentation-${{ matrix.ARCH }}-rpm
+          name: splunk-otel-auto-instrumentation-${{ matrix.ARCH }}-${{ env.SYS_PACKAGE }}
           path: ./instrumentation/dist
 
-      - name: Set up QEMU
-        if: ${{ matrix.ARCH == 'arm64'}}
-        uses: docker/setup-qemu-action@v3
+      - uses: docker/setup-qemu-action@v3
+        if: ${{ matrix.ARCH != 'amd64' }}
         with:
-          platforms: arm64
+          platforms: ${{ matrix.ARCH }}
           image: tonistiigi/binfmt:qemu-v7.0.0
 
-      - name: Test
-        run: docker run --platform linux/${{ matrix.ARCH }} --rm -v $(pwd):/repo -w /repo ${{ matrix.DISTRO }} /repo/instrumentation/packaging/fpm/test.sh rpm ${{ matrix.ARCH }}
+      - name: Setup python
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+          cache: 'pip'
+          cache-dependency-path: ${{ env.REQUIREMENTS_PATH }}
+
+      - name: Install pytest
+        run: |
+          if which pip; then
+              pip install --upgrade 'pip==${{ env.PIP_VERSION }}'
+          else
+              curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
+              python get-pip.py 'pip==${{ env.PIP_VERSION }}'
+          fi
+          pip install -r "${{ env.REQUIREMENTS_PATH }}"
+
+      - name: Test ${{ env.SYS_PACKAGE }} on ${{ matrix.DISTRO }} ${{ matrix.ARCH }}
+        id: pytest
+        continue-on-error: true
+        run: |
+          distro="${{ matrix.DISTRO }}"
+          if [[ "$distro" = "amazonlinux-2" ]]; then
+            # workaround for pytest substring matching
+            distro="amazonlinux-2 and not amazonlinux-2023"
+          fi
+          python3 -u -m pytest -s --verbose -k "$distro and ${{ matrix.ARCH }}" \
+            internal/buildscripts/packaging/tests/instrumentation/instrumentation_test.py
+
+      # qemu, networking, running systemd in containers, etc., can be flaky
+      - name: Re-run failed tests
+        if: ${{ steps.pytest.outcome == 'failure' }}
+        run: |
+          distro="${{ matrix.DISTRO }}"
+          if [[ "$distro" = "amazonlinux-2" ]]; then
+            # workaround for pytest substring matching
+            distro="amazonlinux-2 and not amazonlinux-2023"
+          fi
+          python3 -u -m pytest -s --verbose -k "$distro and ${{ matrix.ARCH }}" \
+            --last-failed \
+            internal/buildscripts/packaging/tests/instrumentation/instrumentation_test.py
diff --git a/.gitignore b/.gitignore
@@ -45,4 +45,5 @@ deployments/heroku/test/node_modules
 /instrumentation/dist/
 /instrumentation/obj/
 /instrumentation/so/
-/instrumentation/tests
+/instrumentation/tests/java/libsplunk.so
+/instrumentation/tests/nodejs/libsplunk.so
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -157,6 +157,16 @@ update-javaagent:
     - .gitlab/install-gh-cli.sh
     - .gitlab/update-javaagent.sh
 
+update-nodejs-agent:
+  only:
+    - schedules
+  stage: update-deps
+  needs: []
+  dependencies: []
+  script:
+    - .gitlab/install-gh-cli.sh
+    - .gitlab/update-nodejs-agent.sh
+
 tidy-dependabot-pr:
   rules:
     - if: $CI_COMMIT_BRANCH =~ /^dependabot\/go_modules\/.*/ && $CI_COMMIT_AUTHOR =~ /^dependabot.*/

diff --git a/.gitlab/update-nodejs-agent.sh b/.gitlab/update-nodejs-agent.sh
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# NOTE: this script is meant to be run on the GitLab CI, it depends on GitLab CI variables
+# Based on https://github.com/signalfx/splunk-otel-java/blob/c9134906c84e9a32a974dec4b380453fe1757410/scripts/propagate-version.sh
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+# shellcheck source-path=SCRIPTDIR
+source "${SCRIPT_DIR}/common.sh"
+
+ROOT_DIR="${SCRIPT_DIR}/../"
+cd "${ROOT_DIR}"
+
+create_collector_pr() {
+  local repo="signalfx/splunk-otel-collector"
+  local repo_url="https://srv-gh-o11y-gdi:${GITHUB_TOKEN}@github.com/${repo}.git"
+  local branch="create-pull-request/update-nodejs-agent"
+  local message="Update splunk-otel-js to latest"
+
+  echo ">>> Cloning the $repo repository ..."
+  git clone "$repo_url" collector-mirror
+  cd collector-mirror
+
+  setup_branch "$branch" "$repo_url"
+
+  echo ">>> Getting latest splunk-otel-js release ..."
+  tag="$( gh release view --repo "https://github.com/signalfx/splunk-otel-js" --json tagName --jq 'select(.isDraft|not and .isPrelease|not) | .tagName' )"
+  if [[ -n "$tag" ]]; then
+    echo ">>> Updating splunk-otel-js version to $tag ..."
+    echo "$tag" > instrumentation/packaging/nodejs-agent-release.txt
+  else
+    echo "ERROR: Failed to get latest release tag from https://github.com/signalfx/splunk-otel-js !" >&2
+    exit 1
+  fi
+
+  # Only create the PR if there are changes
+  if ! git diff --exit-code >/dev/null 2>&1; then
+    git commit -S -am "$message"
+    git push -f "$repo_url" "$branch"
+    echo ">>> Creating the PR ..."
+    gh pr create \
+      --draft \
+      --repo "$repo" \
+      --title "$message" \
+      --body "$message" \
+      --base main \
+      --head "$branch"
+  fi
+}
+
+setup_gpg
+import_gpg_secret_key "$GITHUB_BOT_GPG_KEY"
+setup_git
+create_collector_pr
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,10 +2,30 @@
 
 ## Unreleased
 
+### 🛑 Breaking changes 🛑
+
+- (Splunk) Auto Instrumentation for Linux (manual installation/upgrade) ([#3791](https://github.com/signalfx/splunk-otel-collector/pull/3791)):
+  - The `/usr/lib/splunk-instrumentation/instrumentation.conf` config file is no longer
+    supported, and is replaced by `/etc/splunk/zeroconfig/java.conf`. If the `splunk-otel-auto-instrumentation` deb/rpm
+    package is manually upgraded, the options within `/usr/lib/splunk-instrumentation/instrumentation.conf` will need to
+    be manually migrated to their corresponding environment variables within `/etc/splunk/zeroconfig/java.conf`.
+  - Manual installation of the `splunk-otel-auto-instrumentation` deb/rpm package no longer automatically adds
+    `/usr/lib/splunk-instrumentation/libsplunk.so` to `/etc/ld.so.preload`.
+  - Manual upgrade of the `splunk-otel-auto-instrumentation` deb/rpm package will automatically remove
+    `/usr/lib/splunk-instrumentation/libsplunk.so` from `/etc/ld.so.preload`.
+  - The `splunk.linux-autoinstr.executions` metric is no longer generated by `libsplunk.so`.
+  - See [Splunk OpenTelemetry Zero Configuration Auto Instrumentation for Linux](https://github.com/signalfx/splunk-otel-collector/blob/main/instrumentation/README.md)
+    for manual installation/configuration details.
+
 ### 💡 Enhancements 💡
 
 - (Splunk) Update golang to 1.20.10 ([#3770](https://github.com/signalfx/splunk-otel-collector/pull/3770))
 - (Splunk) Add debian 12 support to installer ([#3766](https://github.com/signalfx/splunk-otel-collector/pull/3766))
+- (Splunk) Add new Auto Instrumentation options for the Linux installer script ([#3791](https://github.com/signalfx/splunk-otel-collector/pull/3791)):
+  - `--with[out]-systemd-instrumentation`: Activate auto instrumentation for only `systemd` services without preloading
+    the `libsplunk.so` shared object library (default: `--without-systemd-instrumentation`)
+  - `--otlp-endpoint host:port`: Set the OTLP gRPC endpoint for captured traces (default: `http://LISTEN_INTERFACE:4317`
+    where `LISTEN_INTERFACE` is the value from the `--listen-interface` option if specified, or `127.0.0.1` otherwise)
 
 ## v0.86.0