Incorporate NR into our custom Solr image

[nickumia-reisys]

User Story

In order to have a complete monitoring picture, the Data.gov Infrastructure Team wants to install the NR agent in our Solr Deployment.

Acceptance Criteria

• GIVEN the NR Infrastructure Agent has been installed
  AND Solr is running with this agent
  WHEN I look at NR
  THEN I can see Solr information being populated

Background

We talked to NR Support on 9/27/2023 and were given insights into how some things work. This seems like a really good thing to have. We should also instrall it in our CKAN deployments, but I'll make that a different ticket since that is a different deployment mechanism.

Security Considerations (required)

Improving Monitoring Posture.

Sketch

• Follow the install instructions in NR.
• Create new version of docker image
• (Restart Solr in production? Should pull new image?)

[nickumia-reisys]

There are two new relic agent classes: (1) Infrastructure Agent, (2) Application Agent.

The infrastructure agent does not make sense to install because the happy path is running it as a separate container in the "cluster". Since we cannot guarantee that the new relic container will be on the same hardware as the Solr container, I do not think this is helpful. We can work to integrate the new relic container into a complex Solr dockerfile build, but that seems like it's more work than should be required, with no guaranteed results. If we get all of the necessary information from the application agent, we might be able to not go this route.

For historical reference, to launch newrelic as a container: (not to mention big red flags with the docker options)

docker run
   --detach
   --name newrelic-infra
   --network=host
   --cap-add=SYS_PTRACE
   --privileged
   --pid=host
   --volume "/:/host:ro"
   --volume "/var/run/docker.sock:/var/run/docker.sock"
   --volume "newrelic-infra:/etc/newrelic-infra"
   --env NRIA_LICENSE_KEY=<some-key>
 newrelic/infrastructure:latest

The current struggle is finding the best way to integrate the java new relic application agent into the Solr dockerfile. We use a helper script solr-fg to set up the environment and run the Solr application. There are two ways to run the application agent: (1) As a background process (newrelic-unix-monitor), (2) As a wrapper to the java application.

The first way does not get metrics to report to new relic:

wget https://github.com/newrelic/newrelic-unix-monitor/releases/download/r1.0.2/newrelic-unix-monitor-1.0.2-distribution.tar.gz
<edit> newrelic-unix-monitor/config/plugin.json
<edit> newrelic-unix-monitor/pluginctl.sh

diff --git a/solr/Dockerfile b/solr/Dockerfile
index 15a7eae..74671c0 100644
--- a/solr/Dockerfile
+++ b/solr/Dockerfile
@@ -71,6 +71,9 @@ RUN perl -0777  -i -pe 's/ \
+
+ADD newrelic-unix-monitor/ /opt/solr/server/solr/newrelic-unix-monitor/
+
diff --git a/solr/local_setup.sh b/solr/local_setup.sh
index 329d853..b4e503d 100755
--- a/solr/local_setup.sh
+++ b/solr/local_setup.sh
@@ -8,6 +8,7 @@ set -e
 # Start solr
 # Not sure how the path gets messed up, but it does :/ (so we have to fix it)
 su -c "\
+  /opt/solr/server/solr/newrelic-unix-monitor/pluginctl.sh start & \
   export PATH=$PATH:/opt/docker-solr/scripts/:/opt/solr/bin/;\
   init-var-solr; precreate-core ckan /tmp/ckan_config; chown -R 8983:8983 /var/solr/data; solr-fg -Dsolr.lock.type=simple \
 " -m solr

Doing a simple wrapper of the that script requires making a copy of the solr-fg script and then editing the java start command. This seems relatively safe.. we would just need to update the script anytime the version of solr changes..

The team has to decide which way we want to go.. try to get the unix background process to work OR make a copy of the start script and use the java new relic library. @GSA/data-gov-team

[btylerburton]

Simple wrapped seems most logical given the other way doesn't work and has security issues with the docker init