NH-26615 Service Instance #70

Lin-Lin_swi · 2023-01-09T20:17:32Z

Jira

https://swicloud.atlassian.net/browse/NH-26615
See also https://github.com/librato/joboe/pull/1572

Changes

use joboe v7.8.1 with HostID.uuid
conform to new SWO __Init message spec
- delay the call to report __Init message so that the HostID info is populated for the postStatus RPC.
- stumbled around looking for the Resource attributes on the (auto-)configured agent, finally found Expose AutoConfiguredOpenTelemetrySdk to AgentListener open-telemetry/opentelemetry-java-instrumentation#4831.
- mask the sw.apm.service.key system property if captured by Process Resource -- opted for simplest replacement of any char with **** instead of first/last four char, etc. since this information is not customer facing.
see https://github.com/librato/trace/pull/318 for example __Init messages with these changes

Test

In a Linux container and the test collector configured to console.STATUS.printHostId: true, seeing the expected HostID (including uuid) and __Init message:

2023-01-13 23:12:18,739 INFO [ConsoleWriter] - STATUS Finished preprocessing 1 messages
HostId [hostname=c3e925ad327f, ec2InstanceId=, ec2AvailabilityZone=, dockerContainerId=, herokuDynoId=, azureInstanceId=, macAddresses=[], hostType=PERSISTENT, uamsClientId=, uuid=f620e874-ff3d-4f33-825d-cc4b12b2d005] javatest : {"__Init":true,"process.runtime.version":"1.8.0_352-b08","os.type":"linux","process.pid":13,"telemetry.sdk.name":"opentelemetry","telemetry.sdk.language":"java","process.runtime.name":"OpenJDK Runtime Environment","APM.Version":"0.14.2","os.description":"Linux 5.15.49-linuxkit","process.executable.path":"/opt/java/openjdk:bin:java","host.arch":"aarch64","process.command_line":"/opt/java/openjdk:bin:java -Djava.io.tmpdir=/tmp/jetty -javaagent:/opt/solarwinds-apm/solarwinds-apm-agent.jar","host.name":"c3e925ad327f","telemetry.sdk.version":"1.12.0","process.runtime.description":"Temurin OpenJDK 64-Bit Server VM 25.352-b08","telemetry.auto.version":"0.14.2"}
2023-01-13 23:12:18,748 INFO [ConsoleWriter] - STATUS Finished sending messages numbers=1

__Init message pretty-printed:

{
    "__Init": true,
    "process.runtime.version": "1.8.0_352-b08",
    "os.type": "linux",
    "process.pid": 13,
    "Layer": null,
    "telemetry.sdk.name": "opentelemetry",
    "telemetry.sdk.language": "java",
    "process.runtime.name": "OpenJDK Runtime Environment",
    "APM.Version": "0.14.2",
    "os.description": "Linux 5.15.49-linuxkit",
    "process.executable.path": "/opt/java/openjdk:bin:java",
    "host.arch": "aarch64",
    "process.command_line": "/opt/java/openjdk:bin:java -Djava.io.tmpdir=/tmp/jetty -javaagent:/opt/solarwinds-apm/solar
winds-apm-agent.jar",
    "host.name": "c3e925ad327f",
    "telemetry.sdk.version": "1.12.0",
    "process.runtime.description": "Temurin OpenJDK 64-Bit Server VM 25.352-b08",
    "telemetry.auto.version": "0.14.2"
}

Another run in a Linux container, when the env var OTEL_JAVA_DISABLED_RESOURCE_PROVIDERS=io.opentelemetry.sdk.extension.resources.ProcessResourceProvider,io.opentelemetry.sdk.extension.resources.ProcessRuntimeResourceProvider is used to disable Process and ProcessRuntime Resources in the agent, seeing new HostID.uuid and the required __Init message attributes are still set:

2023-01-13 23:20:12,864 INFO [ConsoleWriter] - STATUS Finished preprocessing 1 messages
HostId [hostname=1f253810184d, ec2InstanceId=, ec2AvailabilityZone=, dockerContainerId=, herokuDynoId=, azureInstanceId=, macAddresses=[], hostType=PERSISTENT, uamsClientId=, uuid=4f95cf90-a106-4311-9488-3c3e0817fe5c] javatest : {"__Init":true,"process.runtime.version":"1.8.0_352-b08","os.type":"linux","telemetry.sdk.name":"opentelemetry","telemetry.sdk.language":"java","process.runtime.name":"OpenJDK Runtime Environment","APM.Version":"0.14.2","os.description":"Linux 5.15.49-linuxkit","host.arch":"aarch64","host.name":"1f253810184d","telemetry.sdk.version":"1.12.0","process.runtime.description":"Temurin OpenJDK 64-Bit Server VM 25.352-b08","telemetry.auto.version":"0.14.2"}
2023-01-13 23:20:12,872 INFO [ConsoleWriter] - STATUS Finished sending messages numbers=1

__Init message pretty-printed:

{
    "__Init": true,
    "process.runtime.version": "1.8.0_352-b08",
    "os.type": "linux",
    "Layer": null,
    "telemetry.sdk.name": "opentelemetry",
    "telemetry.sdk.language": "java",
    "process.runtime.name": "OpenJDK Runtime Environment",
    "APM.Version": "0.14.2",
    "os.description": "Linux 5.15.49-linuxkit",
    "host.arch": "aarch64",
    "host.name": "1f253810184d",
    "telemetry.sdk.version": "1.12.0",
    "process.runtime.description": "Temurin OpenJDK 64-Bit Server VM 25.352-b08",
    "telemetry.auto.version": "0.14.2"
}

Another run in a Linux container, where the service key is passed in the command line as a Java system property (and otel debugging is enabled), e.g. -Dsw.apm.service.key=top-secret:foo and -Dotel.javaagent.debug=true, seeing service key masked:

Instrumented application log:

[otel.javaagent 2023-01-13 23:35:48:754 +0000] [SolarwindsAPM-post-startup-tasks-1] DEBUG com.appoptics.opentelemetry.extensions.initialize.Initializer - Resource attributes {host.arch="aarch64", host.name="fd2719b73337", os.description="Linux 5.15.49-linuxkit", os.type="linux", process.command_line="/opt/java/openjdk:bin:java -Djava.io.tmpdir=/tmp/jetty -javaagent:/opt/solarwinds-apm/solarwinds-apm-agent.jar -Dsw.apm.service.key=**** -Dotel.javaagent.debug=true", process.executable.path="/opt/java/openjdk:bin:java", process.pid=13, process.runtime.description="Temurin OpenJDK 64-Bit Server VM 25.352-b08", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_352-b08", service.name="unknown_service:java", telemetry.auto.version="0.14.2", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.12.0"}

__Init message pretty-printed:

{
    "__Init": true,
    "process.runtime.version": "1.8.0_352-b08",
    "os.type": "linux",
    "process.pid": 13,
    "Layer": null,
    "telemetry.sdk.name": "opentelemetry",
    "telemetry.sdk.language": "java",
    "process.runtime.name": "OpenJDK Runtime Environment",
    "APM.Version": "0.14.2",
    "os.description": "Linux 5.15.49-linuxkit",
    "process.executable.path": "/opt/java/openjdk:bin:java",
    "host.arch": "aarch64",
    "process.command_line": "/opt/java/openjdk:bin:java -Djava.io.tmpdir=/tmp/jetty -javaagent:/opt/solarwinds-apm/solarwinds-apm-agent.jar -Dsw.apm.service.key=**** -Dotel.javaagent.debug=true",
    "host.name": "fd2719b73337",
    "telemetry.sdk.version": "1.12.0",
    "process.runtime.description": "Temurin OpenJDK 64-Bit Server VM 25.352-b08",
    "telemetry.auto.version": "0.14.2"
}

On a Windows EC2 instance with metadata disabled (to simulate on-prem host) and the test collector configured to console.STATUS.printHostId: true, seeing the expected HostID (including uuid) and __Init message:

2023-01-14 00:03:42,024 INFO [ConsoleWriter] - STATUS Finished preprocessing 1 messages
HostId [hostname=EC2AMAZ-IOH4MF4, ec2InstanceId=, ec2AvailabilityZone=, dockerContainerId=, herokuDynoId=, azureInstanceId=, macAddresses=[12:8E:6F:95:8F:EB], hostType=PERSISTENT, uamsClientId=7d339695-3feb-4bad-bd9c-a93cd5c85388, uuid=d875fca2-f963-4ed1-8b4a-1e15689d5b1c] javatest : {"__Init":true,"process.runtime.version":"13+33","os.type":"windows","process.pid":5280,"telemetry.sdk.name":"opentelemetry","telemetry.sdk.language":"java","process.runtime.name":"OpenJDK Runtime Environment","APM.Version":"0.14.2","os.description":"Windows Server 2019 10.0","process.executable.path":"C:\\openjdk-13\\jdk-13;bin;java.exe","host.arch":"amd64","process.command_line":"C:\\openjdk-13\\jdk-13;bin;java.exe --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=c:\\Program Files\\Apache Software Foundation\\Tomcat 9.0\\conf\\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -javaagent:C:\\Program Files\\SolarWinds\\APM\\java\\solarwinds-apm-agent.jar -Dsw.apm.trustedpath=C:\\Program Files\\java-collector\\server-grpc.pem -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dignore.endorsed.dirs= -Dcatalina.base=c:\\Program Files\\Apache Software Foundation\\Tomcat 9.0 -Dcatalina.home=c:\\Program Files\\Apache Software Foundation\\Tomcat 9.0 -Djava.io.tmpdir=c:\\Program Files\\Apache Software Foundation\\Tomcat 9.0\\temp","host.name":"EC2AMAZ-IOH4MF4","telemetry.sdk.version":"1.12.0","process.runtime.description":"Oracle Corporation OpenJDK 64-Bit Server VM 13+33","telemetry.auto.version":"0.14.2"}
2023-01-14 00:03:42,028 INFO [ConsoleWriter] - STATUS Finished sending messages numbers=1

__Init message pretty-printed:

{
  "__Init": true,
  "process.runtime.version": "13+33",
  "os.type": "windows",
  "process.pid": 5280,
  "telemetry.sdk.name": "opentelemetry",
  "telemetry.sdk.language": "java",
  "process.runtime.name": "OpenJDK Runtime Environment",
  "APM.Version": "0.14.2",
  "os.description": "Windows Server 2019 10.0",
  "process.executable.path": "C:\\openjdk-13\\jdk-13;bin;java.exe",
  "host.arch": "amd64",
  "process.command_line": "C:\\openjdk-13\\jdk-13;bin;java.exe --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=c:\\Program Files\\Apache Software Foundation\\Tomcat 9.0\\conf\\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -javaagent:C:\\Program Files\\SolarWinds\\APM\\java\\solarwinds-apm-agent.jar -Dsw.apm.trustedpath=C:\\Program Files\\java-collector\\server-grpc.pem -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dignore.endorsed.dirs= -Dcatalina.base=c:\\Program Files\\Apache Software Foundation\\Tomcat 9.0 -Dcatalina.home=c:\\Program Files\\Apache Software Foundation\\Tomcat 9.0 -Djava.io.tmpdir=c:\\Program Files\\Apache Software Foundation\\Tomcat 9.0\\temp",
  "host.name": "EC2AMAZ-IOH4MF4",
  "telemetry.sdk.version": "1.12.0",
  "process.runtime.description": "Oracle Corporation OpenJDK 64-Bit Server VM 13+33",
  "telemetry.auto.version": "0.14.2"
}

tammy-baylis_swi · 2023-01-17T19:34:38Z

custom/src/main/java/com/appoptics/opentelemetry/extensions/initialize/Initializer.java

@@ -59,7 +72,6 @@ public static void initialize() throws InvalidConfigException {
        //future = executeStartupTasks(); //Cannot call this here, see https://github.com/appoptics/opentelemetry-custom-distro/issues/7


Could this line be removed completely? Or if good to keep for future, is there a way this method can be called with AutoConfiguredOpenTelemetrySdk otelSDK to match sig update?

Yes that comment seems very much leftover from exploratory stage so I've removed it.

tammy-baylis_swi

Overall lgtm! Just added one comment.

Thank you for posting the detailed example messages that match the updated spec! Great use of OTel Java's resource attributes and afterAgent to get those defaults.

Xuan-Cao_swi

LGTM!

malin-olafson_swi · 2023-01-17T23:12:36Z

custom/src/main/java/com/appoptics/opentelemetry/extensions/initialize/Initializer.java

+            initMessage.put(TELEMETRY_SDK_LANGUAGE.getKey(), "java");
+        }
+        try {
+            if (!initMessage.containsKey(PROCESS_RUNTIME_DESCRIPTION.getKey())) {


The previous code checked to make sure the system property returned in System.getProperty wasn't null (eg java.version). Should the uses of System.getProperty continue to do that?

Yes! Certain system properties are always available so I've added defaults for ones that are not: java.runtime.name, java.runtime.version

malin-olafson_swi

PR looks good, just had one minor comment.

tammy-baylis_swi

Lgtm! Thanks for the revisions!

malin-olafson_swi

Looks good 👍

Use joboe with HostID.uuid; conform to new SWO __Init message spec

00196e0

Lin-Lin_swi mannequin requested a review from ghost January 14, 2023 00:39

tammy-baylis_swi mannequin reviewed Jan 17, 2023

View reviewed changes

Xuan-Cao_swi mannequin reviewed Jan 17, 2023

View reviewed changes

malin-olafson_swi mannequin reviewed Jan 17, 2023

View reviewed changes

Removed old comment; handle possible null system properties

736c152

Lin-Lin_swi mannequin dismissed stale reviews from ghost via 736c152 January 18, 2023 23:23

Lin-Lin_swi mannequin requested review from malin-olafson_swi, Xuan-Cao_swi and tammy-baylis_swi January 18, 2023 23:33

tammy-baylis_swi mannequin approved these changes Jan 19, 2023

View reviewed changes

malin-olafson_swi mannequin approved these changes Jan 19, 2023

View reviewed changes

Lin-Lin_swi mannequin merged commit 81c6d2f into main Jan 21, 2023

Lin-Lin_swi mannequin deleted the NH-26615-service-instance branch January 21, 2023 00:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NH-26615 Service Instance #70

NH-26615 Service Instance #70

Lin-Lin_swi mannequin commented Jan 9, 2023

tammy-baylis_swi mannequin Jan 17, 2023

Lin-Lin_swi mannequin Jan 18, 2023

tammy-baylis_swi mannequin left a comment

Xuan-Cao_swi mannequin left a comment

malin-olafson_swi mannequin Jan 17, 2023

Lin-Lin_swi mannequin Jan 18, 2023

malin-olafson_swi mannequin left a comment

tammy-baylis_swi mannequin left a comment

malin-olafson_swi mannequin left a comment

		@@ -59,7 +72,6 @@ public static void initialize() throws InvalidConfigException {
		//future = executeStartupTasks(); //Cannot call this here, see https://github.com/appoptics/opentelemetry-custom-distro/issues/7

NH-26615 Service Instance #70

NH-26615 Service Instance #70

Conversation

Lin-Lin_swi mannequin commented Jan 9, 2023

Jira

Changes

Test

tammy-baylis_swi mannequin Jan 17, 2023

Choose a reason for hiding this comment

Lin-Lin_swi mannequin Jan 18, 2023

Choose a reason for hiding this comment

tammy-baylis_swi mannequin left a comment

Choose a reason for hiding this comment

Xuan-Cao_swi mannequin left a comment

Choose a reason for hiding this comment

malin-olafson_swi mannequin Jan 17, 2023

Choose a reason for hiding this comment

Lin-Lin_swi mannequin Jan 18, 2023

Choose a reason for hiding this comment

malin-olafson_swi mannequin left a comment

Choose a reason for hiding this comment

tammy-baylis_swi mannequin left a comment

Choose a reason for hiding this comment

malin-olafson_swi mannequin left a comment

Choose a reason for hiding this comment