[pinpoint-apm#125] mysql module support

* pinpoint node agent development db modules fixtures
* MysqlContainer withCopyFilesToContainer for fixtures
* Fix `ERROR 1064 (42000) at line 16: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 3`
* testcontainer docker-entrypoint-initdb.d learning test
* Support SQL_ID AnnotationKey
* createConnection endPoint SpanEvent
* MySQL connect SpanEvent
* MySQL connect async callback SpanEvent
* Declare MySQL Service Type
* Java agent InstrumentMethod, addScopedInterceptor implementation
* MySQL createConnection Interceptor
* MySQLConnectionInterceptor
* We know hook method API. we need to translate *stack name to human understanding information.
Fix wrong Class name for connection test
* Query Hook
* databaseInfo to MySQLStatementExecuteQueryInterceptor
* transformation sql string to normalizedSql
* SQL Meta Data gRPC API
* DefaultAnnotationKey is no needs. I think it is better to remove it.
* If object uses JSON.stringify, It's right. But other cases, It's wrong. ex) string '123' -> '"123"' It is wrong
* Fix test failure
* annotation
* Support Java agent annotationValue and data type annotation to Node Agent
* fix async spanevent
* Fix audit warning
*  update testcontainer version
* DefaultPredefinedMethodDescriptorRegistry rename refactoring
* delete no used commented code
* AsyncId must have sequence
* recordHTTPURL no standard pinpoint spanevent API
* method descriptor builder needs ApiDescriptor Setter method
* trace.traceBlockEnd needs log.error('spanevent does not exists in call-stack') guard
* Support SQL Bind Value
* fix express test failure
* mysql query matcher
* createPool interceptor
* Support mysql Pool query and  query with getConnection
* MySQL PoolConnection Support
* Scope in interceptor like java agent
* mysql service type and execute query
* Upgrade nodejs engine version matrix version
* mysql2 fixtures

Logger Adaptor pattern
   * The Logger must implements pinpoint.json config loading status and a  log level.
    * support integration test
    * if adaptor doesn't have adaptor outer method, Agent outs logs to console
[pinpoint-apm#117] loglevel default logger integration test
[pinpoint-apm#117] loglevel default logger
[pinpoint-apm#119] Fix "SequenceGenerator is not a constructor" error
[pinpoint-apm#121] Fix testcontainer `It was resolved to update testcontainer v9`
[pinpoint-apm#122] Fix testContainer test failure
[pinpoint-apm#117] make Agent d.ts in repository
[pinpoint-apm#117] node-agent index.d.ts
[pinpoint-apm#122] Fix test failure
[pinpoint-apm#122] fix github action failure
[pinpoint-apm#123] Fix Github action test failure
#2 formatting
#2 express get function name
pinpoint-apm#86 expectedStackString
pinpoint-apm#86 Fix test failure
pinpoint-apm#86 trace
pinpoint-apm#87 HTTP param annotation key
pinpoint-apm#87 url.parse exception guard
pinpoint-apm#87 setAttribute search Param
pinpoint-apm#87 annotation key
pinpoint-apm#87 remove no needs callstack
pinpoint-apm#87 Express with method name
pinpoint-apm#86 express callstack
pinpoint-apm#86 parse stack trace
pinpoint-apm#86 express method name matcher
pinpoint-apm#86 route name
pinpoint-apm#86 use layer symbol
pinpoint-apm#86 use layer symbol
pinpoint-apm#86 extract method refactoring
pinpoint-apm#86 method descriptor
pinpoint-apm#86 method descriptor Builder
pinpoint-apm#86 method name matcher
pinpoint-apm#86 express API Meta cache
pinpoint-apm#86 extract class refactoring
pinpoint-apm#86 callstack extract method refactoring pattern
pinpoint-apm#86 rename refactoring
pinpoint-apm#86 linenumber
pinpoint-apm#86 methodDescriptorBuilder
pinpoint-apm#86 makeMethodDescriptorBuilder
pinpoint-apm#86 method descriptor builder
pinpoint-apm#86 at Type.functionName [as methodName] (location) and at Type.name (location) in https://v8.dev/docs/stack-trace-api
pinpoint-apm#86 fullName
pinpoint-apm#86 post test
 [pinpoint-apm#98] Fix no details error case informations to  Error callstack
By using the string return value of the stack in the toString method of Error, the developer can check the error information in detail.
Fix redis 4.0 redis connection error
   - upgrade testcontainer to v7
[pinpoint-apm#101] The middleware should be displayed with the same depth as next, but the bug was fixed in the hierarchical display incorrectly.
Provides Middleware Code level visibility
Fixed a bug where Error Handler's location was displayed as anonymous Middleware
 [pinpoint-apm#101] TypeScript Express Demo app
 [pinpoint-apm#101] react app sample
 [pinpoint-apm#101] When using React as the front end and Express as the back end, the react sample is unnecessary because it is the same environment as express.
[pinpoint-apm#104] make koa example
[pinpoint-apm#104] koa support code level visibility
[pinpoint-apm#104] remove no used module hook
[pinpoint-apm#104] koa support code level visibility
[pinpoint-apm#104] koa wrap guard
[pinpoint-apm#104] koa module parameter descriptor
https://github.com/koajs/router/blob/master/API.md
router.get('/', (ctx, next) => {
  // ctx.router available
});

[pinpoint-apm#104] remove no needs files
[pinpoint-apm#104] koa module
https://github.com/koajs/router/blob/1aead99e0e0fdb8666e9c6fa2f52b0463c622025/lib/layer.js#L25
[pinpoint-apm#104] koa moudle callstack parsing
[pinpoint-apm#104] Support call stack `Router.<computed> [as get]` with `Function.app.<computed> [as get]`
[pinpoint-apm#104] koa module MethodDescriptorBuilder
[pinpoint-apm#104] koa support code level visibility
[pinpoint-apm#104] koa and express module status code 200 verification

[pinpoint-apm#104] remove no needs match wrap
[pinpoint-apm#111] When a large number of grpc streams occur, the subChannelPool running test has different results depending on the server environment and is deleted

[pinpoint-apm#117] Support logger, authentication adaptor pattern

typescript tools support for .d.ts
    * https://nodejs.org/api/modules.html#enabling
    * fix license convention by https://docs.npmjs.com/cli/v8/configuring-npm/package-json#license
ddac37
Fix submodule url

.github/workflows/main.yml

@@ -3,7 +3,10 @@
 name: CI
 
 # Controls when the action will run. 
-on: [push, pull_request]
+on: [push, pull_request_target]
+
+permissions:
+  checks: write # needed for coverage
 
 # A workflow run is made up of one or more jobs that can run sequentially or in parallel
 jobs:
@@ -14,13 +17,13 @@ jobs:
     strategy:
       matrix:
         node_version:
-          - 12
           - 14
           - 16
+          - 18
 
     name: Node ${{ matrix.node_version }} on ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
       - name: setup node environment
         uses: actions/setup-node@v2
         with:
@@ -32,11 +35,11 @@ jobs:
       - uses: jwgmeligmeyling/checkstyle-github-action@master
         with:
           path: '**/checkstyle-result.xml'
-          github-token: ${{ secrets.GITHUB_TOKEN }}
       - name: Coverage
         run: npm run coverage
+
       - name: Report coverage
-        uses: romeovs/lcov-reporter-action@v0.2.21
+        uses: romeovs/lcov-reporter-action@2a28ec3e25fb7eae9cb537e9141603486f810d1a
         with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
           lcov-file: ./coverage/lcov.info
+          github-token: ${{ secrets.ACTIONS_TOKEN }}

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "grpc/grpc-idl"]
 	path = grpc/grpc-idl
-	url = https://github.com/feelform/pinpoint-grpc-idl.git
+	url = https://github.com/pinpoint-apm/pinpoint-grpc-idl.git

demo/express/routes/index.js

@@ -16,7 +16,7 @@ router.get('/', function(req, res, next) {
     console.log(`${callcount} ioredis data ${data}`)
   })
 
-  axios.get(`https://eonet.sci.gsfc.nasa.gov/api/v2.1/categories`)
+  axios.get(`https://naver.com`)
   .then(function (response) {
     console.log(`response ${response}`)
     res.render('index', { title: 'Express' });

demo/http-server/.env

@@ -0,0 +1,9 @@
+PINPOINT_COLLECTOR_IP=localhost
+PINPOINT_APPLICATION_NAME=http-node-sample-name
+PINPOINT_AGENT_ID=http-node-sample-id
+PINPOINT_SAMPLING_RATE=1
+PINPOINT_LOG_LEVEL=DEBUG
+_PINPOINT_TRACE_EXCLUSION_URL_PATTERN=/api
+_PINPOINT_TRACE_EXCLUSION_URL_CACHE_SIZE=3
+_GRPC_VERBOSITY=DEBUG
+_GRPC_TRACE=call_stream

demo/http-server/index.js

@@ -0,0 +1,37 @@
+import 'dotenv/config'
+import 'pinpoint-node-agent';
+import http from 'http';
+import axios from 'axios';
+
+const host = 'localhost';
+const port = 8000;
+
+const requestListener = function (req, res) {
+    (async () => {
+        try {
+            axios.get('http://localhost:7010/api')
+                .then(function (response) {
+                    // handle success
+                    console.log(response);
+                    res.writeHead(200);
+                    res.end(response.data);
+                })
+                .catch(function (error) {
+                    // handle error
+                    console.log(error);
+                })
+                .finally(function () {
+                    // always executed
+                });
+        } catch (error) {
+            res.status(500).send(response);
+            res.writeHead(500);
+            res.end("My first server!");
+        }
+    })();
+};
+
+const server = http.createServer(requestListener);
+server.listen(port, host, () => {
+    console.log(`Server is running on http://${host}:${port}`);
+});

demo/http-server/package.json

@@ -0,0 +1,17 @@
+{
+  "name": "http-server",
+  "version": "1.0.0",
+  "description": "HTTP module Test",
+  "type": "module",
+  "main": "index.js",
+  "scripts": {
+    "test": "npm test"
+  },
+  "author": "",
+  "license": "ISC",
+  "dependencies": {
+    "axios": "^1.3.2",
+    "dotenv": "^16.0.3",
+    "node-fetch": "^2.6.11"
+  }
+}

demo/springboot/src/main/resources/application.properties

@@ -1 +1 @@
-
+server.port = 7010

demo/springboot/start.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 
-AGENT_PATH=~/Downloads/pinpoint-agent-2.1.0
-SPRING_BOOT_OPT="-javaagent:$AGENT_PATH/pinpoint-bootstrap-2.1.0.jar -Dpinpoint.agentId=express-spring-sampleid -Dpinpoint.applicationName=express-spring-sample"
+AGENT_PATH=~/Downloads/pinpoint-agent-2.5.1
+SPRING_BOOT_OPT="-javaagent:$AGENT_PATH/pinpoint-bootstrap-2.5.1.jar -Dpinpoint.agentId=express-spring-sampleid -Dpinpoint.applicationName=express-spring-sample"
 
 # echo $AGENT_PATH
 # echo $GRADLE_OPTS

lib/agent.js

@@ -76,50 +76,6 @@ class Agent {
 
   completeTraceObject(trace) {
     this.traceContext.completeTraceObject(trace)
-    if (trace.span) {
-      /*
-      const spanEvent = []
-      // step1. 우선 비동기 코드 (추후 더미코드) 를 먼저 가져오도록 한다.
-      const dummyEventMap = trace.span.spanEventList.reduce((map, dummyEvent) => {
-        if (dummyEvent.serviceType === ServiceTypeCode.callback_dummy) {
-          map[dummyEvent.nextDummyId] = dummyEvent
-        }
-        return map
-      }, {})
-      // step2. 콜백코드로 저장된 걸 가져오도록 한다. (순서가 보장 안되기 때문에 해당 과정이 필요하다.. )
-      const callEvent = trace.span.spanEventList.reduce((map, callbackEvent) => {
-        if (callbackEvent.dummyId !== null) {
-          map.push(callbackEvent)
-        }
-        return map
-      }, [])
-      // step3. 비동기 코드를 순차적으로 우회하며 돌도록 한다.
-      callEvent.forEach((event) => {
-        const dummyRecoder = dummyEventMap[event.dummyId]
-        if (dummyRecoder) {
-          event.sequence = dummyRecoder.sequence
-          event.depth = dummyRecoder.depth
-          //TODO. agent view 에서 문제가 있기에 우선 임의로 넣어둔다. (paas와 논의 필요)
-          // event.startElapsed = dummyRecoder.startElapsed
-          // event.elapsedTime = dummyRecoder.elapsedTime
-
-          if (event.nextDummyId !== null) {
-            //TODO. event chaiing인 경우 depth 등을 조정해야한다. 순서가 보장안될 수 있으니.. 차순 변경 후 가도 될 듯
-            dummyEventMap[event.nextDummyId].sequence = event.sequence + 1
-            dummyEventMap[event.nextDummyId].depth = event.depth + 1
-          }
-        }
-      })
-
-      trace.span.spanEventList.forEach((event) => {
-        if (event.serviceType !== ServiceTypeCode.callback_dummy) {
-          spanEvent.push(event)
-        }
-      })
-      trace.span.spanEventList = spanEvent
-      log.debug('finally->\n', trace.span)
-      */
-    }
   }
 
   createAgentInfo(config, agentStartTime) {

lib/client/data-sender-factory.js

@@ -9,17 +9,42 @@
 const GrpcDataSender = require('./grpc-data-sender')
 const DataSender = require('./data-sender')
 
+let dataSender
 const create = (config, agentInfo) => {
-  return new DataSender(config, new GrpcDataSender(
+  dataSender = new DataSender(config, new GrpcDataSender(
     config.collectorIp,
     config.collectorTcpPort,
     config.collectorStatPort,
     config.collectorSpanPort,
     agentInfo,
     config
   ))
+  join()
+  return dataSender
+}
+
+const services = []
+const join = (service) => {
+  if (service) {
+    services.push(service)
+  }
+
+  if (!dataSender) {
+    return
+  }
+
+  services.forEach((service) => {
+    service.dataSender = dataSender
+  })
+}
+
+const setDataSender = (sender) => {
+  dataSender = sender
+  join()
 }
 
 module.exports = {
-  create
+  create,
+  join,
+  setDataSender
 }

lib/client/data-sender.js

@@ -12,6 +12,7 @@ const ApiMetaInfo = require('../data/dto/api-meta-info')
 const StringMetaInfo = require('../data/dto/string-meta-info')
 const Span = require('../context/span')
 const SpanChunk = require('../context/span-chunk')
+const SqlMetaData = require('./sql-meta-data')
 
 class DataSender {
   constructor(config, dataSender) {
@@ -42,6 +43,8 @@ class DataSender {
       this.dataSender.sendSpan(data)
     } else if (data instanceof SpanChunk) {
       this.dataSender.sendSpanChunk(data)
+    } else if (data instanceof SqlMetaData) {
+      this.dataSender.sendSqlMetaInfo(data)
     }
   }
 

lib/client/grpc-data-sender.js

@@ -72,6 +72,7 @@ class GrpcDataSender {
     this.metadataClient = new services.MetadataClient(collectorIp + ":" + collectorTcpPort, grpc.credentials.createInsecure(), { interceptors: [headerInterceptor] })
     this.requestApiMetaData = new GrpcUnaryRPC('requestApiMetaData', this.metadataClient, this.metadataClient.requestApiMetaData, DEFAULT_METADATA_RETRY_DELAY_MILLIS, DEFAULT_METADATA_RETRY_MAX_COUNT)
     this.requestStringMetaData = new GrpcUnaryRPC('requestStringMetaData', this.metadataClient, this.metadataClient.requestStringMetaData, DEFAULT_METADATA_RETRY_DELAY_MILLIS, DEFAULT_METADATA_RETRY_MAX_COUNT)
+    this.requestSqlMetaData = new GrpcUnaryRPC('requestSqlMetaData', this.metadataClient, this.metadataClient.requestSqlMetaData, DEFAULT_METADATA_RETRY_DELAY_MILLIS, DEFAULT_METADATA_RETRY_MAX_COUNT)
   }
 
   initializeSpanStream(collectorIp, collectorSpanPort, headerInterceptor, config) {
@@ -111,7 +112,7 @@ class GrpcDataSender {
   sendAgentInfo(agentInfo, callback) {
     const pAgentInfo = dataConvertor.convertAgentInfo(agentInfo)
     if (log.isDebug()) {
-      log.debug(`sendAgentInfo pAgentInfo: ${JSON.stringify(pAgentInfo.toObject())}`)
+      log.debug(`sendAgentInfo pAgentInfo: ${pAgentInfo}`)
     }
     this.requestAgentInfo.request(pAgentInfo, (err, response) => {
       if (callback) {
@@ -162,11 +163,26 @@ class GrpcDataSender {
     })
   }
 
+  sendSqlMetaInfo(sqlMetaData, callback) {
+    const pSqlMetaData = sqlMetaData.valueOfProtocolBuffer()
+    if (log.isDebug()) {
+      log.debug(`sendSqlMetaInfo sqlMetaData: ${JSON.stringify(pSqlMetaData.toObject())}`)
+    }
+    this.requestSqlMetaData.request(pSqlMetaData, (err, response) => {
+      if (err) {
+        log.error(err)
+      }
+      if (callback) {
+        callback(err, response)
+      }
+    })
+  }
+
   sendSpan(span) {
     try {
       const pSpan = span.spanMessage
       if (log.isDebug()) {
-        log.debug(`sendSpan pSpan: ${JSON.stringify(pSpan.toObject())}`)
+        log.debug(`sendSpan pSpan: ${pSpan}`)
       }
       this.spanStream.write(pSpan)
     } catch (e) {
@@ -180,7 +196,7 @@ class GrpcDataSender {
     try {
       const pSpanChunk = spanChunk.spanMessage
       if (log.isDebug()) {
-        log.debug(`sendSpanChunk pSpanChunk: ${JSON.stringify(pSpanChunk.toObject())}`)
+        log.debug(`sendSpanChunk pSpanChunk: ${pSpanChunk}`)
       }
       this.spanStream.write(pSpanChunk)
     } catch (e) {
@@ -194,7 +210,7 @@ class GrpcDataSender {
     try {
       const pStatMessage = dataConvertor.convertStat(stat)
       if (log.isDebug()) {
-        log.debug(`sendStats pStatMessage: ${JSON.stringify(stat)}`)
+        log.debug(`sendStats pStatMessage: ${stat}`)
       }
       this.statStream.write(pStatMessage)
     } catch (e) {
@@ -215,7 +231,7 @@ class GrpcDataSender {
   sendPing() {
     const pPing = dataConvertor.convertPing()
     if (log.isDebug()) {
-      log.debug(`sendPing pPing: ${JSON.stringify(pPing)}`)
+      log.debug(`sendPing pPing: ${JSON.stringify(pPing.toObject())}`)
     }
     this.pingStream.write(pPing)
   }

lib/client/sql-meta-data.js

@@ -0,0 +1,23 @@
+/**
+ * Pinpoint Node.js Agent
+ * Copyright 2023-present NAVER Corp.
+ * Apache License v2.0
+ */
+
+'use strict'
+const spanMessages = require('../data/v1/Span_pb')
+class SqlMetaData {
+    constructor(parsingResult) {
+        this.sqlId = parsingResult.getId()
+        this.sql = parsingResult.getSql()
+    }
+
+    valueOfProtocolBuffer() {
+        const pSqlMetaData = new spanMessages.PSqlMetaData()
+        pSqlMetaData.setSqlid(this.sqlId)
+        pSqlMetaData.setSql(this.sql)
+        return pSqlMetaData
+    }
+}
+
+module.exports = SqlMetaData

lib/constant/annotation-key.js

@@ -10,9 +10,13 @@
 const AnnotationKey = require('../context/annotation-key')
 const AnnotationKeyProperty = require('./annotation').AnnotationKeyProperty
 
-const DefaultAnnotationKey = {
+module.exports = {
   API: new AnnotationKey(12, 'API'),
   API_METADATA: new AnnotationKey(13, 'API-METADATA'),
+
+  SQL_ID: new AnnotationKey(20, "SQL-ID"),
+  SQL_BINDVALUE: new AnnotationKey(24, "SQL-BindValue", AnnotationKeyProperty.VIEW_IN_RECORD_SET),
+
   HTTP_URL: new AnnotationKey(40, 'http.url'),
   HTTP_PARAM: new AnnotationKey(41, 'http.param', AnnotationKeyProperty.VIEW_IN_RECORD_SET),
   HTTP_STATUS_CODE: new AnnotationKey(46, 'http.status.code'),
@@ -29,12 +33,3 @@ const DefaultAnnotationKey = {
   ARGS9: new AnnotationKey(-10, 'args[9]'),
   ARGSN: new AnnotationKey(-11, 'args[N]'),
 }
-
-const EventAnnotationKey = {
-  MONGO_JSON_DATA: new AnnotationKey(150, "MONGO-JSON-Data")
-}
-
-module.exports = {
-  DefaultAnnotationKey,
-  EventAnnotationKey,
-}