[APM] Collect telemetry about data/queries

[dgieselaar]

We currently don't have a lot of insight into the amounts of data that our customers have and how long it takes for our ES queries to be processed. This makes it hard to judge at which scale current and new functionalities need to operate. For instance, in some cases it might be reasonable to process things in memory on the Node server rather than in ES, which might simplify our implementation. Additionally, optimizing right now is hard because we don't know where our users are experiencing slowness.

Ideally we would have telemetry about:

• The data volume (how many errors/error groups? how many transactions? how many transactions/spans per trace? how many services? etc). This could be collected with a Kibana task that queries the data indices at a set interval and sends the data back home.
• Query response times. We could instrument our ES client facade to store telemetry about ES response times. There's also the possibility of using the nodejs agent to instrument Kibana, but the ongoing efforts are explicitly scoped to non-production usage: Instrument Kibana with Elastic APM #43548

@graphaelli: any idea if the monitoring data that we have provides an answer to any of these questions?

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[formgeist]

Not sure about this, but would it be apm-server telemetry that reports the no. of transactions, errors and metrics? Or would it be better to query against the various document types in ES not to put additional pressure on server?

[dgieselaar]

I'd think that query should happen in ES. AFAIK, APM Server is stateless, and at most knows about the amount of documents it is currently processing (and there could be multiple instances of APM Server as well). Telemetry is reported from Kibana as well, so I would imagine this as a Kibana task that queries ES and then sends up telemetry to our telemetry cluster.

[formgeist]

@dgieselaar OK, that makes sense

[sorenlouv]

Thanks for creating this issue. Super good point about us needing better insights into the users data volumes and pain points in querying this.

Query response times
I'd really hate for us to re-build the nodejs APM agent all over again, so I'd much prefer if we could investigate ways to piggyback on the extensive auto-instrumentation it has.
Afair we ran into two problems when enabling the agent in prod:

1. PII: we were worried about sending sensitive data
2. secretToken needed to be bundled with kibana, which is not great

Regarding #1 I think the node agent is quite configurable and we might be able to turn off most things except for elasticsearch query performance numbers. And we might be able to enable it just for specific plugins.

Regarding #2 we might be able to disable the transport via APM Server, and instead send it as telemetry data, and thus avoid needing secretToken. This would probably require us to parse it first but it would still provide us with a solution that potentially can auto-instrument the entire Kibana and send consistent performance telemetry.
I've emphasised consistent here because it's key that performance data it collected similarly between plugins for us to be able to compare it

If using the APM agent is not feasible in the short term I'm good with starting out doing it ourselves.

Data volume
APM agent doesn't do anything in this area so I feel much better about solving this on our own.
Again, it would be optimal if we can build something that other plugins can use too, to improve the consistency of the collected data.

[dgieselaar]

Looks like nested could work for us, nice stuff:

Copy and paste this into the console

PUT apm-telemetry-test

PUT apm-telemetry-test/_mapping
{
  "properties": {
    "plugins": {
      "properties": {
        "apm": {
          "properties": {
            "has_any_services": {
              "type": "boolean"
            },
            "services_per_agent": {
              "properties": {
                "go": {
                  "type": "long"
                },
                "java": {
                  "type": "long"
                },
                "js-base": {
                  "type": "long"
                },
                "rum-js": {
                  "type": "long"
                },
                "nodejs": {
                  "type": "long"
                },
                "python": {
                  "type": "long"
                },
                "dotnet": {
                  "type": "long"
                },
                "ruby": {
                  "type": "long"
                }
              }
            },
            "endpoint_responses": {
              "type": "nested",
              "properties": {
                "path": {
                  "type": "keyword"
                },
                "status_code": {
                  "type": "integer"
                },
                "response_time_ms": {
                  "type": "double"
                }
              }
            }
          }
        }
      }
    }
  }
}

POST apm-telemetry-test/_delete_by_query
{
  "query": {
    "match_all": {}
  }
}

POST apm-telemetry-test/_doc
{
  "plugins": {
    "apm": {
      "endpoint_responses": [
        {
          "path": "POST /api/apm/foo",
          "status_code": 200,
          "response_time_ms": 100
        },
        {
          "path": "GET /api/apm/foo",
          "status_code": 200,
          "response_time_ms": 200
        },
        {
          "path": "GET /api/apm/bar",
          "status_code": 500,
          "response_time_ms": 300
        }
      ]
    }
  }
}

GET apm-telemetry-test/_search

GET apm-telemetry-test/_search
{
  "size": 0,
  "aggs": {
    "average_response_time_per_endpoint": {
      "nested": {
        "path": "plugins.apm.endpoint_responses"
      },
      "aggs": {
        "by_path": {
          "terms": {
            "field": "plugins.apm.endpoint_responses.path"
          },
          "aggs": {
            "average_response_time":  {
              "avg": {
                "field": "plugins.apm.endpoint_responses.response_time_ms"
              }
            }
          }
        }
      }
    }
  }
}

[dgieselaar]

What about:

interface TimeframeMap {
  '1d': number;
  '1mo': number;
  '6mo': number;
  all: number;
}

interface APMDataTelemetry {
  has_any_services: boolean;
  services_per_agent: {
    go: number;
    java: number;
    'js-base': number;
    'rum-js': number;
    nodejs: number;
    python: number;
    dotnet: number;
    ruby: number;
  };
  data_characteristics: {
    transactions: TimeframeMap;
    spans: TimeframeMap;
    errors: TimeframeMap;
    metrics: TimeframeMap;
    transaction_groups: Pick<TimeframeMap, 'all'>;
    error_groups: Pick<TimeframeMap, 'all'>;
    traces: Pick<TimeframeMap, 'all'>;
    services: Pick<TimeframeMap, 'all'>;
    agent_configurations: Pick<TimeframeMap, 'all'>;
  };
  integrations: {
    alerting:boolean;
    ml:boolean;
  }
}

interface APMPerformanceMeasurement {
  path: string;
  response_time_ms: number;
  status_code: number;
}

interface APMPerformanceTelemetry {
  total_api_requests: number;
  requests: APMPerformanceMeasurement[];
}

type APMTelemetry = APMDataTelemetry & APMPerformanceTelemetry;