usage reporting: add sendTraces, rename sendErrorsInTraces, etc (#6855)

- We've decided to leave the usage reporting defaults in AS4 as "field
  level instrumentation on for all operations, send some operations as
  traces". But we'd like to at least provide a non-experimental
  mechanism for entirely disabling sending traces to Studio. So there's
  a new ApolloServerPluginUsageReporting: `sendTraces: false`.
- Rename the (new in AS4) usage reporting option `sendErrorsInTraces` to
  `sendErrors`, because it does also affect error statistics in stats
  reports if your `transform` function returns `null`.
- Remove Apollo-internal `internal_includeTracesContributingToStats`
  option. This enabled some internal consistency monitoring which we are
  no longer paying attention to.
- If you enable `debugPrintReports`, send the reports as `info` rather
  than `warn`, which primarily lets us delete some large comments (and
  makes sense because these debug messages are not warnings).

Fixes #6051. Fixes #6078.

.changeset/healthy-glasses-reflect.md

@@ -0,0 +1,5 @@
+---
+"@apollo/server": patch
+---
+
+New usage reporting option `sendTraces: false` to only send usage reports as aggregated statistics, not per-request traces.

.changeset/many-plums-smile.md

@@ -0,0 +1,5 @@
+---
+"@apollo/server": patch
+---
+
+Remove Apollo-internal `internal_includeTracesContributingToStats`. This should not have been used other than inside Apollo's own servers.

.changeset/short-lies-kneel.md

@@ -0,0 +1,5 @@
+---
+"@apollo/server": patch
+---
+
+The usage reporting option `debugPrintReports` now displays reports via `logger.info` rather than `logger.warn`.

.changeset/silly-icons-taste.md

@@ -0,0 +1,6 @@
+---
+"@apollo/server-integration-testsuite": patch
+"@apollo/server": patch
+---
+
+Rename usage reporting option `sendErrorsInTraces` (added in 4.0.0-alpha.4) to `sendErrors`, as it also affects error statistics outside of traces.

docs/source/data/errors.mdx

@@ -438,7 +438,7 @@ You can use Apollo Studio to analyze your server's error rates. By default, the
 
 If you _do_ want error information sent to Studio, you can send every error, or you can modify or redact specific errors before they're transmitted.
 
-To send all errors to Studio you can pass `{ unmodified: true }` to `sendErrorsInTraces`, like so:
+To send all errors to Studio you can pass `{ unmodified: true }` to `sendErrors`, like so:
 
 ```ts {7}
 new ApolloServer({
@@ -447,23 +447,23 @@ new ApolloServer({
  ApolloServerPluginUsageReporting({
  // If you pass unmodified: true to the usage reporting
  // plugin, Apollo Studio receives ALL error details
- sendErrorsInTraces: { unmodified: true },
+ sendErrors: { unmodified: true },
  }),
  ],
 });
 ```
 
-If you want to report specific errors or modify an error before reporting it, you can pass a function to the `sendErrorsInTraces.transform` option, like so:
+If you want to report specific errors or modify an error before reporting it, you can pass a function to the `sendErrors.transform` option, like so:
 
 ```ts {4-6}
 new ApolloServer({
  // etc.
  plugins: [
  ApolloServerPluginUsageReporting({
- sendErrorsInTraces: {
+ sendErrors: {
  transform: (err) => {
  if (err.extensions.code === 'MY_CUSTOM_CODE') {
- // returning null will skip reporting this error 
+ // returning null will skip reporting this error
  return null;
  }
 
@@ -483,6 +483,8 @@ The function you pass to `transform` is called for each error (`GraphQLError`) t
 - Return a modified form of the error (e.g., by changing the `err.message` to remove potentially sensitive information)
 - Return `null` to prevent the error from being reported entirely
 
+Note that returning `null`` also affects Studio's aggregated statistics about how many operations contain errors and at what paths those errors appear.
+
 [As mentioned above](#for-client-responses), you can use the `unwrapResolverError` (from `@apollo/server/errors`) to remove the `GraphQLError` wrapping an original error.
 
 <!-- TODO(AS4) Replace link once plugin docs are updated -->
@@ -502,7 +504,7 @@ const server = new ApolloServer({
  plugins: [
  // highlight-start
  ApolloServerPluginUsageReporting({
- sendErrorsInTraces: {
+ sendErrors: {
  transform: (err) => {
  // Return `null` to avoid reporting `UNAUTHENTICATED` errors
  if (err.extensions.code === 'UNAUTHENTICATED') {
@@ -536,7 +538,7 @@ const server = new ApolloServer({
  resolvers,
  plugins: [
  ApolloServerPluginUsageReporting({
- sendErrorsInTraces: {
+ sendErrors: {
  transform: (err) => {
  // Using a more stable, known error extension (e.g. `err.code`) would be
  // more defensive, however checking the `message` might serve most needs!
@@ -565,7 +567,7 @@ If you _do_ want to send an error's details to Apollo Studio, but need to redact
 
 For example, if there is personally identifiable information in the error `message`, like an API key:
 
-```ts 
+```ts
 import { GraphQLError } from 'graphql';
 
 throw new GraphQLError(
@@ -584,7 +586,7 @@ const server = new ApolloServer({
  resolvers,
  plugins: [
  ApolloServerPluginUsageReporting({
- sendErrorsInTraces: {
+ sendErrors: {
  transform: (err) => {
  // Make sure that a specific pattern is removed from all error messages.
  err.message = err.message.replace(/x-api-key:[A-Z0-9-]+/, 'REDACTED');

docs/source/migration.mdx

@@ -936,7 +936,7 @@ ApolloServerPluginUsageReporting({
 
 In Apollo Server 3, you can specify a function to rewrite errors before sending them to Apollo's server via the `rewriteError` option to `ApolloServerPluginUsageReporting` (for monoliths) and `ApolloServerPluginInlineTrace` (for subgraphs).
 
-In Apollo Server 4, you specify the same function as the `transform` option on the `sendErrorsInTraces` option to `ApolloServerPluginUsageReporting` and the `includeErrors` option to `ApolloServerPluginInlineTrace`.
+In Apollo Server 4, you specify the same function as the `transform` option on the `sendErrors` option to `ApolloServerPluginUsageReporting` and the `includeErrors` option to `ApolloServerPluginInlineTrace`.
 
 (Additionally, the [default behavior has changed to mask errors](#usage-reporting-and-inline-trace-plugins-mask-errors-by-default).)
 
@@ -962,7 +962,7 @@ you can now write:
 // monoliths
 new ApolloServer({
  plugins: [ApolloServerPluginUsageReporting({
- sendErrorsInTraces: { transform: rewriteError },
+ sendErrors: { transform: rewriteError },
  })],
  // ...
 })
@@ -1599,7 +1599,7 @@ To restore the Apollo Server 3 behavior, you can pass `{ unmodified: true }` to
 // monoliths
 new ApolloServer({
  plugins: [ApolloServerPluginUsageReporting({
- sendErrorsInTraces: { unmodified: true },
+ sendErrors: { unmodified: true },
  })],
  // ...
 })
@@ -1613,7 +1613,7 @@ new ApolloServer({
 })
 ```
 
-(As [described above](#rewriteerror-plugin-option), the `rewriteError` option has been replaced by a `transform` option on `sendErrorsInTraces` or `includeErrors`.)
+(As [described above](#rewriteerror-plugin-option), the `rewriteError` option has been replaced by a `transform` option on `sendErrors` or `includeErrors`.)
 
 ## Renamed packages
 

packages/integration-testsuite/src/apolloServerTests.ts

@@ -1139,14 +1139,14 @@ export function defineIntegrationTestSuiteApolloServerTests(
  });
 
  describe('error munging', () => {
- describe('sendErrorsInTraces', () => {
+ describe('sendErrors', () => {
  it('new error', async () => {
  throwError.mockImplementationOnce(() => {
  throw new Error('transform nope');
  });
 
  await setupApolloServerAndFetchPair({
- sendErrorsInTraces: {
+ sendErrors: {
  transform: () =>
  new GraphQLError('rewritten as a new error'),
  },
@@ -1191,7 +1191,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
  });
 
  await setupApolloServerAndFetchPair({
- sendErrorsInTraces: {
+ sendErrors: {
  transform: (err) => {
  err.message = 'rewritten as a modified error';
  return err;
@@ -1236,7 +1236,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
  });
 
  await setupApolloServerAndFetchPair({
- sendErrorsInTraces: { transform: () => null },
+ sendErrors: { transform: () => null },
  });
 
  const result = await apolloFetch({
@@ -1271,7 +1271,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
  });
 
  await setupApolloServerAndFetchPair({
- sendErrorsInTraces: {
+ sendErrors: {
  // @ts-expect-error (not allowed to be undefined)
  transform: () => undefined,
  },
@@ -1319,7 +1319,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
  });
 
  await setupApolloServerAndFetchPair({
- sendErrorsInTraces: {
+ sendErrors: {
  unmodified: true,
  },
  });
@@ -1370,7 +1370,7 @@ export function defineIntegrationTestSuiteApolloServerTests(
  });
 
  await setupApolloServerAndFetchPair({
- sendErrorsInTraces: {
+ sendErrors: {
  masked: true,
  },
  });

packages/server/src/__tests__/plugin/usageReporting/plugin.test.ts

@@ -174,6 +174,24 @@ describe('end-to-end', () => {
  ).toBeTruthy();
  });
 
+ it('sendTraces: false', async () => {
+ const { report } = await runTest({ pluginOptions: { sendTraces: false } });
+
+ expect(Object.keys(report.tracesPerQuery)).toHaveLength(1);
+ expect(Object.keys(report.tracesPerQuery)[0]).toMatch(/^# q\n/);
+ const tracesAndStats = Object.values(report.tracesPerQuery)[0]!;
+ expect(tracesAndStats.trace).toHaveLength(0);
+ expect(tracesAndStats.statsWithContext).toHaveLength(1);
+ const contextualizedStats = (
+ tracesAndStats.statsWithContext as ContextualizedStats[]
+ )[0]!;
+ expect(contextualizedStats.queryLatencyStats?.requestCount).toBe(1);
+ expect(
+ contextualizedStats.perTypeStat['User'].perFieldStat?.['name']
+ .observedExecutionCount,
+ ).toBe(1);
+ });
+
  [
  {
  testName: 'fails parse for non-parsable gql',

packages/server/src/plugin/usageReporting/options.ts

@@ -14,23 +14,65 @@ export interface ApolloServerPluginUsageReportingOptions<
 > {
  //#region Configure exactly which data should be sent to Apollo.
  /**
- * By default, Apollo Server does not send the values of any GraphQL variables to Apollo's servers, because variable
- * values often contain the private data of your app's users. If you'd like variable values to be included in traces, set this option.
- * This option can take several forms:
+ * Apollo Server's usage reports describe each individual request in one of
+ * two ways: as a "trace" (a detailed description of the specific request,
+ * including a query plan and resolver tree with timings and errors, as well
+ * as optional details like variable values and HTTP headers), or as part of
+ * aggregated "stats" (where invocations of the same operation from the same
+ * client program are aggregated together rather than described individually).
+ * Apollo Server uses an heuristic to decide which operations to describe as
+ * traces and which to aggregate as stats.
+ *
+ * By setting the `sendTraces` option to `false`, Apollo Server will describe
+ * *all* operations as stats; individual requests will never be broken out
+ * into separate traces. If you set `sendTraces: false`, then Apollo Studio's
+ * Traces view won't show any traces (other Studio functionality will be
+ * unaffected).
+ *
+ * Note that the values of `sendVariableValues`, `sendHeaders`, and
+ * `sendUnexecutableOperationDocuments` are irrelevant if you set
+ * `sendTraces: false`, because those options control data that is contained
+ * only in traces (not in stats).
+ *
+ * Setting `sendTraces: false` does *NOT* imply `fieldLevelInstrumentation:
+ * 0`. Apollo Server can still take advantage of field-level instrumentation
+ * (either directly for monolith servers, or via federated tracing for
+ * Gateways) in order to accurately report field execution usage in "stats".
+ * This option only controls whether data is sent to Apollo's servers as
+ * traces, not whether traces are internally used to learn about usage.
+ */
+ sendTraces?: boolean;
+
+ /**
+ * By default, Apollo Server does not send the values of any GraphQL variables
+ * to Apollo's servers, because variable values often contain the private data
+ * of your app's users. If you'd like variable values to be included in
+ * traces, set this option. This option can take several forms:
  * - { none: true }: don't send any variable values (DEFAULT)
  * - { all: true}: send all variable values
- * - { transform: ... }: a custom function for modifying variable values. Keys added by the custom function will
- * be removed, and keys removed will be added back with an empty value. For security reasons, if an error occurs within this function, all variable values will be replaced with `[PREDICATE_FUNCTION_ERROR]`.
- * - { exceptNames: ... }: a case-sensitive list of names of variables whose values should not be sent to Apollo servers
- * - { onlyNames: ... }: A case-sensitive list of names of variables whose values will be sent to Apollo servers
+ * - { transform: ... }: a custom function for modifying variable values. The
+ * function receives `variables` and `operationString` and should return a
+ * record of `variables` with the same keys as the `variables` it receives
+ * (added variables will be ignored and removed variables will be reported
+ * with an empty value). For security reasons, if an error occurs within
+ * this function, all variable values will be replaced with
+ * `[PREDICATE_FUNCTION_ERROR]`.
+ * - { exceptNames: ... }: a case-sensitive list of names of variables whose
+ * values should not be sent to Apollo servers
+ * - { onlyNames: ... }: A case-sensitive list of names of variables whose
+ * values will be sent to Apollo servers
+ *
+ * Defaults to not sending any variable values if both this parameter and the
+ * deprecated `privateVariables` are not set. The report will indicate each
+ * private variable key whose value was redacted by { none: true } or {
+ * exceptNames: [...] }.
  *
- * Defaults to not sending any variable values if both this parameter and
- * the deprecated `privateVariables` are not set. The report will
- * indicate each private variable key whose value was redacted by { none: true } or { exceptNames: [...] }.
+ * The value of this option is not relevant if you set `sendTraces: false`,
+ * because variable values only appear in traces.
  */
  sendVariableValues?: VariableValueOptions;
  /**
- * By default, Apollo Server does not send the list of HTTP headers and values
+ * By default, Apollo Server does not send the HTTP request headers and values
  * to Apollo's servers, as these headers may contain your users' private data.
  * If you'd like this information included in traces, set this option. This
  * option can take several forms:
@@ -44,6 +86,9 @@ export interface ApolloServerPluginUsageReportingOptions<
  *
  * Unlike with sendVariableValues, names of dropped headers are not reported.
  * The headers 'authorization', 'cookie', and 'set-cookie' are never reported.
+ *
+ * The value of this option is not relevant if you set `sendTraces: false`,
+ * because request headers only appear in traces.
  */
  sendHeaders?: SendValuesBaseOptions;
  /**
@@ -65,10 +110,12 @@ export interface ApolloServerPluginUsageReportingOptions<
  * (either a new error, or its potentially-modified argument) or `null`.
  * This error is used in the report to Apollo servers; if `null`, the error
  * is not included in traces or error statistics.
+ *
+ * If you set `sendTraces: false`, then the only relevant aspect of this
+ * option is whether you return `null` from a `transform` function or not
+ * (which affects aggregated error statistics).
  */
- sendErrorsInTraces?: SendErrorsOptions;
-
- // We should strongly consider changing the default to false in AS4.
+ sendErrors?: SendErrorsOptions;
 
  /**
  * This option allows you to choose if Apollo Server should calculate detailed
@@ -127,8 +174,9 @@ export interface ApolloServerPluginUsageReportingOptions<
  * (Note that returning true here does *not* mean that the data derived from
  * field-level instrumentation must be transmitted to Apollo Studio's servers
  * in the form of a trace; it may still be aggregated locally to statistics.
- * But either way this operation will contribute to the "field executions"
- * statistic and timing hints.)
+ * Similarly, setting `sendTraces: false` does not affect
+ * `fieldLevelInstrumentation`. But either way this operation will contribute
+ * to the "field executions" statistic and timing hints.)
  *
  * The default `fieldLevelInstrumentation` is a function that always returns
  * true.
@@ -211,6 +259,9 @@ export interface ApolloServerPluginUsageReportingOptions<
  * and the operation name and signature will always be reported with a constant
  * identifier. Whether the operation was a parse failure or a validation
  * failure will be embedded within the stats report key itself.
+ *
+ * The value of this option is not relevant if you set `sendTraces: false`,
+ * because unexecutable operation documents only appear in traces.
  */
  sendUnexecutableOperationDocuments?: boolean;
 
@@ -225,6 +276,9 @@ export interface ApolloServerPluginUsageReportingOptions<
  * Apollo's servers perform their own sampling on received traces; not all
  * traces sent to Apollo's servers can be later retrieved via the trace UI.)
  *
+ * If you just want to send all operations as stats, set `sendTraces: false`
+ * instead of using this experimental hook.
+ *
  * This option is highly experimental and may change or be removed in future
  * versions.
  */
@@ -298,7 +352,7 @@ export interface ApolloServerPluginUsageReportingOptions<
  /**
  * If set, prints all reports as JSON when they are sent. (Note that for
  * technical reasons, traces embedded in a report are printed separately when
- * they are added to a report.)
+ * they are added to a report.) Reports are sent through `logger.info`.
  */
  debugPrintReports?: boolean;
  /**
@@ -307,12 +361,6 @@ export interface ApolloServerPluginUsageReportingOptions<
  * about how the signature relates to the operation you executed.
  */
  calculateSignature?: (ast: DocumentNode, operationName: string) => string;
- /**
- * This option includes extra data in reports that helps Apollo validate the
- * stats generation code in this plugin. Do not set it; the only impact on
- * your app will be a decrease in performance.
- */
- internal_includeTracesContributingToStats?: boolean;
  //#endregion
 }