feat: Instrument Runner Service (#602)

Runner is lacking instrumentation. It is responsible for many things and
it's become hard to understand what tasks contribute to the overall
latency of an indexer. In addition, we are now at a point where we need
to drive down latencies to facilitate new * indexer use cases such as
access keys.

I've chosen to instrument Runner with OpenTelemetry. Tracing generally
requires 3 items: An instrumented service, a trace collector, and a
trace visualizer. The service is responsible for collecting and
transmitting trace data to the collector. The collector should be able
to receive trace data with little fuss to prevent performance impacts to
the instrumented service. The collector then processes the trace data
and transmits the processed data to the visualizer. The visualizer
visualizes trace data and allows for filtering on traces.

The benefit of OpenTelemetry over other options like Zipkin and Jaeger
is that GCP already supports ingesting OpenTelemetry data. As such, we
don't need to provision a collector ourselves, and can instead leverage
GCP's existing collector & visualizer Tracing service. For local
development, traces can be output to console, a Zipkin all-in-one
container or to GCP (Requires Cloud Trace Agent role and specifying
project ID). This is done by simply initializing the NodeSDK
differently.

In addition, we do not want to enable traces in prod yet, so by not
specifying any exporter. This creates a No-Op Trace Exporter which won't
attempt to record traces.

No code changes were made changing code execution path. All tests pass
with no changes, aside from having to replace snapshots due to changes
in tabbing of mutation strings. I have manually verified mutation
strings are still the same by stripping whitespace and checking against
original.

Author: Darun Seethammagari

docker-compose.yml

@@ -56,6 +56,11 @@ services:
       AWS_ACCESS_KEY_ID:
       AWS_SECRET_ACCESS_KEY:
       GRPC_SERVER_PORT: 7001
+      PREFETCH_QUEUE_LIMIT: 10
+      TRACING_EXPORTER: ZIPKIN # CONSOLE, GCP, ZIPKIN, or NONE
+      ZIPKIN_ENDPOINT: http://zipkin:9411/api/v2/spans
+      GCP_PROJECT_ID: 
+      TRACING_SAMPLE_RATE: 0.1
     ports:
       - "7001:7001"
 
@@ -104,6 +109,7 @@ services:
       HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
       HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
       HASURA_GRAPHQL_AUTH_HOOK: http://hasura-auth:4000/auth
+
   grafana:
     image: grafana/grafana
     volumes:
@@ -112,13 +118,20 @@ services:
       - "3000:3000"
     environment:
       - GF_SECURITY_ADMIN_PASSWORD=secret
-    
+
   prometheus:
     image: prom/prometheus
     volumes:
       - ./prometheus.yml:/etc/prometheus/prometheus.yml
     ports:
       - "9090:9090"
+
+  zipkin:
+    image: openzipkin/zipkin
+    ports:
+      - "9411:9411"
+    environment:
+      - STORAGE_TYPE=mem
 
 volumes:
   postgres: