Add jaeger-v2 single binary based on OTEL collector (jaegertracing#4766)

## Which problem is this PR solving?
- Third prototype of "Jaeger-v2"
- Another alternative approach to jaegertracing#3500

## Description of the changes
- Adds a new binary `jaeger-v2` using OTEL Collector framework
- Minimal amount of extensions is included, to mimic what
`jaeger-collector` normally has
- It will combine all previous functions of agent/collector/query in one
binary, but controllable via config file

```
$ go run -tags=ui ./cmd/jaeger-v2 --config ./cmd/jaeger-v2/config.yaml
```

## Roadmap


https://docs.google.com/document/d/1s4_6VgAS7qAVp6iEm5KYvpiGw3h2Ja5T5HpKo29iv00/edit

## Design

* the ingestion and storing of traces will be done via standard
receivers/processors/exporters OTEL Collector components
* the jaeger-query and UI are implemented as `jaeger_query` extension
(already working in this PR)

### Storage
In order to keep the flexibility of mixing & matching storage
implementations, all backends can be configured via `jaeger_storage`
extension (we may need to add `jaeger_metrics_storage` extension in the
future). It might look like this:
```yaml
jaeger_storage:
  memory:  # defines Factory
    memstore:
      max_traces: 100000
  cassandra:
    cassandra_primary:
      servers: [...]
      namespace: jaeger
    cassandra_archive:
      servers: [...]
      namespace: jaeger_archive
```

The `jaeger_query` extension then references specific storage factories
by name:
```yaml
  jaeger_query:
    trace_storage: memstore
    dependencies: something_else
    metrics_store: prometheus_store
```

It's not clear yet if `jaeger_query` extension should simply subsume
`jaeger_storage` extension, because Query is the only one that needs
this _generic_ access to storage, while things like exporters or Kafka
ingester (receiver) always deal with a single implementation (because
OTEL Coll pipeline allows to connect them with each other, which is not
possible with extensions).

## Trade-offs
- This not using OTEL Collector builder `ocb`. That means people won't
be able to assemble a different version of the collector with other
extensions.
- We may want to support `ocb` in the future, as it makes it easier to
write custom in-process exporters for custom storage. It will require
converting all the components into their own modules.

## Next steps

* [x] Get feedback from the community on the approach
* [x] Fully implement all-in-one by wiring receivers / exporters
correctly

## Open Questions
* How can we implement all-in-one equivalent that can be run without any
config file?
* Do we want
[healthcheckextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/healthcheckextension/README.md)
to be included by default?
* Investigate startup error `2023-09-23T19:55:46.661-0400 warn
zapgrpc/zapgrpc.go:195 [core] [Channel #2 SubChannel #3] grpc:
addrConn.createTransport failed to connect to {Addr: ":16685",
ServerName: "localhost:16685", }. Err: connection error: desc =
"transport: Error while dialing: dial tcp :16685: connect: connection
refused" {"grpc_log": true}`

---------

Signed-off-by: Yuri Shkuro <github@ysh.us>
Signed-off-by: Yuri Shkuro <yurishkuro@users.noreply.github.com>
Co-authored-by: Albert <26584478+albertteoh@users.noreply.github.com>

Makefile

@@ -219,14 +219,18 @@ jaeger-ui/packages/jaeger-ui/build/index.html:
 .PHONY: rebuild-ui
 rebuild-ui:
 	bash ./scripts/rebuild-ui.sh
-	
+
 .PHONY: build-all-in-one-linux
 build-all-in-one-linux:
 	GOOS=linux $(MAKE) build-all-in-one
 
 build-all-in-one-debug build-agent-debug build-query-debug build-collector-debug build-ingester-debug build-remote-storage-debug: DISABLE_OPTIMIZATIONS = -gcflags="all=-N -l"
 build-all-in-one-debug build-agent-debug build-query-debug build-collector-debug build-ingester-debug build-remote-storage-debug: SUFFIX = -debug
 
+.PHONY: build-jaeger-v2
+build-jaeger-v2:
+	$(GOBUILD) $(DISABLE_OPTIMIZATIONS) -tags ui -o ./cmd/jaeger-v2/jaeger-v2$(SUFFIX)-$(GOOS)-$(GOARCH) $(BUILD_INFO) ./cmd/jaeger-v2/main.go
+
 .PHONY: build-all-in-one build-all-in-one-debug
 build-all-in-one build-all-in-one-debug: build-ui
 	$(GOBUILD) $(DISABLE_OPTIMIZATIONS) -tags ui -o ./cmd/all-in-one/all-in-one$(SUFFIX)-$(GOOS)-$(GOARCH) $(BUILD_INFO) ./cmd/all-in-one/main.go

cmd/jaeger-v2/.gitignore

@@ -0,0 +1,2 @@
+jaeger-v2-*-*
+jaeger-v2

cmd/jaeger-v2/config.yaml

@@ -0,0 +1,45 @@
+service:
+  extensions: [jaeger_storage, jaeger_query]
+  pipelines:
+    traces:
+      receivers: [otlp, jaeger, zipkin]
+      processors: [batch]
+      exporters: [jaeger_storage_exporter]
+
+extensions:
+  # health_check:
+  # pprof:
+  #   endpoint: 0.0.0.0:1777
+  # zpages:
+  #   endpoint: 0.0.0.0:55679
+
+  jaeger_query:
+    trace_storage: memstore
+
+  jaeger_storage:
+    memory:
+      memstore:
+        max_traces: 100000
+
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+      http:
+
+  jaeger:
+    protocols:
+      grpc:
+      thrift_binary:
+      thrift_compact:
+      thrift_http:
+
+  zipkin:
+
+processors:
+  batch:
+
+exporters:
+  jaeger_storage_exporter:
+    trace_storage: memstore

cmd/jaeger-v2/internal/.nocover

@@ -0,0 +1,2 @@
+FIXME
+

cmd/jaeger-v2/internal/command.go

@@ -0,0 +1,43 @@
+// Copyright (c) 2023 The Jaeger Authors.
+// SPDX-License-Identifier: Apache-2.0
+
+package internal
+
+import (
+	"log"
+
+	"github.com/spf13/cobra"
+	"go.opentelemetry.io/collector/component"
+	"go.opentelemetry.io/collector/otelcol"
+
+	"github.com/jaegertracing/jaeger/pkg/version"
+)
+
+const description = "Jaeger backend v2"
+
+func Command() *cobra.Command {
+	factories, err := components()
+	if err != nil {
+		log.Fatalf("failed to build components: %v", err)
+	}
+
+	versionInfo := version.Get()
+
+	info := component.BuildInfo{
+		Command:     "jaeger-v2",
+		Description: description,
+		Version:     versionInfo.GitVersion,
+	}
+
+	cmd := otelcol.NewCommand(
+		otelcol.CollectorSettings{
+			BuildInfo: info,
+			Factories: factories,
+		},
+	)
+
+	cmd.Short = description
+	cmd.Long = description
+
+	return cmd
+}

cmd/jaeger-v2/internal/components.go

@@ -0,0 +1,98 @@
+// Copyright (c) 2023 The Jaeger Authors.
+// SPDX-License-Identifier: Apache-2.0
+
+package internal
+
+import (
+	"github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector"
+	"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/kafkaexporter"
+	"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver"
+	"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver"
+	"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver"
+	"go.opentelemetry.io/collector/connector"
+	"go.opentelemetry.io/collector/connector/forwardconnector"
+	"go.opentelemetry.io/collector/exporter"
+	"go.opentelemetry.io/collector/exporter/loggingexporter"
+	"go.opentelemetry.io/collector/exporter/otlpexporter"
+	"go.opentelemetry.io/collector/exporter/otlphttpexporter"
+	"go.opentelemetry.io/collector/extension"
+	"go.opentelemetry.io/collector/extension/ballastextension"
+	"go.opentelemetry.io/collector/extension/zpagesextension"
+	"go.opentelemetry.io/collector/otelcol"
+	"go.opentelemetry.io/collector/processor"
+	"go.opentelemetry.io/collector/processor/batchprocessor"
+	"go.opentelemetry.io/collector/processor/memorylimiterprocessor"
+	"go.opentelemetry.io/collector/receiver"
+	"go.opentelemetry.io/collector/receiver/otlpreceiver"
+
+	"github.com/jaegertracing/jaeger/cmd/jaeger-v2/internal/exporters/storageexporter"
+	"github.com/jaegertracing/jaeger/cmd/jaeger-v2/internal/extension/jaegerquery"
+	"github.com/jaegertracing/jaeger/cmd/jaeger-v2/internal/extension/jaegerstorage"
+)
+
+func components() (otelcol.Factories, error) {
+	var err error
+	factories := otelcol.Factories{}
+
+	factories.Extensions, err = extension.MakeFactoryMap(
+		// standard
+		ballastextension.NewFactory(),
+		zpagesextension.NewFactory(),
+		// add-ons
+		jaegerquery.NewFactory(),
+		jaegerstorage.NewFactory(),
+		// TODO add adaptive sampling
+	)
+	if err != nil {
+		return otelcol.Factories{}, err
+	}
+
+	factories.Receivers, err = receiver.MakeFactoryMap(
+		// standard
+		otlpreceiver.NewFactory(),
+		// add-ons
+		jaegerreceiver.NewFactory(),
+		kafkareceiver.NewFactory(),
+		zipkinreceiver.NewFactory(),
+	)
+	if err != nil {
+		return otelcol.Factories{}, err
+	}
+
+	factories.Exporters, err = exporter.MakeFactoryMap(
+		// standard
+		loggingexporter.NewFactory(),
+		otlpexporter.NewFactory(),
+		otlphttpexporter.NewFactory(),
+		// add-ons
+		storageexporter.NewFactory(), // generic exporter to Jaeger v1 spanstore.SpanWriter
+		kafkaexporter.NewFactory(),
+		// elasticsearch.NewFactory(),
+	)
+	if err != nil {
+		return otelcol.Factories{}, err
+	}
+
+	factories.Processors, err = processor.MakeFactoryMap(
+		// standard
+		batchprocessor.NewFactory(),
+		memorylimiterprocessor.NewFactory(),
+		// add-ons
+		// TODO add adaptive sampling
+	)
+	if err != nil {
+		return otelcol.Factories{}, err
+	}
+
+	factories.Connectors, err = connector.MakeFactoryMap(
+		// standard
+		forwardconnector.NewFactory(),
+		// add-ons
+		spanmetricsconnector.NewFactory(),
+	)
+	if err != nil {
+		return otelcol.Factories{}, err
+	}
+
+	return factories, nil
+}

cmd/jaeger-v2/internal/exporters/storageexporter/.nocover

@@ -0,0 +1,2 @@
+FIXME
+

cmd/jaeger-v2/internal/exporters/storageexporter/README.md

@@ -0,0 +1,17 @@
+# jaeger_storage_exporter
+
+This module implements `exporter.Traces` and writes spans into Jaeger native `spanstore.SpanWriter`, that it obtains from the [jaeger_storage](../../extension/jaegerstorage/) extension. This is primarily needed to wire a memory storage into the exporter pipeline (used for all-in-one), but the design of the exporter is such that it can do this for any V1 storage implementation.
+
+## Configuration
+
+```yaml
+exporters:
+  jaeger_storage_exporter:
+    trace_storage: memstore
+
+extensions:
+  jaeger_storage:
+    memory:
+      memstore:
+        max_traces: 100000
+```

cmd/jaeger-v2/internal/exporters/storageexporter/config.go

@@ -0,0 +1,24 @@
+// Copyright (c) 2023 The Jaeger Authors.
+// SPDX-License-Identifier: Apache-2.0
+
+package storageexporter
+
+import (
+	"github.com/asaskevich/govalidator"
+	"go.opentelemetry.io/collector/component"
+)
+
+var (
+	_ component.Config          = (*Config)(nil)
+	_ component.ConfigValidator = (*Config)(nil)
+)
+
+// Config defines configuration for jaeger_storage_exporter.
+type Config struct {
+	TraceStorage string `valid:"required" mapstructure:"trace_storage"`
+}
+
+func (cfg *Config) Validate() error {
+	_, err := govalidator.ValidateStruct(cfg)
+	return err
+}

cmd/jaeger-v2/internal/exporters/storageexporter/exporter.go

@@ -0,0 +1,66 @@
+// Copyright (c) 2023 The Jaeger Authors.
+// SPDX-License-Identifier: Apache-2.0
+
+package storageexporter
+
+import (
+	"context"
+	"errors"
+	"fmt"
+
+	otlp2jaeger "github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/jaeger"
+	"go.opentelemetry.io/collector/component"
+	"go.opentelemetry.io/collector/pdata/ptrace"
+	"go.uber.org/zap"
+
+	"github.com/jaegertracing/jaeger/cmd/jaeger-v2/internal/extension/jaegerstorage"
+	"github.com/jaegertracing/jaeger/storage/spanstore"
+)
+
+type storageExporter struct {
+	config     *Config
+	logger     *zap.Logger
+	spanWriter spanstore.Writer
+}
+
+func newExporter(config *Config, otel component.TelemetrySettings) *storageExporter {
+	return &storageExporter{
+		config: config,
+		logger: otel.Logger,
+	}
+}
+
+func (exp *storageExporter) start(_ context.Context, host component.Host) error {
+	f, err := jaegerstorage.GetStorageFactory(exp.config.TraceStorage, host)
+	if err != nil {
+		return fmt.Errorf("cannot find storage factory: %w", err)
+	}
+
+	if exp.spanWriter, err = f.CreateSpanWriter(); err != nil {
+		return fmt.Errorf("cannot create span writer: %w", err)
+	}
+
+	return nil
+}
+
+func (exp *storageExporter) close(_ context.Context) error {
+	// span writer is not closable
+	return nil
+}
+
+func (exp *storageExporter) pushTraces(ctx context.Context, td ptrace.Traces) error {
+	batches, err := otlp2jaeger.ProtoFromTraces(td)
+	if err != nil {
+		return fmt.Errorf("cannot transform OTLP traces to Jaeger format: %w", err)
+	}
+	var errs []error
+	for _, batch := range batches {
+		for _, span := range batch.Spans {
+			if span.Process == nil {
+				span.Process = batch.Process
+			}
+			errs = append(errs, exp.spanWriter.WriteSpan(ctx, span))
+		}
+	}
+	return errors.Join(errs...)
+}

cmd/jaeger-v2/internal/exporters/storageexporter/factory.go

@@ -0,0 +1,44 @@
+// Copyright (c) 2023 The Jaeger Authors.
+// SPDX-License-Identifier: Apache-2.0
+
+package storageexporter
+
+import (
+	"context"
+
+	"go.opentelemetry.io/collector/component"
+	"go.opentelemetry.io/collector/consumer"
+	"go.opentelemetry.io/collector/exporter"
+	"go.opentelemetry.io/collector/exporter/exporterhelper"
+)
+
+// componentType is the name of this extension in configuration.
+const componentType = component.Type("jaeger_storage_exporter")
+
+// NewFactory creates a factory for jaeger_storage_exporter.
+func NewFactory() exporter.Factory {
+	return exporter.NewFactory(
+		componentType,
+		createDefaultConfig,
+		exporter.WithTraces(createTracesExporter, component.StabilityLevelDevelopment),
+	)
+}
+
+func createDefaultConfig() component.Config {
+	return &Config{}
+}
+
+func createTracesExporter(ctx context.Context, set exporter.CreateSettings, config component.Config) (exporter.Traces, error) {
+	cfg := config.(*Config)
+	ex := newExporter(cfg, set.TelemetrySettings)
+	return exporterhelper.NewTracesExporter(ctx, set, cfg,
+		ex.pushTraces,
+		exporterhelper.WithCapabilities(consumer.Capabilities{MutatesData: false}),
+		// Disable Timeout/RetryOnFailure and SendingQueue
+		exporterhelper.WithTimeout(exporterhelper.TimeoutSettings{Timeout: 0}),
+		exporterhelper.WithRetry(exporterhelper.RetrySettings{Enabled: false}),
+		exporterhelper.WithQueue(exporterhelper.QueueSettings{Enabled: false}),
+		exporterhelper.WithStart(ex.start),
+		exporterhelper.WithShutdown(ex.close),
+	)
+}

cmd/jaeger-v2/internal/extension/jaegerquery/.nocover

@@ -0,0 +1,2 @@
+FIXME
+

cmd/jaeger-v2/internal/extension/jaegerquery/README.md

@@ -0,0 +1,15 @@
+# jaeger_query
+
+This extension implements traditional `jaeger-query` service (including Jaeger UI). Because it is not a part of the collector pipeline, it needs access to a storage backend, which can be potentially shared with an exporter (e.g. in order to implement the `all-in-one` functionality). For this reason it depends on the [jaeger_storage](../jaegerstorage/) extension.
+
+## Configuration
+
+This is work in progress, most of the usual settings of `jaeger-query` are not supported yet.
+
+```yaml
+jaeger_query:
+    trace_storage: cassandra_primary
+    trace_archive: cassandra_archive
+    dependencies: memstore
+    metrics_store: prometheus_store
+```