[collector] use the otelcol-k8s distribution of the collector

[TylerHelmuth]

The OpenTelemetry Collector has a new Kubernetes-specific distribution: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s.

We should update the collector chart to use this distribution instead of Contrib since it is smaller and using Contrib is against Collector SIG best practices.

This would be a significant breaking change since we use Contrib today and the K8s distro contains many less components. Our default config wouldn't break, but users who are expecting certain receivers to be present that aren't included in the k8s distro will break.

We'll need to handled this via a feature gate, similar to how we handled GOMEMLIMIT:

1. Introduce a new root-level field that, when set, makes the k8s distribution the used image. Also add a NOTES.txt warning if not using the flag
2. Much later (TBD) switch the flag to be "on" by default.
3. Much later again, remove the flag.

Since the collector image can be directly set, maybe steps 2 and 3 can become "switch the default image" with clear instructions how to set the image/command back to contrib.

@open-telemetry/helm-approvers how long should we wait between steps 1 and 2? 3 months? 6 months?

[dmitryax]

I'm not sure why we need a config option (FG) just to control another option, image.repository. If users don't like the change, they can easily set image.repository back to otel/opentelemetry-collector-contrib. We just need to clearly document the change in the release notes or upgrade guidelines. I don't think we need a feature gate here.

[povilasv]

I agree with @dmitryax we can also add NOTES.txt and document this in changelog. But I would vote for making it easy to change back to contrib if a user wants.

[TylerHelmuth]

The reason I don't want to do it in one breaking change is that the breaking change won't happen on install but after the collector tried to start up.

For breaking changes that fail on install it is a lot easier to write out a meaningful error message and guide the user towards what changed. In this instance, the chart will install fine but the collector won't start, and there isn't any way for us to indicate that it is bc of the new image, it will be the collector's error message.

I am worried about the end-user experience of doing an upgrade and then the collector doesn't startup.

[dmitryax]

We can have an intermediate step with no default image.repository. That will fail on install/upgrade unless users specify an image explicitly. We can recommend the new image in the failure message.

[TylerHelmuth]

I'm good with that. And I think it should be its own PR, not the 0.98.0 bump.