Make it possible to override remoteServiceName in ObservationFilter

[jonatan-ivanov]

Right now it is not possible to set the remoteServiceName of a Span through an Observation.Context (SenderContext) using an ObservationFilter since the remoteServiceName is set on the Span before the ObservationFilter is called.
(In PropagatingSenderTracingObservationHandler is set on the Span from the onStart method while ObservationFilter is called from onStop.)

Here's what happens:

1. SenderContext and the Observation is created, at this point the instrumentation should know the remoteServiceName.
2. The Observation is started and PropagatingSenderTracingObservationHandler's onStart method is called.
3. PropagatingSenderTracingObservationHandler sets remoteServiceName on the Span.
4. The Observation is stopped and the ObservationFilter is called.
5. At this point even if the ObservationFilter modifies the remoteServiceName, it has no effect since it is called after remoteServiceName was set on the Span.

This is true for remoteServiceAddress and remoteIpAndPort too.
Also see: spring-projects/spring-boot#39432

[marcingrzejszczak]

Why do you think this should be done via an ObservationFilter? It is too late to set that, in Brave and OTel you can't modify these values in any other time than start of the span. We can create an additional method to override in PropagatingSenderTracingObservationHandler that has access to the Span.Builder. That way it would be possible to override these values (OTOH they are already in the Observation Context so I don't know how that would help).

The way to do it as late as possible would be to call a SpanFilter and modify the remote service name and ip and port. At this point both Brave and OTel allow that.