Allow Jaeger's GRPC handler to flow the tenant from an HTTP header #3688

esnible · 2022-05-17T14:38:15Z

Signed-off-by: Ed Snible snible@us.ibm.com

If Jaeger is started with for example --collector.tenant-header=x-tenant, flow the value of the x-tenant header to the tenant parameter for span processing.

If Jaeger is not started with --collector.tenant-header, this is essentially a no-op.

cc @pavolloffay

codecov · 2022-05-17T14:45:20Z

Codecov Report

Merging #3688 (c2551c7) into main (557ff71) will decrease coverage by 0.04%.
The diff coverage is 96.47%.

@@            Coverage Diff             @@
##             main    #3688      +/-   ##
==========================================
- Coverage   97.49%   97.45%   -0.05%     
==========================================
  Files         269      271       +2     
  Lines       15934    16014      +80     
==========================================
+ Hits        15535    15606      +71     
- Misses        315      321       +6     
- Partials       84       87       +3

Impacted Files	Coverage Δ
cmd/collector/app/flags/flags.go	`96.73% <57.14%> (-3.27%)`	⬇️
cmd/collector/app/handler/grpc_handler.go	`100.00% <100.00%> (ø)`
cmd/collector/app/handler/otlp_receiver.go	`100.00% <100.00%> (ø)`
cmd/collector/app/span_handler_builder.go	`100.00% <100.00%> (ø)`
pkg/config/tenancy/flags.go	`100.00% <100.00%> (ø)`
pkg/config/tenancy/tenancy.go	`100.00% <100.00%> (ø)`
cmd/query/app/static_handler.go	`94.01% <0.00%> (-3.60%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 557ff71...c2551c7. Read the comment docs.

yurishkuro · 2022-05-17T14:55:03Z

cmd/collector/app/handler/grpc_handler.go

+	}
+	md, ok := metadata.FromIncomingContext(ctx)
+	if !ok {
+		return "", status.Errorf(codes.Internal, "missing metadata")


is it an error to be missing metadata?

For Google's GRPC implementation there is always metadata, and I was thinking this is a 500/unexpected situation.

This code path is hit when Jaeger has been deployed with a requirement for a tenant header, and GRPC hasn't supplied metadata.
There is no security-related reason to treat missing metadata as different from valid metadata lacking the tenant header. Shall I rework this to combine missing-metadata and missing-header handling?

fair enough, but then I would not treat it as codes.Internal, but similar to other ones and I would alter the error message, e.g. "missing metadata, tenant header is required"

yurishkuro · 2022-05-17T14:59:04Z

cmd/collector/app/builder_flags.go

@@ -40,6 +40,7 @@ const (
 	collectorGRPCMaxReceiveMessageLength = "collector.grpc-server.max-message-size"
 	collectorMaxConnectionAge            = "collector.grpc-server.max-connection-age"
 	collectorMaxConnectionAgeGrace       = "collector.grpc-server.max-connection-age-grace"
+	collectorTenantHeader                = "collector.tenant-header"


gRPC is not the only way to submit data to collector. Is the intention to require the same header name across transports?

@yurishkuro I'll need your advice on this. I have only used the GRPC transport. I had assumed that most users only use a single transport. If some users enable multiple transports, and wish to use different tenancy headers, I am happy to change it.

I'm expecting to implement flowing the header from HTTP to the internal tenant for GRPC only, as that is the only transport I use and know how to test. I hope to leave the other transports as a follow-on for others who desire tenancy on the other transports.

Are you aware of any existing header conventions for the tenant? Is x-tenant suitable to both HTTP and gRPC?

My suggestion:

keep a single header for now, we can extend this in the future if needed

I would like to have a choke point in the pipeline that would know that a tenancy is required and would verify that the Context contains a valid tenant. This way, even if we don't implement tenancy in HTTP headers immediately, the system would still be otherwise "safe", i.e. it would not accept data that is missing the tenant value.

The last point requires a configuration, which I would suggest to consolidate, e.g. something like

multi_tenancy: enabled: true header: "x-tenant" # this could be the default value instead of always requiring it tenants: ["a", "b", "c"] # required

Do we need to specify tenants at the deployment time? I wanted to avoid restarting Jaeger (and changing deployments) when a new tenant is added.

The configuration can be extended to accept tenants in other forms, like watching a file or by integrating with another source. This would be a good opportunity to start introducing a control plane where tenants can be provided externally via api call, such that the integration with the source can be decoupled from Jaeger code.

I would propose tackling the tenant restrictions in a separate PR.

More sophisticated deployments should do tenant validation in the auth layer (proxy). Which makes me think do we even need Jaeger to do this validation? A valid requirement for predefined tenants might be if the storage implementation needs to do some initialization per tenant. In the case of ES it should not be required, but maybe other storages like C* need to create a new DB that cannot be done in-flight of the first store call.

The validation I'm most concerned about is not whether tenant is valid, but whether it is sent at all. Without that we open a hole that could allow other tenants to read your data.

I agree that there needs to be a stronger authentication somewhere other than a header, which is trivial to spoof.

@jkowall any experience to share here?

I will relax the handling of valid tenants so that if multi_tenancy.tenants is undefined any tenant will pass. This will allow the spanwriter to reject spans. In the future we can provide a way for the writer to reject earlier.

Storage might have rules about what a valid tenant can be that we can't express on a list. For example, some storage might reject tenants that include non-ASCII characters. Or some storage might want to hot-enable/disable a tenant.

yurishkuro · 2022-05-19T21:51:25Z

pkg/config/tenancy/flags.go

+	validTenants   = "multi_tenancy.tenants"
+)
+
+// TenancyFlagsConfig describes which CLI flags for TLS client should be generated.


the type does not seem necessary, we could just have global functions. Or, at least, it should be the same type as Options, that's the pattern we used, i.e. the config/options type exposes functions to initialize itself via CLI.

yurishkuro · 2022-05-19T21:54:28Z

pkg/config/tenancy/tenancy.go

+	tenants map[string]bool
+}
+
+func (tl tenantList) Valid(candidate string) bool {


are you sure you want by-value receiver here? There's going to be a heap allocation anyway when the struct is converted into Guard interface.

yurishkuro · 2022-05-19T21:57:45Z

cmd/collector/app/handler/grpc_handler.go

+	var tenant string
+	if g.tenancyConfig.Enabled {
+		var err error
+		tenant, err = g.validateTenant(ctx)
+		if err != nil {
+			g.logger.Error("rejecting spans (tenancy)", zap.Error(err))
+			return nil, err
+		}
+	}


suggest moving this to validateTenant

Suggested change

var tenant string

if g.tenancyConfig.Enabled {

var err error

tenant, err = g.validateTenant(ctx)

if err != nil {

g.logger.Error("rejecting spans (tenancy)", zap.Error(err))

return nil, err

}

}

tenant, err := g.validateTenant(ctx)

if err != nil {

g.logger.Error("failed to validate tenant, rejecting spans", zap.Error(err))

return nil, err

}

yurishkuro · 2022-05-19T22:00:42Z

pkg/config/tenancy/tenancy.go

+type TenancyConfig struct {
+	Enabled bool
+	Header  string
+	Valid   Guard


this looks weird in the usage: config.Valid.Valid(). How about:

type TenancyConfig struct { ... guard Guard } func (c *TenancyConfig) Valid(v string) bool { return c.guard.Valid() }

especially since the current Valid variable is force-assigned in the constructor & cannot be user-supplied.

yurishkuro

lgtm

pavolloffay · 2022-05-23T14:27:03Z

pkg/config/tenancy/flags.go

+func AddFlags(flags *flag.FlagSet) {
+	flags.Bool(tenancyEnabled, false, "Enable tenancy header when receiving or querying")
+	flags.String(tenancyHeader, "x-tenant", "HTTP header carrying tenant")
+	flags.String(validTenants, "", "Acceptable tenants")


It should mention it is a comma-separated list. What if the flag is not defined?

pavolloffay · 2022-05-25T11:46:41Z

cmd/collector/app/handler/grpc_handler_test.go

+		},
+	}
+	for _, test := range tests {
+		_, err := client.PostSpans(test.ctx, &api_v2.PostSpansRequest{


the individual test cases should be executed separately via t.Run

I followed the format of TestPostSpans(). No tests in cmd/collector/app/handler use t.Run().

Anyway, I made the changes.

pavolloffay · 2022-05-25T11:47:41Z

cmd/collector/app/handler/grpc_handler_test.go

+		Tenants: validTenants,
+	}))
+	for _, test := range tests {
+		tenant, err := handler.validateTenant(test.ctx)


the same here, test cases should be run via t.Run

pavolloffay · 2022-05-25T11:50:08Z

pkg/config/tenancy/flags.go

+// AddFlags adds flags for tenancy to the FlagSet.
+func AddFlags(flags *flag.FlagSet) {
+	flags.Bool(tenancyEnabled, false, "Enable tenancy header when receiving or querying")
+	flags.String(tenancyHeader, "x-tenant", "HTTP header carrying tenant")


As a first version, we could have a single flag tenant-header which would enable tenancy if it is defined.

Internally we will most likely not use the predefined list of tenants (e.g. in the observatorium project) and if the storage layer does not need to have access to the tenant names I find this configuration redundant.

I think we need to define flags more forward looking than just what works with the first version. Using .enabled is already an established pattern in Jaeger flags.

Signed-off-by: Ed Snible <snible@us.ibm.com>

esnible · 2022-05-31T13:48:59Z

I struggled with re-basing after #3707 . Instead I copied the code changes and tests and manually applied them to main.

yurishkuro · 2022-05-31T14:32:26Z

cmd/collector/app/handler/grpc_handler.go

 	batch := &r.Batch
-	err := g.batchConsumer.consume(batch)
+	err = g.batchConsumer.consume(batch, tenant)


I would suggest changing the signature to consume(context, batch) and handling tenancy inside. This way it can be reused in OTLP receiver.

(ok to do in another PR if you prefer)

…aegertracing#3688) * Flow tenant from GRPC PostSpans header through processors Signed-off-by: Ed Snible <snible@us.ibm.com> * Restore accidentally deleted test case Signed-off-by: Ed Snible <snible@us.ibm.com> Signed-off-by: Albert Teoh <see.kwang.teoh@gmail.com>

esnible requested a review from a team as a code owner May 17, 2022 14:38

esnible requested a review from vprithvi May 17, 2022 14:38

yurishkuro reviewed May 17, 2022

View reviewed changes

esnible force-pushed the flow-tenant-from-grpc branch from cec2799 to c6db258 Compare May 19, 2022 20:25

yurishkuro reviewed May 19, 2022

View reviewed changes

yurishkuro reviewed May 20, 2022

View reviewed changes

esnible force-pushed the flow-tenant-from-grpc branch 2 times, most recently from 9532548 to ecba97d Compare May 23, 2022 13:33

pavolloffay reviewed May 25, 2022

View reviewed changes

esnible mentioned this pull request May 25, 2022

Flow tenant from query #3700

Closed

Flow tenant from GRPC PostSpans header through processors

e982edf

Signed-off-by: Ed Snible <snible@us.ibm.com>

esnible force-pushed the flow-tenant-from-grpc branch from 0c79158 to e982edf Compare May 31, 2022 13:34

Restore accidentally deleted test case

c2551c7

Signed-off-by: Ed Snible <snible@us.ibm.com>

yurishkuro reviewed May 31, 2022

View reviewed changes

yurishkuro approved these changes May 31, 2022

View reviewed changes

yurishkuro merged commit a184e94 into jaegertracing:main May 31, 2022

esnible mentioned this pull request May 31, 2022

Refactor tenancy checking from gRPC to gRPC batch consumer #3718

Merged

This was referenced Jun 8, 2022

Optionally supply tenancy when reporting spans #3742

Closed

Optionally supply tenancy when reporting spans v2 #3750

Closed

esnible mentioned this pull request Aug 18, 2022

[Feature]: The rest of multi-tenancy #3881

Open

pavolloffay mentioned this pull request Jan 10, 2023

Support tenant header propagation in query service #4141

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow Jaeger's GRPC handler to flow the tenant from an HTTP header #3688

Allow Jaeger's GRPC handler to flow the tenant from an HTTP header #3688

esnible commented May 17, 2022

codecov bot commented May 17, 2022 •

edited

Loading

yurishkuro May 17, 2022

esnible May 17, 2022

yurishkuro May 17, 2022

yurishkuro May 17, 2022

esnible May 17, 2022

yurishkuro May 17, 2022

pavolloffay May 19, 2022

yurishkuro May 19, 2022

pavolloffay May 20, 2022

yurishkuro May 20, 2022

esnible May 20, 2022

yurishkuro May 19, 2022

yurishkuro May 19, 2022

yurishkuro May 19, 2022

yurishkuro May 19, 2022

yurishkuro left a comment

pavolloffay May 23, 2022

pavolloffay May 25, 2022

esnible May 25, 2022 •

edited

Loading

pavolloffay May 25, 2022

pavolloffay May 25, 2022

yurishkuro May 25, 2022

esnible commented May 31, 2022

yurishkuro May 31, 2022

yurishkuro May 31, 2022

Allow Jaeger's GRPC handler to flow the tenant from an HTTP header #3688

Allow Jaeger's GRPC handler to flow the tenant from an HTTP header #3688

Conversation

esnible commented May 17, 2022

codecov bot commented May 17, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yurishkuro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

esnible May 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

esnible commented May 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 17, 2022 •

edited

Loading

esnible May 25, 2022 •

edited

Loading