-
Notifications
You must be signed in to change notification settings - Fork 350
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proto schema registry refresh optimizations #1040
Conversation
backend/pkg/schema/client.go
Outdated
errors := make([]error, len(subjectsRes.Subjects)) | ||
hasErrors := atomic.Bool{} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should do errors := make([]error, 0, len(subjectsRes.Subjects))
and then we can check the length of errors to see if we have errors. I think in most cases we do not expect every single subject to return an error.
It feels a bit weird to return a slice of schemas and a slice of errors. At this point we can probably just refactor the function to return schema + error via a channel and handle the error and returned schema on the receiving side.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The intent is to be able to get all the schemas we successfully fetched, even if we failed to retrieve some subset of all the schemas in SR. So we return all the successfully fetched schemas and a collection of errors.
This is the case for both GetSchemasIndividually()
and GetSchemas()
, and then GetProtoDescriptors()
we are able to log all the failed schemas, but still be able to parse and compile the successfully retrieved ones so we could potentially use them.
s.requestGroup.Forget(key) | ||
return nil, fmt.Errorf("failed to get schema from registry: %w", err) | ||
_, err, _ := s.requestGroup.Do(key, func() (any, error) { | ||
schemasRes, errs := s.registryClient.GetSchemas(ctx, false) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
schemaRes will be nil as far as I can tell if all retrieved schemas errored (e.g. schema registry not reachable). I think this func will still work if this is the case, but would be good to test the case where schemaRes is nil.
Rather than using singleflight we could use https://github.com/twmb/go-cache to simplify some things, but I assume you tried to keep as much of the original code as possible
Iterative work for optimizing some of the proto schema registry refreshing.
Changes
GetSchemasBySubject()
,GetSchemas()
,GetSchemasIndividually()
to return pointer types.gci
seems to incorrectly sort the"maps"
import. Updated the tooling hoping it would address the issue, but it didn't seem to help, so I commented that step out for now.I will try and add some tests for this soon.