Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix performance for large schema writes in V1Alpha1 #837

Merged
merged 1 commit into from
Sep 22, 2022

Conversation

josephschorr
Copy link
Member

This will hopefully reduce the flakiness of the E2E test suite, which writes large batches of definitions via the V1Alpha1 WriteSchema call

@github-actions github-actions bot added area/api v1 Affects the v1 API area/datastore Affects the storage system labels Sep 21, 2022
@josephschorr josephschorr force-pushed the fix-v1alpha1-schema-perf branch from 8ee0c08 to 5b1ba4e Compare September 22, 2022 00:09
This will hopefully reduce the flakiness of the E2E test suite, which writes large batches of definitions via the V1Alpha1 WriteSchema call
@josephschorr josephschorr force-pushed the fix-v1alpha1-schema-perf branch from 5b1ba4e to c331676 Compare September 22, 2022 00:30
@josephschorr josephschorr marked this pull request as ready for review September 22, 2022 00:30
@josephschorr josephschorr requested a review from a team September 22, 2022 00:30
Copy link
Contributor

@vroldanbet vroldanbet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Left 2 nit suggestions, and a potential follow up to how we handle overlapping keys in CRDB

) ([]*core.NamespaceDefinition, error) {
read, err := vsr.delegate.LookupNamespaces(ctx, nsNames)
if err != nil {
return read, err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: may confuse the reader, even though read should be nil

Suggested change
return read, err
return nil, err

}
}

return read, err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return read, err
return read, nil

Comment on lines +140 to +142
for _, nsDef := range nsDefs {
cr.addOverlapKey(nsDef.Name)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for very large namespaces (presumably the ones causing the flakiness we are attempting to fix in this PR) this would lead to many "touches" in the transactions table:

for k := range rwt.overlapKeySet {
if _, err := tx.Exec(ctx, queryTouchTransaction, k); err != nil {
return fmt.Errorf("error writing overlapping keys: %w", err)
}
}

@ecordell is there are reason we don't batch all those query operations instead of looping over the keyset?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No real reason other than almost all requests should have only 1 overlap key

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, I think it would be relatively easy to turn that into a batch pgx request

@josephschorr josephschorr merged commit c2ba0bb into authzed:main Sep 22, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Sep 22, 2022
@vroldanbet
Copy link
Contributor

ugh sorry I didn't notice auto-merged was enabled 😭

@josephschorr josephschorr deleted the fix-v1alpha1-schema-perf branch September 22, 2022 14:58
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/api v1 Affects the v1 API area/datastore Affects the storage system
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants