Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DPP-334 Manually partition the event table #9394

Merged
merged 10 commits into from
Apr 19, 2021

Conversation

rautenrieth-da
Copy link
Contributor

@rautenrieth-da rautenrieth-da commented Apr 13, 2021

This PR removes the automatic partitioning of the participant_events table in favor of a manual partitioning. This is easier to port to other databases, and removes unnecessary fields from individual partitions.

Schema changes

  • The schema now contains 4 distinct tables: participant_events_divulgence, participant_events_create, participant_events_consuming_exercise, and participant_events_non_consuming_exercise. The individual tables are leaner than before - they don't contain useless fields, and the remaining fields have been made NOT NULL where possible.
  • There is now a view participant_events that contains exactly the same fields as the previous partitioned table of the same name.
  • In the indexer, events are grouped by their kind, and inserted directly into the respective tables. This increases the number of SQL statements executed (previously: 1 batch of mixed events, now: 4 batches of events of the same kind).

Testing

The conformance test tool is now run against the append-only schema on Postgres, using ledger-on-sql.

Future work

  • Make fields of DBDTOV1.Event* non-optional where applicable.
  • Change all queries to select from the individual partitions instead of from the view, where it makes sense.
  • Implement the removal of partially written divulgence events

@rautenrieth-da rautenrieth-da force-pushed the dpp-334-manual-partitioning branch from f19a66e to c7e0c49 Compare April 13, 2021 22:55
@rautenrieth-da rautenrieth-da force-pushed the dpp-334-manual-partitioning branch from c7e0c49 to 094d6c4 Compare April 15, 2021 23:40
@rautenrieth-da rautenrieth-da marked this pull request as ready for review April 16, 2021 07:14
Copy link
Contributor

@meiersi-da meiersi-da left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rautenrieth-da for this good work! As you said, it is a bit boilerplaty, but it is also simple, which I like. Comments added. Nothing major.

@rautenrieth-da rautenrieth-da force-pushed the dpp-334-manual-partitioning branch from 6755aa5 to a10cecf Compare April 16, 2021 09:12
Copy link
Contributor

@tudor-da tudor-da left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Tested in an E2E run with no observed performance impact.

@rautenrieth-da rautenrieth-da merged commit 3193027 into main Apr 19, 2021
@rautenrieth-da rautenrieth-da deleted the dpp-334-manual-partitioning branch April 19, 2021 19:12
azure-pipelines bot pushed a commit that referenced this pull request Apr 21, 2021
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.

@remyhaemmerle-da is in charge of this release.

Commit log:
```
cf2be78 Support rollback nodes in BlindingInfo (#9445)
adde90b KV: do not use "Effects" part of InputsAndEffects (#9433)
75fa86a Additional metrics for mutable contract state cache (#9444)
d8c34a4 Rename normalization-related params in the integrity checker config (#9455)
e335244 Hash submitters in the deduplication key [DPP-347] (#9417)
43ffa42 Test for duplicate contracts when querying on behalf of multiple parties (#9443)
7644844 Document daml ledger export script (#9446)
3193027 DPP-334 Manually partition the event table (#9394)
14661a8 Clearer variable name for subtype evidence (#9442)
40c85d8 Release v1.13.0-snapshot.20210419.6730.0.8c3a8c04 (#9439)
1cb907c KV: do not use "Inputs" part of InputsAndEffects (#9429)
```
Changelog:
```
- [Integration Kit] - deduplication key will contain hashed submitters instead of concatenated submitters
* [Daml export] Refer to the "Ledger Export" chapter under the "Early
  Access Features" for a description of the new Daml ledger export
  command. This is an early access feature.
```

CHANGELOG_BEGIN
CHANGELOG_END
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants