Detangle collection storage #1964
Assignees
Labels
Milestone
Comments
AndrewSisley
added
area/collections
|
Can you add context here on what "no sense". The other issue regarding fetching has great additional info, let's spruce this one up too ;) |
I don't plan on letting this issue live for very long, and it is both tricky and inexact to explain, and in some sense already solved in the WIP branch 1963-collection-feat-detangle The keys and their values used to persist and index the collection description make no sense. They are tangled, and with poor names and prefixes. They blend schema with collection very heavily. |
|
sounds good! |
|
Currently we have COLLECTION = "/collection/names" // {collectionName} -> schemaVersionID
COLLECTION_SCHEMA = "/collection/schema" // {rootSchema} -> schemaVersionID
COLLECTION_SCHEMA_VERSION = "/collection/version/v" // {schemaVersionID} -> client.CollectionDescription
COLLECTION_SCHEMA_VERSION_HISTORY = "/collection/version/h" // {rootSchemaID}/{previousSchemaVersionID} -> schemaVersionID
SCHEMA_VERSION = "/schema/version" // {schemaVersionID} -> client.SchemaDescription This is what I think we should have: COLLECTION_NAME = "/collection/name" // {collectionName} -> client.CollectionDescription
COLLECTION_ID = "/collection/id" // {collectionID} -> collectionName
SCHEMA_VERSION = "/schema/version" // {schemaVersionID} -> client.SchemaDescription
SCHEMA_VERSION_HISTORY = "/schema/version/h" // {rootSchemaID}/{previousSchemaVersionID} -> schemaVersionID
// maybe also
SCHEMA_ROOT_COLLECTION = "/schema/root/collection" // "{rootSchemaID}/{collectionName} -> someEmptyMarker
SCHEMA_VERSION_COLLECTION = "/schema/version/collection" // "{schemaVersionID}/{collectionName} -> someEmptyMarkerUser will interact most often with collections via their names so it makes sense to have The P2P system will interact with collections using their collection ID. In this case we have The schema description can be stored via There are 2 other avenues that we may want to cover: |
AndrewSisley
added a commit
that referenced
this issue
Oct 17, 2023
## Relevant issue(s) Resolves #1958 ## Description Removes CollectionDescription.Schema. Also splits the storage of schema out from within collection. The storage of schema has been broken out to a new sub-package of db, at the moment it is a very simple file, but collection will be moved there in #1964. I was planning on doing that in this PR (in part, to provide context for reviewers, as atm it is basically a single-file package), but it proved to be non-trivial due to some existing messiness in that space and was broken out to two more tasks. I also wish for stuff in that directory to eventually follow a repository-like pattern, where stuff is cached (within a context/txn's context) instead of fetching from store on each call. Moving this stuff out to a new directory instead of preserving it in the (already very large) db directory should make both db and the new sub-package a fair bit more cohesive and easier to read.
nasdf
pushed a commit
to nasdf/defradb
that referenced
this issue
Oct 18, 2023
## Relevant issue(s) Resolves sourcenetwork#1958 ## Description Removes CollectionDescription.Schema. Also splits the storage of schema out from within collection. The storage of schema has been broken out to a new sub-package of db, at the moment it is a very simple file, but collection will be moved there in sourcenetwork#1964. I was planning on doing that in this PR (in part, to provide context for reviewers, as atm it is basically a single-file package), but it proved to be non-trivial due to some existing messiness in that space and was broken out to two more tasks. I also wish for stuff in that directory to eventually follow a repository-like pattern, where stuff is cached (within a context/txn's context) instead of fetching from store on each call. Moving this stuff out to a new directory instead of preserving it in the (already very large) db directory should make both db and the new sub-package a fair bit more cohesive and easier to read.
AndrewSisley
added a commit
that referenced
this issue
Oct 19, 2023
## Relevant issue(s) Resolves #1963 ## Description Changes GetCollectionBySchemaFoo function signatures to return many Collections, instead of single values. Conceptually these functions were broken and relied on the fact that we cannot yet link multiple collections to the same schema(version)(s). This changes their signatures and the code that calls them to reflect what they should do long term. The code that calls these functions now hosts the 'broken' code. These will still need to change once we add global IDs. The implementation of these functions will change later in #1964.
AndrewSisley
added a commit
that referenced
this issue
Oct 24, 2023
## Relevant issue(s) Resolves #1964 #1913 ## Description Reorganises collection description storage so that it actually makes sense. The existing key/value setup has been replaced with the following keys: ``` [Collection.ID] => json // Json should be stored against an immutable index, i.e. not against `Name` [Collection.Name] => [Collection.ID] // This means collection names must be unique ``` We also need the following for the short-term, before Collection.GlobalID gets added, as otherwise if a collection is updated to a new schema version, we lose track of what collection it could be. The locations that use this have been linked to #1085 ``` [Collection.SchemaVersionId]/[Collection.ID] => nil ``` With this change the storage setup should fully support multiple collections from a single schema, however this is blocked off by `setDefaultSchemaVersion` (it doesn't allow users to explicitly specify a collection, so atm there is no way to try and do this). Later when we introduce `patchCollection` we will need to be mindful of this and block off user's ability to do this (if done before #1085).
shahzadlone
pushed a commit
to shahzadlone/defradb
that referenced
this issue
Feb 23, 2024
## Relevant issue(s) Resolves sourcenetwork#1958 ## Description Removes CollectionDescription.Schema. Also splits the storage of schema out from within collection. The storage of schema has been broken out to a new sub-package of db, at the moment it is a very simple file, but collection will be moved there in sourcenetwork#1964. I was planning on doing that in this PR (in part, to provide context for reviewers, as atm it is basically a single-file package), but it proved to be non-trivial due to some existing messiness in that space and was broken out to two more tasks. I also wish for stuff in that directory to eventually follow a repository-like pattern, where stuff is cached (within a context/txn's context) instead of fetching from store on each call. Moving this stuff out to a new directory instead of preserving it in the (already very large) db directory should make both db and the new sub-package a fair bit more cohesive and easier to read.
shahzadlone
pushed a commit
to shahzadlone/defradb
that referenced
this issue
Feb 23, 2024
…work#1984) ## Relevant issue(s) Resolves sourcenetwork#1963 ## Description Changes GetCollectionBySchemaFoo function signatures to return many Collections, instead of single values. Conceptually these functions were broken and relied on the fact that we cannot yet link multiple collections to the same schema(version)(s). This changes their signatures and the code that calls them to reflect what they should do long term. The code that calls these functions now hosts the 'broken' code. These will still need to change once we add global IDs. The implementation of these functions will change later in sourcenetwork#1964.
shahzadlone
pushed a commit
to shahzadlone/defradb
that referenced
this issue
Feb 23, 2024
## Relevant issue(s) Resolves sourcenetwork#1964 sourcenetwork#1913 ## Description Reorganises collection description storage so that it actually makes sense. The existing key/value setup has been replaced with the following keys: ``` [Collection.ID] => json // Json should be stored against an immutable index, i.e. not against `Name` [Collection.Name] => [Collection.ID] // This means collection names must be unique ``` We also need the following for the short-term, before Collection.GlobalID gets added, as otherwise if a collection is updated to a new schema version, we lose track of what collection it could be. The locations that use this have been linked to sourcenetwork#1085 ``` [Collection.SchemaVersionId]/[Collection.ID] => nil ``` With this change the storage setup should fully support multiple collections from a single schema, however this is blocked off by `setDefaultSchemaVersion` (it doesn't allow users to explicitly specify a collection, so atm there is no way to try and do this). Later when we introduce `patchCollection` we will need to be mindful of this and block off user's ability to do this (if done before sourcenetwork#1085).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The way we store collections makes no sense at the moment, and needs to be sorted out.
It should probably be done after #1963 though.
There will be the following keys:
We also need the following for the short-term, before Collection.GlobalID gets added, as otherwise if a collection is updated to a new schema version, we lose track of what collection it could (could as, this only narrows it down) be.
The text was updated successfully, but these errors were encountered: