Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CollectionDescription Source refactor #2198

Closed
AndrewSisley opened this issue Jan 12, 2024 · 0 comments · Fixed by #2223 or #2286
Closed

CollectionDescription Source refactor #2198

AndrewSisley opened this issue Jan 12, 2024 · 0 comments · Fixed by #2223 or #2286
Assignees
Labels
area/collections Related to the collections system area/db-system Related to the core system related components of the DB refactor This issue specific to or requires *notable* refactoring of existing codebases and components
Milestone

Comments

@AndrewSisley
Copy link
Contributor

Long term we wish to have something like the below. This issue begins moving us to that.

type CollectionDescription struct {
    // immutable
    ID uint32

    // immutable once set
    GlobalID immutable.Option[string]

    // These properties become immutable if/when GlobalID is set
    Sources         []any
    SchemaVersionID string
    MergeMethod     MergeMethod

    // Always mutable
    Name        immutable.Option[string]
    Indexes     []IndexDescription
    CacheMethod CacheMethod
}

type BaseSource struct {
    ID        uint32  // Used to ID this source, and used to link cached docs to the source (required for a bunch of reasons including update of cache)
    Name      string
    Transform immutable.Option[model.Lens]
}

type QuerySource struct {
    BaseSource
    BaseQuery request.Select
}

type CollectionSource struct {
    BaseSource
    SourceCollectionID uint32
}

type DataStoreSource struct {
    BaseSource
}

type MergeMethod uint32

const (
    MERGE_METHOD_NONE MergeMethod = 0
    MERGE_METHOD_LWW  MergeMethod = 1 //* example only
    // etc
)

type CacheMethod uint32

const (
    CACHE_METHOD_NONE     CacheMethod = 0
    CACHE_METHOD_PER_ITEM CacheMethod = 1
    // etc
)
@AndrewSisley AndrewSisley added area/collections Related to the collections system area/db-system Related to the core system related components of the DB refactor This issue specific to or requires *notable* refactoring of existing codebases and components labels Jan 12, 2024
@AndrewSisley AndrewSisley added this to the DefraDB v0.9 milestone Jan 12, 2024
@AndrewSisley AndrewSisley self-assigned this Jan 12, 2024
@AndrewSisley AndrewSisley changed the title CoolectionDescription Source refactor CollectionDescription Source refactor Jan 12, 2024
AndrewSisley added a commit that referenced this issue Jan 19, 2024
## Relevant issue(s)

Partially resolves #2198

## Description

Makes CollectionDescription.Name Option. Changes secondary indexes to be
indexed by Collection ID. Moves view queries into Sources property.
@AndrewSisley AndrewSisley reopened this Jan 19, 2024
shahzadlone pushed a commit to shahzadlone/defradb that referenced this issue Jan 22, 2024
## Relevant issue(s)

Partially resolves sourcenetwork#2198

## Description

Makes CollectionDescription.Name Option. Changes secondary indexes to be
indexed by Collection ID. Moves view queries into Sources property.
nasdf pushed a commit to nasdf/defradb that referenced this issue Jan 23, 2024
## Relevant issue(s)

Partially resolves sourcenetwork#2198

## Description

Makes CollectionDescription.Name Option. Changes secondary indexes to be
indexed by Collection ID. Moves view queries into Sources property.
AndrewSisley added a commit that referenced this issue Feb 12, 2024
## Relevant issue(s)

Resolves #2198

## Description

Models Collection SchemaVersions and migrations on Collections, instead
of in the Lens Registry.

Primary motivators for the change:
1. Boost the DevEx RE Lens and schema versions. It seemed pretty painful
before to find transforms, yet alone figure out where that lay in the
version history, or add them to a collection.
2. Unify the Collection history/schema history with the View system, so
that they both roughly work in the same way. Reducing the cognitive and
maintenance overhead of having two different systems for the same thing
for both us and the users.

Noteworthy stuff:
1. The branching of schema is not new, it is just more visible with this
change. There is nothing we can really do to block it in a distributed
system. I do now see it as supported, excluding Lens, which still needs
a relatively minor tweak in order to handle the migration-pathing.
2. The model change in client/descriptions.go
3. The removal of stuff from the LensRegistry. This is currently kept to
a minimum, however I would very much like to completely remove
LensRegistry in the near future.
4. The CLI and the Http clients used to rely on the fact that
db.SetMigration and registry.SetMigration did the same thing. This is no
longer the case.
5. The CLI and the Http clients followed a different, flatter structure
than the Go client, flattening the lens registry into the
`lens`/`schema` pathways. This was inconsistent, and only worked when
(4) was as it was. This has been changed out of necessity and the 3
clients now have matching structures.
6. The CLI still paths everything under `schema`, and the http client
everything under `lens`, which is strange, but out of scope. I will open
a ticket to make them the same.
7. Some of the schema versions in the existing schema tests have
changed, I think there was a bug in the framework where it was not
caring about their accuracy. It now cares.
8. Users can now specify their transforms when patching a schema.
9. Out of the 4 collection getting funcs on `client.Store`, only
`GetAllCollections` has the ability to get inactive collections. I
currently very much dislike that we have 4 functions for the same thing
in the Go client, especially given that this is not reflected in the
Http and CLI clients. In another ticket I would very much like us to
collapse these four functions into one, and have that one function
support the fetching of inactive collections.
shahzadlone pushed a commit to shahzadlone/defradb that referenced this issue Feb 23, 2024
## Relevant issue(s)

Partially resolves sourcenetwork#2198

## Description

Makes CollectionDescription.Name Option. Changes secondary indexes to be
indexed by Collection ID. Moves view queries into Sources property.
shahzadlone pushed a commit to shahzadlone/defradb that referenced this issue Feb 23, 2024
…#2286)

## Relevant issue(s)

Resolves sourcenetwork#2198

## Description

Models Collection SchemaVersions and migrations on Collections, instead
of in the Lens Registry.

Primary motivators for the change:
1. Boost the DevEx RE Lens and schema versions. It seemed pretty painful
before to find transforms, yet alone figure out where that lay in the
version history, or add them to a collection.
2. Unify the Collection history/schema history with the View system, so
that they both roughly work in the same way. Reducing the cognitive and
maintenance overhead of having two different systems for the same thing
for both us and the users.

Noteworthy stuff:
1. The branching of schema is not new, it is just more visible with this
change. There is nothing we can really do to block it in a distributed
system. I do now see it as supported, excluding Lens, which still needs
a relatively minor tweak in order to handle the migration-pathing.
2. The model change in client/descriptions.go
3. The removal of stuff from the LensRegistry. This is currently kept to
a minimum, however I would very much like to completely remove
LensRegistry in the near future.
4. The CLI and the Http clients used to rely on the fact that
db.SetMigration and registry.SetMigration did the same thing. This is no
longer the case.
5. The CLI and the Http clients followed a different, flatter structure
than the Go client, flattening the lens registry into the
`lens`/`schema` pathways. This was inconsistent, and only worked when
(4) was as it was. This has been changed out of necessity and the 3
clients now have matching structures.
6. The CLI still paths everything under `schema`, and the http client
everything under `lens`, which is strange, but out of scope. I will open
a ticket to make them the same.
7. Some of the schema versions in the existing schema tests have
changed, I think there was a bug in the framework where it was not
caring about their accuracy. It now cares.
8. Users can now specify their transforms when patching a schema.
9. Out of the 4 collection getting funcs on `client.Store`, only
`GetAllCollections` has the ability to get inactive collections. I
currently very much dislike that we have 4 functions for the same thing
in the Go client, especially given that this is not reflected in the
Http and CLI clients. In another ticket I would very much like us to
collapse these four functions into one, and have that one function
support the fetching of inactive collections.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/collections Related to the collections system area/db-system Related to the core system related components of the DB refactor This issue specific to or requires *notable* refactoring of existing codebases and components
Projects
None yet
1 participant