Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-architect node data storage to only store relevant data #1294

Open
joepavitt opened this issue Sep 16, 2024 · 1 comment · May be fixed by #1465
Open

Re-architect node data storage to only store relevant data #1294

joepavitt opened this issue Sep 16, 2024 · 1 comment · May be fixed by #1465
Assignees
Labels
feature-request New feature or request that needs to be turned into Epic/Story details size:L - 5 Sizing estimation point

Comments

@joepavitt
Copy link
Collaborator

Description

We made a decision early on to store full msg objects against each node. This was because there was a lot of contextual data nodes could call upon, and we want to be sure that _client was considered for multi-tenant use cases, but that the time, the _client implementation hadn't been completed.

Since then, we settled on any data used in multi-user use cases are not stored in the datastore, as such, we can guarantee any data saved should be replayed to all clients.

Given the above, we should clean our data storage (as recommended many times by @bartbutenaers and @colinl) to only save a mapping of the relevant data.

The APIs do not need to change here for the data store, just what data is saved, and how that data is processed when retrieved.

We're seeing problems now whereby dynamic update messages are being saved as the most recent message, but overriding saved msg.payload values. It is actually valid for both of those values to be saved.

Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

@joepavitt joepavitt added needs-triage Needs looking at to decide what to do feature-request New feature or request that needs to be turned into Epic/Story details size:L - 5 Sizing estimation point and removed needs-triage Needs looking at to decide what to do labels Sep 16, 2024
@joepavitt joepavitt moved this from Backlog to Up Next in Dashboard Backlog Sep 16, 2024
@joepavitt joepavitt moved this from Up Next to In Progress in Dashboard Backlog Nov 12, 2024
@joepavitt joepavitt self-assigned this Nov 12, 2024
@joepavitt
Copy link
Collaborator Author

The solution for #679 is a lot easier if we also tackle this at the same time, so I think I'm going to bite off this meaty item and try to tackle it together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request New feature or request that needs to be turned into Epic/Story details size:L - 5 Sizing estimation point
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

1 participant