Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

Update Saas Configs and Datasets on System Startup #1098

Closed
pattisdr opened this issue Aug 16, 2022 · 2 comments · Fixed by #1307
Closed

Update Saas Configs and Datasets on System Startup #1098

pattisdr opened this issue Aug 16, 2022 · 2 comments · Fixed by #1307
Assignees
Labels
enhancement New feature or request

Comments

@pattisdr
Copy link
Contributor

pattisdr commented Aug 16, 2022

Is your feature request related to a specific problem?

We update the saas config and saas dataset files over time, which can cause them to diverge with saas configs and saas datasets that have been saved in a user's database.

On system startup, if the version in the registry is newer than the SaaS config version in the connection config table, update the saas config and the dataset. We also need to standardize the version number that already exists in the SaaS configs, they are all currently set to version 0.0.1

Describe the solution you'd like

Instead of creating data migrations every time we update a saas config or dataset config template file, add logic to just make the changes automatically on startup.

Steps:

  • get all saas connection configs
  • get version number from SaaS config (figure out versioning)
  • if registry version is newer
    • update saas config
    • update dataset

Add ability to turn off auto update

Describe alternatives you've considered, if any

A description of any alternative solutions or features you've considered.

Additional context

Separated from #814 to release that functionality more quickly

@adamsachs
Copy link
Contributor

if a template dataset is changed in the registry, i assume we want to update the datasets in the db that were created from that template. that brings up a question though:

  • dataset templates don't seem to have any version number, so would we rely on the associated saas config version number? this would mean we'd have to update the saas config version number even if the config itself stayed the same and only the dataset was updated. alternatively we could update datasets to have a version field.

i'm assuming the intention here is that an update to a given saas provider template, whether that's in its config or its dataset or both, is recorded by incrementing the version number stored in the config. the code would then compare that version number with saas config records in the db that have the same type field. if the version number is lower in the db, then it would replace the db config and the associated dataset (linked by means of a shared fides_key) with what's provided by the template in the registry.

for now, turning off auto updates can be accomplished by the user simply changing the type on their given saas config so that it no longer is "linked" to the template provided by the codebase (is that right @galvana ? will it encounter any validation errors?)

@pattisdr @galvana please let me know if any of this seems off-base. happy to discuss more offline too, but wanted to write this down here as a reference.

@adamsachs
Copy link
Contributor

noting here that after discussion with @galvana, we agreed that a separate issue should be created to provide some sort of heads-up to the user if a template is changed such that a param is added, since the user will need to update their config to parameterize the new value

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants