Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Derived datasets to be parents #579

Open
5 tasks
noracato opened this issue Mar 6, 2025 · 0 comments
Open
5 tasks

Allow Derived datasets to be parents #579

noracato opened this issue Mar 6, 2025 · 0 comments

Comments

@noracato
Copy link
Member

noracato commented Mar 6, 2025

This issue is the ETLocal counterpart of quintel/atlas#171.

Here I'll describe what is necessary to allow ETLocal to accept a Derived dataset as a parent of a dataset. And which inputs/values should be added to ETLocal.

1. Allowing Derived datasets as parents

We need to make a few tweaks to the code in ETLocal and set up new validations for who can be a parent.

Currently, instead of a parent, datasets have a country. In Dataset we should update the attribute into being called country. For example:

-  validates :country, presence: true
- validates_each :country, allow_nil: true do |record, _attr, value|
-    unless Etsource.available_countries.include?(value.downcase)
-      record.errors.add(value.downcase, 'must be in ETsource')
-    end
-  end
+ validates :parent, inclusion: {in: Etsource.available_countries}
  • Create a migration to change the column name
  • Update the word country to parent throughout the app
  • Review validations: e.g. update Etsource.available_countries to Etsource.available_datasets, where all datasets will now be elegible
  • Adjust specs

2. Attributes to be checked

The Atlas::Dataset.attribute_set

In the Dataset in Atlas we define all area attributes needed to initialise a dataset. Let's make sure we have a correspondent input for each of them.

  • Check alignment of Atlas's attributes on Dataset with ETLocal inputs

ETLocal works as follows: if an input or attribute is not specified in ETLocal, it looks at the value represented in ETSource.
This value that is in ETSource might be the result of a sparse graph query, or copied from the parent set upon export from etlocal to etsource. Meaning: 1) a dataset in etlocal does not have to be complete and 2) changes in a parent within the etocal environment will not change anything in the children until export to ETSource.

This last point could be a plus, but I can imagine you'd like to see the effect of parent changes on the children immediately, and not only after export. @mabijkerk let's discuss this part in our brainstorm.

Curves

To ensure the future of parenting, we should make sure a full curve set is required for root ancestors. As in Atlas we currently still enforce a Full dataset as root ancestor, this is not yet necessary at this point, but it would be nice to take some time to design the passing on of curves through Transformer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant