[infra] conform Metadata to new metadata changes #1093
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: terminal_name <github_email>
* Sobe diretorio etnia_indigena * Update table_config.yaml * Update table_config.yaml
d116626
added a commit
that referenced
this pull request
Feb 12, 2022
* [infra] fix dataset_config.yaml folder path (#1067) * feat(infra) merge master * [infra] conform Metadata to new metadata changes (#1093) * [dados-bot] br_ms_vacinacao_covid19 (2022-01-23) (#1086) Co-authored-by: terminal_name <github_email> * [dados] br_bd_diretorios_brasil.etnia_indigena (#1087) * Sobe diretorio etnia_indigena * Update table_config.yaml * Update table_config.yaml * feat: conform Metadata's schema to new one * fix: conform yaml generation to new schema * fix: delete test_dataset folder Co-authored-by: Lucas Moreira <65978482+lucasnascm@users.noreply.github.com> Co-authored-by: Gustavo Aires Tiago <36206956+gustavoairestiago@users.noreply.github.com> Co-authored-by: Ricardo Dahis <6617207+rdahis@users.noreply.github.com> Co-authored-by: Lucas Moreira <65978482+lucasnascm@users.noreply.github.com> Co-authored-by: Gustavo Aires Tiago <36206956+gustavoairestiago@users.noreply.github.com>
lucascr91
added a commit
that referenced
this pull request
Mar 14, 2022
* feat(infra): create version 1.6.2 * feat(infra): create version 1.6.2 * feat(infra): create version 1.6.2 * [infra] python-v1.6.2 (#1089) * [infra] fix dataset_config.yaml folder path (#1067) * feat(infra) merge master * [infra] conform Metadata to new metadata changes (#1093) * [dados-bot] br_ms_vacinacao_covid19 (2022-01-23) (#1086) Co-authored-by: terminal_name <github_email> * [dados] br_bd_diretorios_brasil.etnia_indigena (#1087) * Sobe diretorio etnia_indigena * Update table_config.yaml * Update table_config.yaml * feat: conform Metadata's schema to new one * fix: conform yaml generation to new schema * fix: delete test_dataset folder Co-authored-by: Lucas Moreira <65978482+lucasnascm@users.noreply.github.com> Co-authored-by: Gustavo Aires Tiago <36206956+gustavoairestiago@users.noreply.github.com> Co-authored-by: Ricardo Dahis <6617207+rdahis@users.noreply.github.com> Co-authored-by: Lucas Moreira <65978482+lucasnascm@users.noreply.github.com> Co-authored-by: Gustavo Aires Tiago <36206956+gustavoairestiago@users.noreply.github.com> * feat(infra): 1.6.2a3 version * feat(infra): 1.6.2a3 version * fix(ingra): edit partitions and update_locally * feat(infra): update_columns new fields and accepts local files * [infra] option to make dataset public (#1020) * feat(infra): option to make dataset public * feat(infra): fix None data * fix(infra): roll back * fix(infra): fix retry in storage upload * fix(infra): add option to dataset data location * feat(infra): make staging dataset not public * feat(infra): make staging dataset not public * fix(infra): change bd version in actions * fix(infra): add toml to install in ci * fix(infra): remove a forget print * fix(infra): fix location location * fix(infra): fix dataset description * feat(infra): bump-version * feat(infra): temporal coverage as list in update_columns * feat(infra): add new parameters to cli * feat(infra): fix cli options * [infra] change download functions to consume CKAN endpoints #1129 (#1130) * [infra] add function to wrap bd_dataset_search endpoint * Update download.py * [infra] modify list_datasets function to consume CKAN endpoint * [infra] fix list_dataset function to include limit and remove order_by * [infra] change function list_dataset_tables to use CKAN endpoint * [infra] apply PEP8 to list_dataset_tables and respective tests * add get_dataset_description, get_table_description, get_table_columns * [infra] fix dataset_config.yaml folder path (#1067) * feat(infra) merge master * fix files organization to match master * remove download.py * remove test_download * Delete test_download.py * remove test files * remove test_download.py * remove test_download.py * remove test_download.py * remove test_download.py * add tests metadata * remove test_download.py * remove unused imports * [infra] add _safe_fetch and get_table_size functions Co-authored-by: lucascr91 <lucas.ecomg@gmail.com> * fix(infra): add a empty list to not a partition * [infra] Adiciona suporte a Avro e Parquet (#1145) * adiciona suporte a Avro e Parquet para upload * Adds test for source formats * [infra] update tests for avro, parquet, and csv upload Co-authored-by: Gabriel Gazola Milan <gabriel.gazola@poli.ufrj.br> Co-authored-by: Isadora Bugarin <isadorabugarin@gmail.com > Co-authored-by: lucascr91 <lucas.ecomg@gmail.com> * [infra] Feedback messages in upload methods [issue #1059] (#1085) * Creating dataclass config * Success messages - create and update (table.py) using loguru * feat: improve log level control * refa: move logger config to Base.__init__ * Improving log level control * Adjusting log level control function in base.py * Fixing repeated 'DELETE' messages everytime Table is replaced. * Importing 'dataclass' from 'dataclasses' to make config work. * Fixing repeated 'UPDATE' messages inside other functions. * Defining a new script message format. * Definng standard log messages for 'dataset.py' functions * Definng standard log messages for 'storage.py' functions * Definng standard log messages for 'table.py' functions * Definng standard log messages for 'metadata.py' functions * Adds standard configuration to billing_project_id in download.py * Configuring billing_project_id in download.py * Configuring config_path in base.py Co-authored-by: Guilherme Salustiano <guissalustiano@gmail.com> Co-authored-by: Isadora Bugarin <isadorabugarin@gmail.com> * update toml Co-authored-by: Ricardo Dahis <6617207+rdahis@users.noreply.github.com> Co-authored-by: Lucas Moreira <65978482+lucasnascm@users.noreply.github.com> Co-authored-by: Gustavo Aires Tiago <36206956+gustavoairestiago@users.noreply.github.com> Co-authored-by: lucascr91 <lucas.ecomg@gmail.com> Co-authored-by: Isadora Bugarin <57679195+isadorabugarin@users.noreply.github.com> Co-authored-by: Gabriel Gazola Milan <gabriel.gazola@poli.ufrj.br> Co-authored-by: Isadora Bugarin <isadorabugarin@gmail.com > Co-authored-by: Guilherme Salustiano <guissalustiano@gmail.com> Co-authored-by: Isadora Bugarin <isadorabugarin@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Contexto
Mudamos o arquivo
metadata.pypara conformar às mudanças do schema em basedosdados/website#211.Changelog
observation_level,short_description, etc), matamos campos (e.g.entity,time_unit, etc).dataset_idnão ser automaticamente criado emdataset_config.yaml.