Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dados-atualizacao] br_inep_saeb #1041

Merged
merged 8 commits into from
Jan 6, 2022
Merged

[dados-atualizacao] br_inep_saeb #1041

merged 8 commits into from
Jan 6, 2022

Conversation

lucasnascm
Copy link
Collaborator

Acho que agora sobe pra produção

@lucasnascm lucasnascm added the data-update Update dataset and tables label Dec 31, 2021
@lucasnascm lucasnascm requested a review from rdahis December 31, 2021 01:21
@lucasnascm lucasnascm self-assigned this Dec 31, 2021
@github-actions
Copy link

github-actions bot commented Jan 2, 2022

Data Check Report

Table br_inep_saeb/dicionario

✔️ Check if table exists

SELECT "dicionario" IN (
    SELECT `table_name`
    FROM `basedosdados-dev.br_inep_saeb.INFORMATION_SCHEMA.TABLES`
) AS success

✔️ Check if any column is completely null

WITH
    n_nulls AS (
        SELECT col_name, COUNT(1) nulls_count
        FROM `basedosdados-dev.br_inep_saeb.dicionario` t,
        UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(t), r'"(\w+)":null')) col_name
        GROUP BY col_name
    ),
    n_total AS (
        SELECT COUNT(*) total_count
        FROM `basedosdados-dev.br_inep_saeb.dicionario` t
    )
SELECT col_name, nulls_count / total_count null_percent
FROM n_nulls, n_total

✔️ Check if primary key has unique values (QueryDoesNotExist Exception)


Table br_inep_saeb/proficiencia

✔️ Check if table exists

SELECT "proficiencia" IN (
    SELECT `table_name`
    FROM `basedosdados-dev.br_inep_saeb.INFORMATION_SCHEMA.TABLES`
) AS success

✔️ Check if any column is completely null

WITH
    n_nulls AS (
        SELECT col_name, COUNT(1) nulls_count
        FROM `basedosdados-dev.br_inep_saeb.proficiencia` t,
        UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(t), r'"(\w+)":null')) col_name
        GROUP BY col_name
    ),
    n_total AS (
        SELECT COUNT(*) total_count
        FROM `basedosdados-dev.br_inep_saeb.proficiencia` t
    )
SELECT col_name, nulls_count / total_count null_percent
FROM n_nulls, n_total

✔️ Check if primary key has unique values (QueryDoesNotExist Exception)


Action Output

# Exemplos: id_municipio, ano.
# Pode ser vazio pois certas tabelas não possuem identificadores.
identifying_columns:
- table_id
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

id_tabela

@rdahis rdahis merged commit 2aff3d9 into master Jan 6, 2022
@rdahis rdahis deleted the update_saeb branch January 6, 2022 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data-check-trigger data-update Update dataset and tables
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants