Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

define schema for tts services synced table #2441

Merged
merged 1 commit into from
Mar 30, 2023
Merged

Conversation

lauriemerrell
Copy link
Contributor

@lauriemerrell lauriemerrell commented Mar 30, 2023

Description

Fixes #2440 by defining a schema for the TTS Services table and suppressing the column that was erroring, which we don't need anyway. We only use this table to sync IDs with the California Transit version of this table so we literally only need the ID and name.

Resolves #2440

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation
  • agencies.yml

How has this been tested?

[2023-03-30, 14:10:26 UTC] {logging_mixin.py:137} INFO - Deleting external table if exists: cal-itp-data-infra-staging.external_airtable.transit_technology_stacks__services
[2023-03-30, 14:10:26 UTC] {logging_mixin.py:137} INFO - Creating external table: cal-itp-data-infra-staging.external_airtable.transit_technology_stacks__services Table(TableReference(DatasetReference('cal-itp-data-infra-staging', 'external_airtable'), 'transit_technology_stacks__services')) ['gs://test-calitp-airtable/transit_technology_stacks__services/*.jsonl.gz'] {'mode': 'AUTO', 'require_partition_filter': False, 'source_uri_prefix': 'transit_technology_stacks__services/'}
[2023-03-30, 14:10:28 UTC] {logging_mixin.py:137} INFO - Successfully ran SELECT *
FROM `cal-itp-data-infra-staging`.external_airtable.transit_technology_stacks__services
LIMIT 1;
[2023-03-30, 14:10:28 UTC] {taskinstance.py:1401} INFO - Marking task as SUCCESS. dag_id=create_external_tables, task_id=external_airtable_transit_tech_stacks_services, execution_date=20230329T110000, start_date=20230330T141024, end_date=20230330T141028
[2023-03-30, 14:10:28 UTC] {local_task_job.py:159} INFO - Task exited with return code 0
[2023-03-30, 14:10:28 UTC] {taskinstance.py:2623} INFO - 0 downstream tasks scheduled from follow-on schedule check
laurie ~/git/data-infra/warehouse [fix-tts-services-ntd] $ poetry run dbt run -s +int_transit_database__service_components_dim --vars 'DEFAULT_SOURCE_DATABASE: cal-itp-data-infra-staging'
14:23:24  Running with dbt=1.4.5
14:23:25  Found 336 models, 804 tests, 0 snapshots, 0 analyses, 811 macros, 0 operations, 6 seed files, 151 sources, 5 exposures, 0 metrics
14:23:25  
14:23:28  Concurrency: 4 threads (target='dev')
14:23:28  
14:23:28  1 of 14 START sql view model laurie_staging.base_tts_organizations_ct_organizations_map  [RUN]
14:23:28  2 of 14 START sql view model laurie_staging.base_tts_services_ct_services_map .. [RUN]
14:23:28  3 of 14 START sql view model laurie_staging.stg_transit_database__components ... [RUN]
14:23:28  4 of 14 START sql view model laurie_staging.stg_transit_database__organizations  [RUN]
14:23:29  1 of 14 OK created sql view model laurie_staging.base_tts_organizations_ct_organizations_map  [CREATE VIEW (0 processed) in 0.83s]
14:23:29  5 of 14 START sql view model laurie_staging.stg_transit_database__services ..... [RUN]
14:23:29  2 of 14 OK created sql view model laurie_staging.base_tts_services_ct_services_map  [CREATE VIEW (0 processed) in 0.86s]
14:23:29  6 of 14 START sql view model laurie_staging.base_tts_products_idmap ............ [RUN]
14:23:29  3 of 14 OK created sql view model laurie_staging.stg_transit_database__components  [CREATE VIEW (0 processed) in 1.28s]
14:23:29  4 of 14 OK created sql view model laurie_staging.stg_transit_database__organizations  [CREATE VIEW (0 processed) in 1.29s]
14:23:29  7 of 14 START sql view model laurie_staging.base_tts_service_components_idmap .. [RUN]
14:23:29  8 of 14 START sql table model laurie_staging.int_transit_database__components_dim  [RUN]
14:23:29  6 of 14 OK created sql view model laurie_staging.base_tts_products_idmap ....... [CREATE VIEW (0 processed) in 0.69s]
14:23:29  9 of 14 START sql table model laurie_staging.int_transit_database__organizations_dim  [RUN]
14:23:30  7 of 14 OK created sql view model laurie_staging.base_tts_service_components_idmap  [CREATE VIEW (0 processed) in 0.70s]
14:23:30  10 of 14 START sql view model laurie_staging.stg_transit_database__products .... [RUN]
14:23:30  5 of 14 OK created sql view model laurie_staging.stg_transit_database__services  [CREATE VIEW (0 processed) in 1.25s]
14:23:30  11 of 14 START sql view model laurie_staging.stg_transit_database__service_components  [RUN]
14:23:31  11 of 14 OK created sql view model laurie_staging.stg_transit_database__service_components  [CREATE VIEW (0 processed) in 1.22s]
14:23:31  12 of 14 START sql table model laurie_staging.int_transit_database__services_dim  [RUN]
14:23:31  10 of 14 OK created sql view model laurie_staging.stg_transit_database__products  [CREATE VIEW (0 processed) in 1.35s]
14:23:31  13 of 14 START sql table model laurie_staging.int_transit_database__products_dim  [RUN]
14:23:32  8 of 14 OK created sql table model laurie_staging.int_transit_database__components_dim  [CREATE TABLE (123.0 rows, 1.1 MB processed) in 3.44s]
14:23:33  9 of 14 OK created sql table model laurie_staging.int_transit_database__organizations_dim  [CREATE TABLE (1.7k rows, 138.9 MB processed) in 4.22s]
14:23:35  13 of 14 OK created sql table model laurie_staging.int_transit_database__products_dim  [CREATE TABLE (309.0 rows, 54.3 MB processed) in 3.62s]
14:23:35  12 of 14 OK created sql table model laurie_staging.int_transit_database__services_dim  [CREATE TABLE (1.9k rows, 146.1 MB processed) in 4.09s]
14:23:35  14 of 14 START sql table model laurie_staging.int_transit_database__service_components_dim  [RUN]
14:23:40  14 of 14 OK created sql table model laurie_staging.int_transit_database__service_components_dim  [CREATE TABLE (4.0k rows, 98.9 MB processed) in 4.50s]
14:23:40  
14:23:40  Finished running 9 view models, 5 table models in 0 hours 0 minutes and 14.28 seconds (14.28s).
14:23:40  
14:23:40  Completed successfully
14:23:40  
14:23:40  Done. PASS=14 WARN=0 ERROR=0 SKIP=0 TOTAL=14

Screenshots (optional)

Sorry, something went wrong.

Verified

This commit was signed with the committer’s verified signature.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Handle invalid data in transit tech stacks services table
2 participants