Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add exclude columns option to BaseSQLToGCSOperator #23694

Closed
2 tasks done
jaegwonseo opened this issue May 13, 2022 · 2 comments
Closed
2 tasks done

add exclude columns option to BaseSQLToGCSOperator #23694

jaegwonseo opened this issue May 13, 2022 · 2 comments
Assignees
Labels
kind:feature Feature Requests

Comments

@jaegwonseo
Copy link
Contributor

jaegwonseo commented May 13, 2022

Description

when transfer data from rdb to gcs, some column should be droped for security issue

Use case/motivation

When retrieving data from each table, mainly query to get all columns as shown below.

select * from table 

we can use specify the columns to import for each table, but in this case, we have to know the schema of all tables.
By adding exclude_column args to the operator as shown below, data from various tables can be sent to gcs with the same query.

to_tcs_task = MySQLToGCSOperator(
                    task_id='',
                    sql=f'select * from target_table,
                    bucket='',
                    filename='',
                    schema_filename='',
                    export_format='json',
                    exclude_column=exclude_column,
                    mysql_conn_id=source_connect_id,
                    gcp_conn_id=target_connect_id
                )

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@jaegwonseo jaegwonseo added the kind:feature Feature Requests label May 13, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented May 13, 2022

Thanks for opening your first issue here! Be sure to follow the issue template!

@eladkal
Copy link
Contributor

eladkal commented Jun 1, 2022

Fixed #23695

@eladkal eladkal closed this as completed Jun 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:feature Feature Requests
Projects
None yet
Development

No branches or pull requests

2 participants