Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indirect write drops policy tags #1043

Closed
khaledh opened this issue Aug 18, 2023 · 2 comments
Closed

Indirect write drops policy tags #1043

khaledh opened this issue Aug 18, 2023 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@khaledh
Copy link

khaledh commented Aug 18, 2023

We noticed that while this issue is resolved for direct writes (#1028), it still happens when we use indirect writes; the policy tags are dropped after the indirect write. We also noticed that non-nullable columns are created as nullable when using indirect write. We're trying to work around this issue, but we're hoping it would be fixed in the connector itself.

We tried both parquet and avro intermediate formats.

(Original comment: #1028 (comment))

@khaledh khaledh changed the title Indirect write drops column nullability and policy tags Indirect write drops ~column nullability~ and policy tags Aug 21, 2023
@khaledh khaledh changed the title Indirect write drops ~column nullability~ and policy tags Indirect write drops policy tags Aug 21, 2023
@khaledh
Copy link
Author

khaledh commented Aug 21, 2023

I updated the title/description to remove the issue about nullability. It turns out to be an issue on our side. The policy tags, however, are still not populated as expected with indirect writes.

@isha97 isha97 added the enhancement New feature or request label May 13, 2024
@isha97 isha97 self-assigned this May 13, 2024
vishalkarve15 added a commit to vishalkarve15/spark-bigquery-connector that referenced this issue Jun 25, 2024
vishalkarve15 added a commit to vishalkarve15/spark-bigquery-connector that referenced this issue Jun 25, 2024
vishalkarve15 added a commit to vishalkarve15/spark-bigquery-connector that referenced this issue Jun 25, 2024
isha97 added a commit that referenced this issue Jul 3, 2024
@isha97
Copy link
Member

isha97 commented Jul 3, 2024

The fix will be available in the next release.

@isha97 isha97 closed this as completed Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants