Skip to content

Conversation

@arvindp25
Copy link
Contributor

@arvindp25 arvindp25 commented Aug 6, 2025


  1. Fix file extension issue on SqlToS3Operator in amazon provider.
    Now loaded file have proper extension. refer below table to understand for extension
    | File Format | Extension |
    |------------------|--------------|
    | CSV | .csv |
    | CSV (gzip) | .csv.gz |
    | JSON | .json |
    | JSON (gzip) | .json.gz |
    | Parquet | .parquet |
    | Parquet (gzip) | .parquet |

updated and refactor unittest case for the same and added more

closes: #53976
related: #53976
POW:-

  1. without groupby_kwargs
pow_without_groupby.mp4
  1. with groupby_kwargs
    https://github.com/user-attachments/assets/fa110550-72f9-40ec-bf05-e4d49dd58b7c

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Aug 6, 2025
@arvindp25 arvindp25 force-pushed the arvindp/amazon/bug-53976 branch from 564932b to 0ad29c5 Compare August 6, 2025 17:38
@arvindp25 arvindp25 marked this pull request as ready for review August 6, 2025 17:45
@arvindp25 arvindp25 changed the title fixing file extension issue fixing file extension issue on SqlToS3Operator Aug 6, 2025
@arvindp25 arvindp25 force-pushed the arvindp/amazon/bug-53976 branch from 6c7134c to 63bba8e Compare August 6, 2025 18:09
Copy link
Member

@guan404ming guan404ming left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one nit.

@arvindp25 arvindp25 requested a review from guan404ming August 7, 2025 16:26
@arvindp25 arvindp25 force-pushed the arvindp/amazon/bug-53976 branch from 44a854f to 3c0c39d Compare August 9, 2025 17:10
@arvindp25 arvindp25 force-pushed the arvindp/amazon/bug-53976 branch from 3c0c39d to 8fbebf6 Compare August 10, 2025 16:21
@o-nikolas
Copy link
Contributor

This is going to change the user experience, but I think we can definitely catalogue this one as a bug fix.

@o-nikolas o-nikolas merged commit 8ef0ac3 into apache:main Aug 11, 2025
75 checks passed
RoyLee1224 pushed a commit to RoyLee1224/airflow that referenced this pull request Aug 15, 2025
@bryanyang0528
Copy link
Contributor

It's a breaking change.
We named our file with .tsv
Now, every file becomes .tsv.csv...

@o-nikolas
Copy link
Contributor

It's a breaking change. We named our file with .tsv Now, every file becomes .tsv.csv...

@bryanyang0528 Can you create a Github Issue with more details and reproduction steps. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SqlToS3Operator do not add the suffix .parquet when load to

5 participants