-
Notifications
You must be signed in to change notification settings - Fork 704
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: simplify README, remove AWS Glue for Ray references #2750
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,12 +1,10 @@ | ||
# AWS SDK for pandas (awswrangler) | ||
|
||
AWS Data Wrangler is now **AWS SDK for pandas (awswrangler)**. We’re changing the name we use when we talk about the library, but everything else will stay the same. You’ll still be able to install using `pip install awswrangler` and you won’t need to change any of your code. As part of this change, we’ve moved the library from AWS Labs to the main AWS GitHub organisation but, thanks to the GitHub’s redirect feature, you’ll still be able to access the project by its old URLs until you update your bookmarks. Our documentation has also moved to [aws-sdk-pandas.readthedocs.io](https://aws-sdk-pandas.readthedocs.io), but old bookmarks will redirect to the new site. | ||
|
||
*Pandas on AWS* | ||
|
||
Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL). | ||
|
||
![AWS SDK for pandas](docs/source/_static/logo2.png?raw=true "AWS SDK for pandas") | ||
![AWS SDK for pandas](https://github.com/aws/aws-sdk-pandas/blob/main/docs/source/_static/logo2.png?raw=true "AWS SDK for pandas") | ||
![tracker](https://d3tiqpr4kkkomd.cloudfront.net/img/pixel.png?asset=GVOYN2BOOQ573LTVIHEW) | ||
|
||
> An [AWS Professional Service](https://aws.amazon.com/professional-services/) open source initiative | aws-proserve-opensource@amazon.com | ||
|
@@ -29,17 +27,13 @@ Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, Q | |
> ⚠️ **Starting version 3.0, optional modules must be installed explicitly:**<br> | ||
➡️`pip install 'awswrangler[redshift]'` | ||
|
||
Powered By [<img src="https://arrow.apache.org/img/arrow.png" width="200">](https://arrow.apache.org/powered_by/) | ||
|
||
## Table of contents | ||
|
||
- [Quick Start](#quick-start) | ||
- [At Scale](#at-scale) | ||
- [Read The Docs](#read-the-docs) | ||
- [Getting Help](#getting-help) | ||
- [Community Resources](#community-resources) | ||
- [Logging](#logging) | ||
- [Who uses AWS SDK for pandas?](#who-uses-aws-sdk-for-pandas) | ||
|
||
## Quick Start | ||
|
||
|
@@ -100,7 +94,7 @@ FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3 | |
## At scale | ||
AWS SDK for pandas can also run your workflows at scale by leveraging [Modin](https://modin.readthedocs.io/en/stable/) and [Ray](https://www.ray.io/). Both projects aim to speed up data workloads by distributing processing over a cluster of workers. | ||
|
||
The quickest way to get started is to use AWS Glue with Ray. Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/scale.html), our blogs ([1](https://aws.amazon.com/blogs/big-data/scale-aws-sdk-for-pandas-workloads-with-aws-glue-for-ray/)/[2](https://aws.amazon.com/blogs/big-data/advanced-patterns-with-aws-sdk-for-pandas-on-aws-glue-for-ray/)), or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to discover even more features. | ||
Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/scale.html) or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to learn more. | ||
|
||
> ⚠️ **Ray is currently not available for Python 3.12. While AWS SDK for pandas supports Python 3.12, it cannot be used at scale.** | ||
|
||
|
@@ -156,7 +150,6 @@ The quickest way to get started is to use AWS Glue with Ray. Read our [docs](htt | |
- [033 - Amazon Neptune](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/033%20-%20Amazon%20Neptune.ipynb) | ||
- [034 - Distributing Calls Using Ray](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/034%20-%20Distributing%20Calls%20using%20Ray.ipynb) | ||
- [035 - Distributing Calls on Ray Remote Cluster](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/035%20-%20Distributing%20Calls%20on%20Ray%20Remote%20Cluster.ipynb) | ||
- [036 - Distributing Calls with Glue Interactive Sessions on Ray](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/036%20-%20Distributing%20Calls%20with%20Glue%20Interactive%20Sessions%20on%20Ray.ipynb) | ||
- [037 - Glue Data Quality](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/037%20-%20Glue%20Data%20Quality.ipynb) | ||
- [038 - OpenSearch Serverless](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/038%20-%20OpenSearch%20Serverless.ipynb) | ||
- [039 - Athena Iceberg](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/039%20-%20Athena%20Iceberg.ipynb) | ||
|
@@ -198,19 +191,6 @@ You may also find help on these community resources: | |
and tag it with `awswrangler` | ||
* [Runbook](https://github.com/aws/aws-sdk-pandas/discussions/1815) for AWS SDK for pandas with Ray | ||
|
||
## Community Resources | ||
|
||
Please [send a Pull Request](https://github.com/aws/aws-sdk-pandas/edit/main/README.md) with your resource reference and @githubhandle. | ||
|
||
- [YouTube channel](https://www.youtube.com/playlist?list=PL7bE4nSzLSWdDdlfRgfKo2JBplB4p_v5O) [[@AdrianoNicolucci](https://github.com/AdrianoNicolucci)] | ||
- [Optimize Python ETL by extending Pandas with AWS SDK for pandas](https://aws.amazon.com/blogs/big-data/optimize-python-etl-by-extending-pandas-with-aws-data-wrangler/) [[@igorborgest](https://github.com/igorborgest)] | ||
- [Reading Parquet Files With AWS Lambda](https://aprakash.wordpress.com/2020/04/14/reading-parquet-files-with-aws-lambda/) [[@anand086](https://github.com/anand086)] | ||
- [Transform AWS CloudTrail data using AWS SDK for pandas](https://aprakash.wordpress.com/2020/09/17/transform-aws-cloudtrail-data-using-aws-data-wrangler/) [[@anand086](https://github.com/anand086)] | ||
- [Rename Glue Tables using AWS SDK for pandas](https://ananddatastories.com/rename-glue-tables-using-aws-sdk-pandas/) [[@anand086](https://github.com/anand086)] | ||
- [Getting started on AWS SDK for pandas and Athena](https://medium.com/@dheerajsharmainampudi/getting-started-on-aws-sdk-pandas-and-athena-7b446c834076) [[@dheerajsharma21](https://github.com/dheerajsharma21)] | ||
- [Simplifying Pandas integration with AWS data related services](https://medium.com/@bv_subhash/aws-sdk-pandas-simplifying-pandas-integration-with-aws-data-related-services-2b3325c12188) [[@bvsubhash](https://github.com/bvsubhash)] | ||
- [Build an ETL pipeline using AWS S3, Glue and Athena](https://www.linkedin.com/pulse/build-etl-pipeline-using-aws-s3-glue-athena-data-wrangler-tom-reid/) [[@taupirho](https://github.com/taupirho)] | ||
|
||
## Logging | ||
|
||
Enabling internal logging examples: | ||
|
@@ -228,36 +208,3 @@ Into AWS lambda: | |
import logging | ||
logging.getLogger("awswrangler").setLevel(logging.DEBUG) | ||
``` | ||
|
||
## Who uses AWS SDK for pandas? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are we getting rid of this entirely? If so, we should get rid of the issue template as well. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is up for discussion. I think the library is popular enough these days that we don't need public endorsing that can be outdated. But yes good point on the issue template, will delete it |
||
|
||
Knowing which companies are using this library is important to help prioritize the project internally. | ||
If you would like us to include your company’s name and/or logo in the README file to indicate that your company is using the AWS SDK for pandas, please raise a "Support Us" issue. If you would like us to display your company’s logo, please raise a linked pull request to provide an image file for the logo. Note that by raising a Support Us issue (and related pull request), you are granting AWS permission to use your company’s name (and logo) for the limited purpose described here and you are confirming that you have authority to grant such permission. | ||
|
||
- [Amazon](https://www.amazon.com/) | ||
- [AWS](https://aws.amazon.com/) | ||
- [Cepsa](https://cepsa.com) [[@alvaropc](https://github.com/alvaropc)] | ||
- [Cognitivo](https://www.cognitivo.ai/) [[@msantino](https://github.com/msantino)] | ||
- [Digio](https://www.digio.com.br/) [[@afonsomy](https://github.com/afonsomy)] | ||
- [DNX](https://www.dnx.solutions/) [[@DNXLabs](https://github.com/DNXLabs)] | ||
- [Fortescue Future Industries](https://ffi.com.au/) [[@spencervoorend](https://github.com/spencervoorend)] | ||
- [Funcional Health Tech](https://www.funcionalcorp.com.br/) [[@webysther](https://github.com/webysther)] | ||
- [Funding Circle](https://www.fundingcircle.com/) [[@pfig](https://github.com/pfig)] | ||
- [Infomach](https://www.infomach.com.br/) | ||
- [Informa Markets](https://www.informamarkets.com/en/home.html) [[@mateusmorato]](http://github.com/mateusmorato) | ||
- [LINE TV](https://www.linetv.tw/) [[@bryanyang0528](https://github.com/bryanyang0528)] | ||
- [LogicalCube](https://www.logicalcube.com) [[@zolabud](https://github.com/zolabud)] | ||
- [Magnataur](https://magnataur.com) [[@brianmingus2](https://github.com/brianmingus2)] | ||
- [M4U](https://www.m4u.com.br/) [[@Thiago-Dantas](https://github.com/Thiago-Dantas)] | ||
- [NBCUniversal](https://www.nbcuniversal.com/) [[@vibe](https://github.com/vibe)] | ||
- [nrd.io](https://nrd.io/) [[@mrtns](https://github.com/mrtns)] | ||
- [OKRA Technologies](https://okra.ai) [[@JPFrancoia](https://github.com/JPFrancoia), [@schot](https://github.com/schot)] | ||
- [Pier](https://www.pier.digital/) [[@flaviomax](https://github.com/flaviomax)] | ||
- [Pismo](https://www.pismo.io/) [[@msantino](https://github.com/msantino)] | ||
- [ringDNA](https://www.ringdna.com/) [[@msropp](https://github.com/msropp)] | ||
- [Serasa Experian](https://www.serasaexperian.com.br/) [[@andre-marcos-perez](https://github.com/andre-marcos-perez)] | ||
- [Shipwell](https://shipwell.com/) [[@zacharycarter](https://github.com/zacharycarter)] | ||
- [strongDM](https://www.strongdm.com/) [[@mrtns](https://github.com/mrtns)] | ||
- [Thinkbumblebee](https://www.thinkbumblebee.com/) [[@dheerajsharma21]](https://github.com/dheerajsharma21) | ||
- [VTEX](https://vtex.com/us-en/) [[@igorborgest]](https://github.com/igorborgest) | ||
- [Zillow](https://www.zillow.com/) [[@nicholas-miles]](https://github.com/nicholas-miles) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we want this to be a "local" path. For example, if I'm on the
release/2.x.
branch, I would want the readme to display that version of the logo.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was just being consistent with the rest of the README but perhaps we should make the change everywhere else