Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: simplify README, remove AWS Glue for Ray references #2750

Merged
merged 4 commits into from
Apr 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 0 additions & 19 deletions .github/ISSUE_TEMPLATE/support-data-wrangler.md

This file was deleted.

57 changes: 2 additions & 55 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
# AWS SDK for pandas (awswrangler)

AWS Data Wrangler is now **AWS SDK for pandas (awswrangler)**. We’re changing the name we use when we talk about the library, but everything else will stay the same. You’ll still be able to install using `pip install awswrangler` and you won’t need to change any of your code. As part of this change, we’ve moved the library from AWS Labs to the main AWS GitHub organisation but, thanks to the GitHub’s redirect feature, you’ll still be able to access the project by its old URLs until you update your bookmarks. Our documentation has also moved to [aws-sdk-pandas.readthedocs.io](https://aws-sdk-pandas.readthedocs.io), but old bookmarks will redirect to the new site.

*Pandas on AWS*

Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

![AWS SDK for pandas](docs/source/_static/logo2.png?raw=true "AWS SDK for pandas")
![AWS SDK for pandas](https://github.com/aws/aws-sdk-pandas/blob/main/docs/source/_static/logo2.png?raw=true "AWS SDK for pandas")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we want this to be a "local" path. For example, if I'm on the release/2.x. branch, I would want the readme to display that version of the logo.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just being consistent with the rest of the README but perhaps we should make the change everywhere else

![tracker](https://d3tiqpr4kkkomd.cloudfront.net/img/pixel.png?asset=GVOYN2BOOQ573LTVIHEW)

> An [AWS Professional Service](https://aws.amazon.com/professional-services/) open source initiative | aws-proserve-opensource@amazon.com
Expand All @@ -29,17 +27,13 @@ Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, Q
> ⚠️ **Starting version 3.0, optional modules must be installed explicitly:**<br>
➡️`pip install 'awswrangler[redshift]'`

Powered By [<img src="https://arrow.apache.org/img/arrow.png" width="200">](https://arrow.apache.org/powered_by/)

## Table of contents

- [Quick Start](#quick-start)
- [At Scale](#at-scale)
- [Read The Docs](#read-the-docs)
- [Getting Help](#getting-help)
- [Community Resources](#community-resources)
- [Logging](#logging)
- [Who uses AWS SDK for pandas?](#who-uses-aws-sdk-for-pandas)

## Quick Start

Expand Down Expand Up @@ -100,7 +94,7 @@ FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3
## At scale
AWS SDK for pandas can also run your workflows at scale by leveraging [Modin](https://modin.readthedocs.io/en/stable/) and [Ray](https://www.ray.io/). Both projects aim to speed up data workloads by distributing processing over a cluster of workers.

The quickest way to get started is to use AWS Glue with Ray. Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/scale.html), our blogs ([1](https://aws.amazon.com/blogs/big-data/scale-aws-sdk-for-pandas-workloads-with-aws-glue-for-ray/)/[2](https://aws.amazon.com/blogs/big-data/advanced-patterns-with-aws-sdk-for-pandas-on-aws-glue-for-ray/)), or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to discover even more features.
Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/scale.html) or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to learn more.

> ⚠️ **Ray is currently not available for Python 3.12. While AWS SDK for pandas supports Python 3.12, it cannot be used at scale.**

Expand Down Expand Up @@ -156,7 +150,6 @@ The quickest way to get started is to use AWS Glue with Ray. Read our [docs](htt
- [033 - Amazon Neptune](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/033%20-%20Amazon%20Neptune.ipynb)
- [034 - Distributing Calls Using Ray](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/034%20-%20Distributing%20Calls%20using%20Ray.ipynb)
- [035 - Distributing Calls on Ray Remote Cluster](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/035%20-%20Distributing%20Calls%20on%20Ray%20Remote%20Cluster.ipynb)
- [036 - Distributing Calls with Glue Interactive Sessions on Ray](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/036%20-%20Distributing%20Calls%20with%20Glue%20Interactive%20Sessions%20on%20Ray.ipynb)
- [037 - Glue Data Quality](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/037%20-%20Glue%20Data%20Quality.ipynb)
- [038 - OpenSearch Serverless](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/038%20-%20OpenSearch%20Serverless.ipynb)
- [039 - Athena Iceberg](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/039%20-%20Athena%20Iceberg.ipynb)
Expand Down Expand Up @@ -198,19 +191,6 @@ You may also find help on these community resources:
and tag it with `awswrangler`
* [Runbook](https://github.com/aws/aws-sdk-pandas/discussions/1815) for AWS SDK for pandas with Ray

## Community Resources

Please [send a Pull Request](https://github.com/aws/aws-sdk-pandas/edit/main/README.md) with your resource reference and @githubhandle.

- [YouTube channel](https://www.youtube.com/playlist?list=PL7bE4nSzLSWdDdlfRgfKo2JBplB4p_v5O) [[@AdrianoNicolucci](https://github.com/AdrianoNicolucci)]
- [Optimize Python ETL by extending Pandas with AWS SDK for pandas](https://aws.amazon.com/blogs/big-data/optimize-python-etl-by-extending-pandas-with-aws-data-wrangler/) [[@igorborgest](https://github.com/igorborgest)]
- [Reading Parquet Files With AWS Lambda](https://aprakash.wordpress.com/2020/04/14/reading-parquet-files-with-aws-lambda/) [[@anand086](https://github.com/anand086)]
- [Transform AWS CloudTrail data using AWS SDK for pandas](https://aprakash.wordpress.com/2020/09/17/transform-aws-cloudtrail-data-using-aws-data-wrangler/) [[@anand086](https://github.com/anand086)]
- [Rename Glue Tables using AWS SDK for pandas](https://ananddatastories.com/rename-glue-tables-using-aws-sdk-pandas/) [[@anand086](https://github.com/anand086)]
- [Getting started on AWS SDK for pandas and Athena](https://medium.com/@dheerajsharmainampudi/getting-started-on-aws-sdk-pandas-and-athena-7b446c834076) [[@dheerajsharma21](https://github.com/dheerajsharma21)]
- [Simplifying Pandas integration with AWS data related services](https://medium.com/@bv_subhash/aws-sdk-pandas-simplifying-pandas-integration-with-aws-data-related-services-2b3325c12188) [[@bvsubhash](https://github.com/bvsubhash)]
- [Build an ETL pipeline using AWS S3, Glue and Athena](https://www.linkedin.com/pulse/build-etl-pipeline-using-aws-s3-glue-athena-data-wrangler-tom-reid/) [[@taupirho](https://github.com/taupirho)]

## Logging

Enabling internal logging examples:
Expand All @@ -228,36 +208,3 @@ Into AWS lambda:
import logging
logging.getLogger("awswrangler").setLevel(logging.DEBUG)
```

## Who uses AWS SDK for pandas?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we getting rid of this entirely? If so, we should get rid of the issue template as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is up for discussion. I think the library is popular enough these days that we don't need public endorsing that can be outdated. But yes good point on the issue template, will delete it


Knowing which companies are using this library is important to help prioritize the project internally.
If you would like us to include your company’s name and/or logo in the README file to indicate that your company is using the AWS SDK for pandas, please raise a "Support Us" issue. If you would like us to display your company’s logo, please raise a linked pull request to provide an image file for the logo. Note that by raising a Support Us issue (and related pull request), you are granting AWS permission to use your company’s name (and logo) for the limited purpose described here and you are confirming that you have authority to grant such permission.

- [Amazon](https://www.amazon.com/)
- [AWS](https://aws.amazon.com/)
- [Cepsa](https://cepsa.com) [[@alvaropc](https://github.com/alvaropc)]
- [Cognitivo](https://www.cognitivo.ai/) [[@msantino](https://github.com/msantino)]
- [Digio](https://www.digio.com.br/) [[@afonsomy](https://github.com/afonsomy)]
- [DNX](https://www.dnx.solutions/) [[@DNXLabs](https://github.com/DNXLabs)]
- [Fortescue Future Industries](https://ffi.com.au/) [[@spencervoorend](https://github.com/spencervoorend)]
- [Funcional Health Tech](https://www.funcionalcorp.com.br/) [[@webysther](https://github.com/webysther)]
- [Funding Circle](https://www.fundingcircle.com/) [[@pfig](https://github.com/pfig)]
- [Infomach](https://www.infomach.com.br/)
- [Informa Markets](https://www.informamarkets.com/en/home.html) [[@mateusmorato]](http://github.com/mateusmorato)
- [LINE TV](https://www.linetv.tw/) [[@bryanyang0528](https://github.com/bryanyang0528)]
- [LogicalCube](https://www.logicalcube.com) [[@zolabud](https://github.com/zolabud)]
- [Magnataur](https://magnataur.com) [[@brianmingus2](https://github.com/brianmingus2)]
- [M4U](https://www.m4u.com.br/) [[@Thiago-Dantas](https://github.com/Thiago-Dantas)]
- [NBCUniversal](https://www.nbcuniversal.com/) [[@vibe](https://github.com/vibe)]
- [nrd.io](https://nrd.io/) [[@mrtns](https://github.com/mrtns)]
- [OKRA Technologies](https://okra.ai) [[@JPFrancoia](https://github.com/JPFrancoia), [@schot](https://github.com/schot)]
- [Pier](https://www.pier.digital/) [[@flaviomax](https://github.com/flaviomax)]
- [Pismo](https://www.pismo.io/) [[@msantino](https://github.com/msantino)]
- [ringDNA](https://www.ringdna.com/) [[@msropp](https://github.com/msropp)]
- [Serasa Experian](https://www.serasaexperian.com.br/) [[@andre-marcos-perez](https://github.com/andre-marcos-perez)]
- [Shipwell](https://shipwell.com/) [[@zacharycarter](https://github.com/zacharycarter)]
- [strongDM](https://www.strongdm.com/) [[@mrtns](https://github.com/mrtns)]
- [Thinkbumblebee](https://www.thinkbumblebee.com/) [[@dheerajsharma21]](https://github.com/dheerajsharma21)
- [VTEX](https://vtex.com/us-en/) [[@igorborgest]](https://github.com/igorborgest)
- [Zillow](https://www.zillow.com/) [[@nicholas-miles]](https://github.com/nicholas-miles)
5 changes: 2 additions & 3 deletions docs/source/scale.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@ At scale
=========

AWS SDK for pandas supports `Ray <https://www.ray.io/>`_ and `Modin <https://modin.readthedocs.io/en/stable/>`_, enabling you to scale your pandas workflows from a single machine to a multi-node environment, with no code changes.

The simplest way to try this is with `AWS Glue for Ray <https://aws.amazon.com/blogs/big-data/introducing-aws-glue-for-ray-scaling-your-data-integration-workloads-using-python/>`_, the new serverless option to run distributed Python code announced at AWS re:Invent 2022. AWS SDK for pandas also supports self-managed Ray on `Amazon Elastic Compute Cloud (Amazon EC2) <https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/035%20-%20Distributing%20Calls%20on%20Ray%20Remote%20Cluster.ipynb>`_.
jaidisido marked this conversation as resolved.
Show resolved Hide resolved
Head to our `tutorial <https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/035%20-%20Distributing%20Calls%20on%20Ray%20Remote%20Cluster.ipynb>`_ to set up a self-managed Ray cluster on Amazon Elastic Compute Cloud (Amazon EC2).

Getting Started
----------------
Expand Down Expand Up @@ -178,6 +177,6 @@ As a result, passing ``boto3_session`` when using the Ray runtime is not support
To learn more
--------------

Read our blog posts `(1) <https://aws.amazon.com/blogs/big-data/scale-aws-sdk-for-pandas-workloads-with-aws-glue-for-ray/>`_ and `(2) <https://aws.amazon.com/blogs/big-data/advanced-patterns-with-aws-sdk-for-pandas-on-aws-glue-for-ray/>`_, then head to our latest `tutorials <https://aws-sdk-pandas.readthedocs.io/en/stable/tutorials.html>`_ to discover even more features.
Head to our latest `tutorials <https://aws-sdk-pandas.readthedocs.io/en/stable/tutorials.html>`_ to discover even more features.

A runbook with common errors when running the library with Ray is available `here <https://github.com/aws/aws-sdk-pandas/discussions/1815>`_.
Loading