SmallPond on S3 Object Store Integration

[soumilshah1995]

SmallPond on S3 Object Store Integration

I’ve been experimenting with smallpond locally, and it works great for processing large datasets. I was able to read a Parquet file, run some SQL queries, and successfully convert the results to a pandas DataFrame. Now, I’m curious about the configuration or settings required to make it work with an S3 object store.

What are the necessary configurations to connect SmallPond to an S3 bucket, and do I need any specific credentials or access settings to enable seamless integration? Any guidance or best practices would be greatly appreciated.

[mike-luabase]

here's a fork with s3 support https://github.com/definite-app/smallpond. I don't think I'll have time to test and contribute this upstream as it doesn't seem smallpond is going to work for my use-case, but feel free to use it!

[soumilshah1995]

@mike-luabase do you have any examples or any configuration we need to set or simply we can pass S3 URI ?

[mike-luabase]

@mike-luabase do you have any examples or any configuration we need to set or simply we can pass S3 URI ?

take a look at test_s3.py

[soumilshah1995]

Works tried your fork thanks a lot