Datalake Studio is an enhanced Data Exploration and Management tool
Quick for big data: Datalake Studio is built on top of DuckDB, a high-performance, embedded SQL OLAP database management system. DuckDB is designed to handle large datasets, making it ideal for data exploration and analysis.
See your data: Plot automatically your data or see data over a map: Points, H3 aggregations, etc
Versatile Data Loading Options: Users can effortlessly upload data from a several sources: directly from local computer, via a URL, or from an Amazon S3 bucket. Additionally, it supports direct data downloads from PostgreSQL databases, enhancing its utility for database administrators and data analysts.
Several data formats: Wide range of data formats, Datalake Studio is compatible with CSV, TSV, Parquet and Shapefile formats. Load data without tedious conversions.
ChatGPT Integration with SQL Assistants: Users with ChatGPT credentials can use the power of SQL assistants. These assistants provide contextual understanding about your tables and fields, making data manipulation and query formulation more intuitive and efficient.
Enhancement through Remote APIs: Users have the ability to enrich their data by integrating information from remote APIs.
API Exposure for Data Sharing: After completing data transformation processes, users can expose their data through APIs. This feature allows for easy sharing and collaboration, making Datalake Studio not just a tool for data exploration, but also a platform for data distribution.
docker-compose up --build
Open http://localhost:8080/ in your browser.
##Â If you dont want to use compose
docker build -t datalakestudioserver . docker run --name datalakestudioserver -p 8000:8000 datalakestudioserver
docker build -t datalakestudiofront . docker run --name datalakestudiofront -p 8080:8080 datalakestudiofront
Inside server folder run:
pip3 install -r requirements.txt
python3 server.py
If you want to use venv:
python3 -m venv venv
source venv/bin/activate
Exit venv:
deactivate
Inside the client folder of the project, run these commands to build the Vue UI project:
npm install
npm run dev -- --port 8080
Open http://localhost:8080/ in your browser.
Inside server folder create a file named config.yml. Example:
port: 8000
database: "data/datalakeStudio.db"
And another file named secrets.yml with properties:
# Optional for DuckDB to work with S3, if not defined, user aws credentials will be loaded through the AWS Default Credentials Provider Chain
s3_access_key_id: "YOUR_S3_ACCESS_KEY_ID"
s3_secret_access_key: "YOUR_S3_SECRET_ACCESS_KEY"
# For OpenAI
openai_organization: "YOUR_OPENAI_ORGANIZATION"
openai_api_key: "YOUR_OPENAI_API_KEY"
# For API search
api_domain: "YOUR_API_DOMAIN"
api_context: "YOUR_API_CONTEXT"
# Database connections
pgpass_file: "YOUR_PG_PASS_FILE"
# Mapbox
mapbox_access_token: "YOUR_MAPBOX_ACCESS_TOKEN"
Also, docker-compose will get the credentials in .aws for AWS access.
If you want to use remote database, copy your pgpass file to the server folder. pgpass is a file with the following format:
hostname:port:database:username:password
You can load data from local filesystem, from any URL or from S3. Try to load this example: https://raw.githubusercontent.com/javitorres/GenericCross/main/public/data/iris.csv
Inspect loaded data. Export data to CSV or Parquet
Get data profile
or use crossfilter to play with your data
If your data has spatial info you can see in a map:
Query your data and generate new tables. Save or load your queries. Use ChatGPT to create new queries
Enrich your datasets calling external APIs
New table:
Explore your external databases and load data into Datalake Studio for local analysis
Publish endpoints serving your data with parametrized queries:
Keep control of endpoints published:
Move in your S3 buckets and write descriptions
Preview files or load them into DatalaleStudio
Talk to explore your data (experimental)
