✔️ Linux ✔️ OS X ❌ Windows (#39)

Welcome to graph-app-kit

Turn your graph data into a secure and interactive visual graph app in 15 minutes!

Why

This open source effort puts together patterns the Graphistry team has reused across many graph projects as teams go from code-heavy Jupyter notebook experiments to deploying streamlined analyst tools. Whether building your first graph app, trying an idea, or wanting to check a reference, this project aims to simplify that process. It covers pieces like: Easy code editing and deployment, a project stucture ready for teams, built-in authentication, no need for custom JS/CSS at the start, batteries-included data + library dependencies, and fast loading & visualization of large graphs.

What

• Minimal core: The barebones dashboard server. In provides a StreamLit docker-compose container with PyData ecosystem libraries and examples of visualizing data from various systems. Install it, plug in credentials to various web services like cloud databases and a free Graphistry Hub visualization account, and launch. It does not have GPU ETL and GPU AI libraries.

• Full core: Initially for AWS, the full core bundles adds to the docker-compose system: Accounts, Jupyter notebooks for authoring, serves StreamLit dashboards with both public + private zones, and runs Graphistry/RAPIDS locally on the same server. Launch with on click via the Cloud Formation template.

• Full core + DB: DB-specific variants are the same as minimal/full, and add simpler DB-specific quick launching/connecting.

Get started

Quick (Local code) - full GPU core + third-party connectors

Note: Base image includes Nvidia RAPIDS and AI dependencies so is quite large, see CPU alternative for a lightweight alternativve

Note: Use sudo for docker-compose commands if your configuration requires it and is giving permission error

# Minimal core
git clone https://github.com/graphistry/graph-app-kit.git
cd graph-app-kit/src/docker

# Enable docker buildkit
# ... or run docker-compose via provided alias script `./dc`
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1

# Build
docker-compose build

# Optional: Edit src/docker/.env (API accounts), docker-compose.yml: Auth, ports, ...

# Launch
docker-compose up -d
docker-compose logs -f -t --tail=100

=> http://localhost:8501/

To add views and relaunch:

# Add dashboards @ src/python/views/<your_custom_view>/__init__.py

docker-compose up -d --force-recreate

Quick (Local code) - minimal CPU core + third-party connectors

Same commands as above, but use ./dc.cpu, which aliases docker-compose -f docker-compose.yml -f override/cpu.override.yml:

git clone https://github.com/graphistry/graph-app-kit.git
cd graph-app-kit/src/docker
./dc.cpu build
...
./dc.cpu up

Quick Launchers - minimal/full core

1. Quick launch options:

Full:

• Public + protected Streamlit dashboards, Jupyter notebooks + editing, Graphistry, RAPIDS
• Login to web UI as admin / i-instanceid -> file uploader, notebooks, ...
• Dashboards: /public/dash and /private/dash
• More info

Admin:

# launch logs
tail -f /var/log/cloud-init-output.log -n 1000

# app logs
sudo docker ps
sudo docker logs -f -t --tail=1 MY_CONTAINER

# restart a graphistry container
cd graphistry && sudo docker-compose restart MY_CONTAINER

# restart caddy (Caddy 1 override)
cd graphistry && sudo docker-compose -f docker-compose.gak.graphistry.yml up -d caddy

# run streamlit
cd graph-app-kit/public/graph-app-kit && docker-compose -p pub run -d --name streamlit-pub streamlit
cd graph-app-kit/private/graph-app-kit && docker-compose -p priv run -d --name streamlit-priv streamlit

Minimal: Open Streamlit, ssh to connect/add free Graphistry Hub username/pass:

Database-specific: Amazon Neptune, TigerGraph

2. Add views

3. Main configurations and extensions: Database connectors, authentication, notebook-based editing, and more

Native (Experimental)

Install dependencies, pick location of views folder, and run:

cd src/python
pip3 install -r requirements-system.txt
pip3 install -r requirements-app.txt
VIEW_PATH="`pwd`/views" streamlit run entrypoint.py 

The pieces

Core

• Prebuilt Python project structure ready for prototyping

• Streamlit quick self-serve dashboarding

• Graphistry point-and-click GPU-accelerated visual graph analytics

• Data frames: Data wrangling via Pandasand Apache Arrow, including handling formats such as CSV, XLS, JSON, Parquet, and more

• Standard Docker and docker-compose cross-platform deployment

GPU acceleration (optional) - Full

For non-minimal installs, if GPUs are present, graph-app-kit leverages GPU cloud acceleration:

• GPU Analytics: RAPIDS and CUDA already setup for use if run with an Nvidia docker runtime - cudf GPU dataframes, BlazingSQL GPU SQL, cuGraph GPU graph algorithms, cuML libraries, and more

• GPU Visualization: Connect to an external Graphistry server or, faster, run on the same GPU server

Prebuilt integrations & recipes

graph-app-kit works well with the Python data ecosystem (pandas, cudf, PySpark, SQL, ...) and we're growing the set of builtins and recipes:

• Graph databases

  • AWS Neptune: quick launch and manual launch

  • TinkerPop Gremlin: query demos

  • TigerGraph: setup

  • Collaborations welcome!

• Jupyter notebooks: Use quick launchers or integrations guide for web-based live editing of dashboards by sharing volume mounts between Jupyter and Streamlit

• Caddy: Reverse proxy for custom URLs, automatic LetsEncrypt TLS certificates, multiple sites on the same domain, pluggable authentication (see integrations guide)

• Also you can install your custom Python Packages.

Contribute

We welcome all sorts of help!

• Deployment: Docker, cloud runners, ...
• Dependencies: Common graph packages
• Connectors: Examples for common databases and how to get a lot of data out
• Demos!

See DEVELOP.md for more contributor information