Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:

strategy:
matrix:
python-version: [ 3.7, 3.8, 3.9 ]
python-version: [ "3.10", "3.11", "3.12", "3.13" ]

steps:

Expand Down
4 changes: 2 additions & 2 deletions DEVELOP.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ These instructions are for contributing to this repository. See main README.md f

```bash
cd src/docker
docker-compose build
docker-compose up
docker compose build
docker compose up
```

## Test
Expand Down
26 changes: 13 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ This open source effort puts together patterns the Graphistry team has reused ac

## What

* **Minimal core**: The barebones dashboard server. In provides a StreamLit docker-compose container with PyData ecosystem libraries and examples of visualizing data from various systems. Install it, plug in credentials to various web services like cloud databases and a free [Graphistry Hub](https://hub.graphistry.com) visualization account, and launch. It does not have GPU ETL and GPU AI libraries.
* **Minimal core**: The barebones dashboard server. In provides a StreamLit docker compose container with PyData ecosystem libraries and examples of visualizing data from various systems. Install it, plug in credentials to various web services like cloud databases and a free [Graphistry Hub](https://hub.graphistry.com) visualization account, and launch. It does not have GPU ETL and GPU AI libraries.

* **Full core**: Initially for AWS, the full core bundles adds to the docker-compose system: Accounts, Jupyter notebooks for authoring, serves StreamLit dashboards with both public + private zones, and runs Graphistry/RAPIDS locally on the same server. Launch with on click via the Cloud Formation template.
* **Full core**: Initially for AWS, the full core bundles adds to the docker compose system: Accounts, Jupyter notebooks for authoring, serves StreamLit dashboards with both public + private zones, and runs Graphistry/RAPIDS locally on the same server. Launch with on click via the Cloud Formation template.

* **Full core + DB**: DB-specific variants are the same as minimal/full, and add simpler DB-specific quick launching/connecting.

Expand All @@ -36,26 +36,26 @@ This open source effort puts together patterns the Graphistry team has reused ac

**Note**: Base image includes Nvidia RAPIDS and AI dependencies so is quite large, see CPU alternative for a lightweight alternativve

**Note**: Use `sudo` for docker-compose commands if your configuration requires it and is giving permission error
**Note**: Use `sudo` for docker compose commands if your configuration requires it and is giving permission error

```bash
# Minimal core
git clone https://github.com/graphistry/graph-app-kit.git
cd graph-app-kit/src/docker

# Enable docker buildkit
# ... or run docker-compose via provided alias script `./dc`
# ... or run docker compose via provided alias script `./dc`
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1

# Build
docker-compose build
docker compose build

# Optional: Edit src/docker/.env (API accounts), docker-compose.yml: Auth, ports, ...

# Launch
docker-compose up -d
docker-compose logs -f -t --tail=100
docker compose up -d
docker compose logs -f -t --tail=100
```

=> `http://localhost:8501/`
Expand All @@ -65,7 +65,7 @@ To [add views](docs/views.md) and relaunch:
```bash
# Add dashboards @ src/python/views/<your_custom_view>/__init__.py

docker-compose up -d --force-recreate
docker compose up -d --force-recreate
```

### Quick (Local code) - minimal CPU core + third-party connectors
Expand Down Expand Up @@ -102,14 +102,14 @@ sudo docker ps
sudo docker logs -f -t --tail=1 MY_CONTAINER

# restart a graphistry container
cd graphistry && sudo docker-compose restart MY_CONTAINER
cd graphistry && sudo docker compose restart MY_CONTAINER

# restart caddy (Caddy 1 override)
cd graphistry && sudo docker-compose -f docker-compose.gak.graphistry.yml up -d caddy
cd graphistry && sudo docker compose -f docker-compose.gak.graphistry.yml up -d caddy

# run streamlit
cd graph-app-kit/public/graph-app-kit && docker-compose -p pub run -d --name streamlit-pub streamlit
cd graph-app-kit/private/graph-app-kit && docker-compose -p priv run -d --name streamlit-priv streamlit
cd graph-app-kit/public/graph-app-kit && docker compose -p pub run -d --name streamlit-pub streamlit
cd graph-app-kit/private/graph-app-kit && docker compose -p priv run -d --name streamlit-priv streamlit
```

**Minimal**: Open Streamlit, ssh to connect/add [free Graphistry Hub username/pass](https://www.graphistry.com/get-started):
Expand Down Expand Up @@ -140,7 +140,7 @@ VIEW_PATH="`pwd`/views" streamlit run entrypoint.py
* [Graphistry](https://www.graphistry.com/get-started) point-and-click GPU-accelerated visual graph analytics
* Data frames: Data wrangling via [Pandas](https://pandas.pydata.org/)and [Apache Arrow](https://arrow.apache.org/), including handling formats such as CSV, XLS, JSON, Parquet, and more

* Standard Docker and docker-compose cross-platform deployment
* Standard Docker and docker compose cross-platform deployment

### GPU acceleration (optional) - Full

Expand Down
4 changes: 2 additions & 2 deletions docs/extend.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Integration settings that deal with external systems such as TLS, accounts, and

By default, `graph-app-kit` logging uses the Docker json file driver:

* Inspect recent activities: `cd src/docker` and then `sudo docker-compose logs -f -t --tail=100`
* Inspect recent activities: `cd src/docker` and then `sudo docker compose logs -f -t --tail=100`

* Setup [alternative logging drivers](https://docs.docker.com/config/containers/logging/configure/)

Expand All @@ -24,7 +24,7 @@ By default, `graph-app-kit` logging uses the Docker json file driver:

## TLS with Cadddy

* Auth: See [Caddy sample](src/caddy/Caddyfile) reverse proxy example for an authentication check against an account system, including the one shipping with your Graphistry server (requires `sudo docker-compose restart caddy` in your Graphistry server upon editing `/var/graphistry/data/config/Caddyfile`)
* Auth: See [Caddy sample](src/caddy/Caddyfile) reverse proxy example for an authentication check against an account system, including the one shipping with your Graphistry server (requires `sudo docker compose restart caddy` in your Graphistry server upon editing `/var/graphistry/data/config/Caddyfile`)

## Public+Private views
* To simulatenously run 1 public and 1 private instance, create two `graph-app-kit` clones `public_dash` and `private_dash`, and for `src/docker/.env`, set:
Expand Down
6 changes: 3 additions & 3 deletions docs/neptune-manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@ Reset and restart your `graph-app-kit` container:

```bash
cd src/docker
sudo docker-compose down -v
sudo docker-compose up -d
sudo docker compose down -v
sudo docker compose up -d
```

Watch logs with `sudo docker-compose logs -f -t --tail=1`
Watch logs with `sudo docker compose logs -f -t --tail=1`

Access your Streamlit instance at http://the.public.ip.address:8501

Expand Down
6 changes: 3 additions & 3 deletions docs/neptune.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,11 @@ Launch using a button at the bottom of the [identity graph sample cloud formatio
* **Streamlit: Public dashboards**
* **http://[the.public.ip.address]/public/dash**
* Installed at `/home/ubuntu/graph-app-kit/public/graph-app-kit`
* Run as `src/docker $ docker-compose -p pub run -d --name streamlit-pub streamlit`
* Run as `src/docker $ docker compose -p pub run -d --name streamlit-pub streamlit`
* **Streamlit: Private dashboards**
* **http://[the.public.ip.address]/private/dash**
* Installed at `/home/ubuntu/graph-app-kit/private/graph-app-kit`
* Run as `src/docker $ docker-compose -p priv run -d --name streamlit-priv streamlit`
* Run as `src/docker $ docker compose -p priv run -d --name streamlit-priv streamlit`
* **Jupyter: Data science notebooks + Streamlit dashboard live-editing**
* **http://[the.public.ip.address]/notebook**
* Live-edit `graph-app-kit` view folders `notebook/graph-app-kit/[public,private]/views`
Expand All @@ -126,7 +126,7 @@ Launch using a button at the bottom of the [identity graph sample cloud formatio
* **Streamlit: Public dashboards**
* **http://[the.public.ip.address]/public/dash**
* Installed at `/home/ubuntu/graph-app-kit/public/graph-app-kit`
* Run as `src/docker $ docker-compose up -d streamlit`
* Run as `src/docker $ docker compose up -d streamlit`

## 4. Next steps

Expand Down
6 changes: 3 additions & 3 deletions docs/setup-manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ For quick launchers, see the [AWS quick launch setup guide](setup.md).

* Ubuntu 18.04 LTS is the most common choice for containerized GPU computing

* Install docker-ce and docker-compose
* Install docker-ce and docker compose

* Optional:
* GPU: If you have a [RAPIDS.ai](https://www.rapids.ai)-compatible GPU (see below), install the Nvidia docker runtime and set it as the default for Docker daemons
Expand Down Expand Up @@ -57,9 +57,9 @@ GRAPHISTRY_PASSWORD=your_password

`cd src/docker` and then:

* Start: `sudo docker-compose up -d`
* Start: `sudo docker compose up -d`
* Use: Go to `http://localhost:8501/dashboard` (or whatever the public IP)
* Stop: `sudo docker-compose down -v`
* Stop: `sudo docker compose down -v`

## Graph!

Expand Down
16 changes: 8 additions & 8 deletions docs/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,11 +85,11 @@ Go to your public Streamlit dashboard and start exploring: http://[the.public.ip
* **Streamlit: Public dashboards**
* **http://[the.public.ip.address]/public/dash**
* Installed at `/home/ubuntu/graph-app-kit/public/graph-app-kit`
* Run as `src/docker $ docker-compose -p pub run -d --name streamlit-pub streamlit`
* Run as `src/docker $ docker compose -p pub run -d --name streamlit-pub streamlit`
* **Streamlit: Private dashboards**
* **http://[the.public.ip.address]/private/dash**
* Installed at `/home/ubuntu/graph-app-kit/private/graph-app-kit`
* Run as `src/docker $ docker-compose -p priv run -d --name streamlit-priv streamlit`
* Run as `src/docker $ docker compose -p priv run -d --name streamlit-priv streamlit`
* **Jupyter: Data science notebooks + Streamlit dashboard live-editing**
* **http://[the.public.ip.address]/notebook**
* Live-edit `graph-app-kit` view folders `notebook/graph-app-kit/[public,private]/views`
Expand All @@ -99,7 +99,7 @@ Go to your public Streamlit dashboard and start exploring: http://[the.public.ip
* **Streamlit: Public dashboards**
* **http://[the.public.ip.address]/public/dash**
* Installed at `/home/ubuntu/graph-app-kit/public/graph-app-kit`
* Run as `src/docker $ docker-compose up -d streamlit`
* Run as `src/docker $ docker compose up -d streamlit`

## 4. Optional - administer

Expand Down Expand Up @@ -127,19 +127,19 @@ For more advanced Graphistry administration, so the [Graphistry admin docs repo]

```bash
# restart a graphistry container
cd graphistry && sudo docker-compose restart MY_CONTAINER # do *not* run it's caddy (v2)
cd graphistry && sudo docker compose restart MY_CONTAINER # do *not* run it's caddy (v2)

# restart caddy (Caddy 1 override over Graphistry's Caddy 2)
cd graphistry && sudo docker-compose -f docker-compose.gak.graphistry.yml up -d caddy
cd graphistry && sudo docker compose -f docker-compose.gak.graphistry.yml up -d caddy
```

### Streamlit

Use `docker-compose` project names (`-p the_name`) to distinguish your public vs private dashboards:
Use `docker compose` project names (`-p the_name`) to distinguish your public vs private dashboards:

```bash
cd graph-app-kit/public/graph-app-kit && docker-compose -p pub run -d --name streamlit-pub streamlit
cd graph-app-kit/private/graph-app-kit && docker-compose -p priv run -d --name streamlit-priv streamlit
cd graph-app-kit/public/graph-app-kit && docker compose -p pub run -d --name streamlit-pub streamlit
cd graph-app-kit/private/graph-app-kit && docker compose -p priv run -d --name streamlit-priv streamlit
```

## 5. Next steps
Expand Down
2 changes: 1 addition & 1 deletion docs/tigergraph.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ TIGERGRAPH_GRAPHNAME=AntiFraud
TIGERGRAPH_SECRET=mysecret
```

3. Restart your Streamlit container with the new creds: `cd src/docker && sudo docker-compose up -d --force-recreate`
3. Restart your Streamlit container with the new creds: `cd src/docker && sudo docker compose up -d --force-recreate`

## Explore

Expand Down
2 changes: 1 addition & 1 deletion docs/views.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Not required but recommended, we recommend setting up Jupyter-based shared web a

* Modify Python files in `src/python/views/[your dashboard]/__init__.py`, and in-tool, hit the `rerun` button that appears
* Add new views by adding `views/[your dsahboard]/__init__.py` with methods `def info(): return {'name': 'x'}` and `def run(): None`
* Add new dependencies: modify `src/python/requirements-app.txt` and rerun `docker-compose build` and restart
* Add new dependencies: modify `src/python/requirements-app.txt` and rerun `docker compose build` and restart

## Toggle views

Expand Down
4 changes: 2 additions & 2 deletions src/bootstraps/core/graphistry.sh
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,11 @@ echo '----- Finish pub vs. priv .env specialization'
echo '----- Launching graph-app-kit as streamlit-pub/priv:8501'
( \
cd "${GAK_PUBLIC}/src/docker" \
&& sudo docker-compose -p pub run -d --name streamlit-pub streamlit \
&& sudo docker compose -p pub run -d --name streamlit-pub streamlit \
)
( \
cd "${GAK_PRIVATE}/src/docker" \
&& sudo docker-compose -p priv run -d --name streamlit-priv streamlit \
&& sudo docker compose -p priv run -d --name streamlit-priv streamlit \
)

./hello-end.sh "$SCRIPT"
4 changes: 2 additions & 2 deletions src/bootstraps/neptune/graphistry.sh
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,11 @@ echo "BASE_PATH=private/dash/" | sudo tee -a "${GAK_PRIVATE}/src/docker/.env"
echo '----- Launching graph-app-kit as streamlit-pub/priv:8501'
( \
cd "${GAK_PUBLIC}/src/docker" \
&& sudo docker-compose -p pub run -d --name streamlit-pub streamlit \
&& sudo docker compose -p pub run -d --name streamlit-pub streamlit \
)
( \
cd "${GAK_PRIVATE}/src/docker" \
&& sudo docker-compose -p priv run -d --name streamlit-priv streamlit \
&& sudo docker compose -p priv run -d --name streamlit-priv streamlit \
)

./hello-end.sh "$SCRIPT"
2 changes: 1 addition & 1 deletion src/bootstraps/neptune/minimal.sh
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ echo '===== Configuring graph-app-kit with Graphistry account and Neptune ====='
echo '----- Launching graph-app-kit as streamlit-pub:8501'
( \
cd "${GAK_PUBLIC}/src/docker" \
&& sudo /usr/local/bin/docker-compose up -d streamlit \
&& sudo /usr/local/bin/docker compose up -d streamlit \
)

./hello-end.sh "$SCRIPT"
2 changes: 1 addition & 1 deletion src/bootstraps/scripts/docker-aws.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@ sudo chkconfig docker on

sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose version
docker compose-version

./hello-end.sh "$SCRIPT"
2 changes: 1 addition & 1 deletion src/bootstraps/scripts/docker-container-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -ex

SCRIPT="Build graph-app-kit docker"
DC_ALIAS=${DC_ALIAS:-docker-compose}
DC_ALIAS=${DC_ALIAS:-docker compose}
./hello-start.sh "$SCRIPT"

( cd ../../docker && $DC_ALIAS build )
Expand Down
4 changes: 2 additions & 2 deletions src/bootstraps/scripts/graphistry-service-account.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ SCRIPT="Get AWS Instance ID"
cd "${GRAPHISTRY_HOME}" \
&& ( \
until ( curl -fsS http://localhost/streamgl-gpu/secondary/gpu/health > /dev/null ); \
do ( docker-compose ps && sleep 1 ); \
do ( docker compose ps && sleep 1 ); \
done \
)
)
Expand All @@ -24,7 +24,7 @@ POST_SCRIPT="CELERY_BROKER_URL=zz python manage.py shell && echo done || { echo

( \
cd "${GRAPHISTRY_HOME}" \
&& docker-compose exec -T nexus \
&& docker compose exec -T nexus \
bash -c \
"source activate rapids && echo \"${ADD_USER_SCRIPT}; ${VERIFY_USER_SCRIPT}\" | ${POST_SCRIPT}" \
)
Expand Down
6 changes: 3 additions & 3 deletions src/bootstraps/scripts/graphistry-wait-healthy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,20 @@ SCRIPT="Wait Graphistry docker containers healthy"

( \
cd "${GRAPHISTRY_HOME}" \
&& for i in `sudo docker-compose ps --services`; do ( \
&& for i in `sudo docker compose ps --services`; do ( \
( \
until ( \
[[ '"healthy"' == $(sudo docker inspect "graphistry_${i}_1" --format "{{json .State.Health.Status}}") ]] \
); do ( \
echo "waiting on $i (5s)" \
&& sudo docker-compose ps \
&& sudo docker compose ps \
&& sleep 5 \
); done \
) && echo "healthy $i" \
); done \
)

echo "--- Graphistry status after healthy waiting ---"
( cd "${GRAPHISTRY_HOME}" && sudo docker-compose ps )
( cd "${GRAPHISTRY_HOME}" && sudo docker compose ps )

./hello-end.sh "$SCRIPT"
10 changes: 5 additions & 5 deletions src/bootstraps/scripts/swap-caddy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
set -ex

## Graphistry now uses Caddy2, and until Caddy 2 supports simple auth, g-a-k still sticks with Caddy1
## ... So we stop Graphistry's Caddy 2 and put in a docker-compose of its old
## ... So we stop Graphistry's Caddy 2 and put in a docker compose of its old

CADDY_FILENAME=${CADDY_FILENAME:-full.Caddyfile}
CADDY_COMPOSE_FILENAME=${CADDY_COMPOSE_FILENAME:-docker-compose.gak.graphistry.yml}
Expand All @@ -16,10 +16,10 @@ sudo cp "../../caddy/${CADDY_COMPOSE_FILENAME}" "${GRAPHISTRY_HOME}/${CADDY_COMP
sudo cp "../../caddy/${CADDY_FILENAME}" "${GRAPHISTRY_HOME}/data/config/Caddyfile"
( \
cd "${GRAPHISTRY_HOME}" \
&& sudo docker-compose stop caddy \
&& sudo docker-compose -f "${CADDY_COMPOSE_FILENAME}" up -d caddy \
&& sudo docker-compose ps \
&& sudo docker-compose -f "${CADDY_COMPOSE_FILENAME}" ps \
&& sudo docker compose stop caddy \
&& sudo docker compose -f "${CADDY_COMPOSE_FILENAME}" up -d caddy \
&& sudo docker compose ps \
&& sudo docker compose -f "${CADDY_COMPOSE_FILENAME}" ps \
)

./hello-end.sh "$SCRIPT"
2 changes: 1 addition & 1 deletion src/caddy/Caddyfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
### #
### -- Relaunch: #
### #
### docker-compose stop caddy && docker-compose up -d #
### docker compose stop caddy && docker compose up -d #
### #
##########################################################################################

Expand Down
2 changes: 1 addition & 1 deletion src/caddy/docker-compose.gak.graphistry.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#Reuse graphistry's docker-compose setup.. except downgrade to caddy1 for auth support
#Reuse graphistry's docker compose setup.. except downgrade to caddy1 for auth support
#Start _after_ graphistry

version: '3.5'
Expand Down
2 changes: 1 addition & 1 deletion src/docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ networks:
name: grph_net

#-----------------------------------------------------------
# Ex: Reuse network from another docker-compose
# Ex: Reuse network from another docker compose
#-----------------------------------------------------------
#networks:
# grph_net:
Expand Down
Loading