poe-stash-indexer

This project aims to build tooling to gather data from Path of Exile's Public Stash Tab API (see also the older community wiki documentation).

The main component in this project is the indexer application. It offers an easy way to consume the Public Stash Tab API river and flush it data sinks like RabbitMQ or S3 for further processing. More on the installation & usage in the dedicated documentation.

Project Structure

├── Makefile
├── README.md       # you are here
├── crates
│   ├── indexer     # the main application
│   ├── stash-api   # an internal library that `indexer` uses
│                   # below are some internal prototypes, you can ignore for now
│   ├── stash-differ
│   ├── trade-api
│   ├── trade-common
├── notes           # some notes for myself
├── infra           # internal scripts, CI parts and documentation for my own `indexer` deployment
└── shell.nix

Local Dev Environment

There is infra/docker-compose.yaml defines a setup of indexer and some of the other prototypes. Feel free to use or copy any of it.

• indexer - setup of the indexer service to start fetching and feed the data stream into a RabbitMQ instance
• rabbitmq - a RabbitMQ instance for indexer to ingest new Stash batches into to test the RabbmitMQ sink
• setup of the trade-ingest & trade-api services to consume above stream and expose it via its REST-like API, respectively
• trade-store - a PostgreSQL instance for trade-ingest to ingest into and trade-api to read data from
• otel-collector - an OTLP setup (integrating with New Relic) to investigate metrics of the indexer, trade-ingest, trade-api
• reverse-proxy - exposes a reverse proxy setup via nginx to easily access all services (see below)

Here is a list of services in this local development setup and and their credentials (username:password):

• Trade API (public)
• RabbitMQ Control Panel (Basic Auth: poe:poe)
• Jaeger Dashboard (public)

For testing the PostgreSQL indexer sink, you can run something like:

docker run --name pg-poe -e POSTGRES_PASSWORD=poe -e POSTGRES_USER=poe -e POSTGRES_DB=poe -p 5432:5432 postgres:latest

and connect to pg://poe:poe@127.0.0.1:5432/poe.