Follow the instructions here to install docker.
docker build . -t tvp
Note: You might need to bump RAM settings as the default value of 2GB might not be enough.
- The CLI command
all, will make sure to fetch all the data, aggregate it, load it to Dune tables and also give insights on Top analysis.docker run --mount type=bind,source="$(pwd)"/data,target=/app/data tvp all <api-key>
To build the Docker container:
docker build . -t tvp
For simplicity, there is a CLI to run the application code.
Use the command below to start the container in interactive mode. This is ideal for running multiple CLI commands, and it ensures that any files created within the container are saved to your local data directory.
docker run -it --entrypoint bash --mount type=bind,source="$(pwd)"/data,target=/app/data tvp
Available commands:
python tvp/cli.py --help
all: Runs all the commands.python tvp/cli.py all <api-key>extract: Extracts raw data from the Dune API for a specific query and time range. By default, runs for a previous day.python tvp/cli.py extract <api-key> <query-id>backfill: Backfills data using Dune API of a specific query execution.python tvp/cli.py backfill <api-key> <query-execution-id>aggregate: Aggregates raw TVP data based on the provided key and writes it to a .parquet file.python tvp/cli.py aggregate <key>top: Displays the top entries based on the specified aggregation key and value. It also loads the results to a Dune table.python tvp/cli.py top <key> <aggregated-value>load: Loads aggregation data to a table using Dune API.python tvp/cli.py load <api_key> <key>
pytest
Use these commands to inspect Parquet files:
- Show the first 10 rows of the Parquet file
parq <file-path> --head 10
- Show the row count of the Parquet file
parq <file-path> -c