COMPASS (COMprehensive Process AnalySiS) is a methodology for optimizing prompts for Large Language Model (LLM) based agents through process mining techniques. COMPASS systematically applies process mining techniques to discover, analyze, and guide agent behavior. This repository contains the artifacts from our case study applying COMPASS to a natural language to SPARQL query transformation agent.
Our case study applies COMPASS to Graf-von-Data (GvD), an agent that transforms natural language questions into SPARQL queries by exploring knowledge graphs. GvD implements the ReAct framework and utilizes three specialized tools:
- Search: Identifies entity URIs based on natural language keywords
- Describe: Extracts incoming and outgoing triples from specified entity URIs
- Query: Executes generated SPARQL queries on the knowledge graph and returns result sets
The agent is deployed on a semiconductor industry supply chain knowledge graph that models semiconductor organizations, their sites, and supply relations. The dataset contains 33,755 triples and represents realistic complexity typical of industrial knowledge graphs.
The knowledge graph and evaluation queries are available in a separate repository under version v0.1:
- SupplyBench Repository
- The evaluation queries are all queries from the SupplyBench Evaluation that are tagged as "one triple pattern", "two triple patterns", "three triple patterns", or "four triple patterns".
data/: Contains trajectories, event logs, and evaluation resultstrajectories/: Raw agent trajectories across optimization iterationsevent-logs/: Processed event logs for process mining analysis- Evaluation result files comparing different prompt versions
pm4py-data-analysis/: Analysis notebooks and scripts using PM4Pypmtk-data-analysis/: Analysis results from the paper using PMTKprompts/: Different versions of the agent prompts developed through COMPASS