This is the official repository for NESTFUL.
- Paper Title: NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
- Link: https://arxiv.org/abs/2409.03797v2
- HuggingFace Data Link: https://huggingface.co/datasets/ibm-research/nestful
We have shared the latest NESTFUL evaluation set under data_v2 dir.
nestful_data.jsonl: It has 1861 evaluation data for nested sequencing.executable_functions: Contains the implementation of all the functions in the benchmark.
The data_v1 directory includes the data for the previous version of the paper - link.
executable: contains data and spec with necessary information to execute them through RapidAPI.non-executable: contains the nested sequencing data from SGD and GLAIVE that are hand-picked by human annotators from data synthetically generated using an LLM.