🛠️ Tool Intelligence 🤖️ Reading-list

Must-read papers about Tool Intelligence.

The ability to create and utilize tools is a manifestation of human intelligence. Humanity once used stones to drive away natural predators and employed fire to retain light and warmth through the night for survival. Later on, humans used airplanes, submarines, and spaceships to navigate the skies, the seas, and expand their knowledge.

The same principle applies to artificial intelligence entities like large language models (LLMs). In recent years, numerous studies have explored how to enable LLMs to recognize, utilize, and even create external tools to enhance their capability to complete real-life tasks assigned by humans. We name this direction Tool Intelligence, which serves as a key component of LLM Agents. We also believe that Tool Intelligence is a convenient solution for LLM 2B applications.

There are some similar previous collections summarizing some of these works, but not in the same view and not being updated.

So we create this repo with a list of resources related to Tool Intelligence, aiming to help the related researchers have a better view of existing works, thus facilitating the development of this area.

📪 Contact

There may exsist cases where we miss important works in this field or have some inaccurate information noted, please feel free to tell us!

Appreciation for the efforts in advance.

Content

• Tool Intelligence Survey
• Tool Intelligence Benchmarks
• Tool Intelligence Methods
  • Tool-using
  • Tool-creating
  • to be discovered...

🗒️ Tool Intelligence Survey

Apr 2023

• Tool Learning with Foundation Models

Feb 2023

• Augmented Language Models: a Survey

📊 Tool Intelligence Benchmarks

May 2024

• SoAyBench

Mar 2024

• StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models

Jan 2024

• ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios

Nov 2023

• ToolTalk: Evaluating Tool-Usage in a Conversational Setting

Oct 2023

• MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use

Jun 2023

• 【ACL'24】ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases

May 2023

• 【NIPS'23 workshop】On the Tool Manipulation Capability of Open-source Large Language Models
• Gorilla: Large Language Model Connected with Massive APIs

Apr 2023

• 【EMNLP'23】API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs

Jan 2023

• 【NIPS'23 Poster】ToolQA: A Dataset for LLM Question Answering with External Tools
• 【ICLR'24 spotlight】ToolBench

Dec 2021

• 【not for LLM, for Software Engineering】Revisiting, Benchmarking and Exploring API Recommendation: How Far Are We?

🔨 Tool-using Methods

May 2024

• A Solution-based LLM API-using Methodology for Academic Information Seeking

Feb 2024

• AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls

Jan 2024

• APIGen: Generative API Method Recommendation

Aug 2023

• Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models
• TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage

Jun 2023

• RestGPT: Connecting Large Language Models with Real-World RESTful APIs
• ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
• GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution

May 2023

• Gorilla: Large Language Model Connected with Massive APIs
• On the Tool Manipulation Capability of Open-source Large Language Models
• ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings

Apr 2023

• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

Mar 2023

• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

Feb 2023

• Toolformer: Language Models Can Teach Themselves to Use Tools
• Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents

Jan 2023

• 【ICLR'24 spotlight】ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
• PAL: Program-aided Language Models

Nov 2022

• Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

Sep 2022

• Code as Policies: Language Model Programs for Embodied Control

Dec 2021

• WebGPT: Browser-assisted question-answering with human feedback

💡 Tool-creating Methods

Oct 2023

• 【EMNLP'23 findings】CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
• Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model

Sep 2023

• 【ICLR'24 poster】CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets

May 2023

• 【ICLR'24 poster】Large Language Models as Tool Makers