Skip to content

Auto Mode Tool Classification

Jump to bottom
Joe Curlee (w4ffl35) edited this page Nov 8, 2025 · 1 revision

Auto Mode Intelligent Tool Classification

Overview

Auto mode now uses an intelligent classifier to select only the relevant tool categories based on the user's prompt, instead of loading all 84 tools upfront. This dramatically improves performance for simple conversations while maintaining full tool access when needed.

How It Works

Architecture

When a user selects "Auto" mode in the chat dropdown:

  1. User enters a prompt → The prompt is sent with tool_categories=None (Auto mode indicator)
  2. Classifier analyzes the prompt → _classify_prompt_for_tools() uses keyword matching to detect intent
  3. Relevant tools are loaded → Only tools from selected categories are bound to the model
  4. Request proceeds → The LLM generates a response with access to only the necessary tools

Classification Logic

The classifier uses fast keyword-based pattern matching (no LLM call) to detect:

Web/Scraping Tools (search category) - Triggered by:

  • scrape, fetch, download, crawl, extract from
  • get content, webpage, website, url, http
  • Note: Web scraping and search tools are both in the ToolCategory.SEARCH category

Math/Calculation Tools (math category) - Triggered by:

  • calculate, compute, solve, what is, how much
  • equation, math, addition, subtract, multiply, divide
  • Math operators (+, *, /, =, -) when no URL is present

File Operations Tools (file category) - Triggered by:

  • read file, write file, save to, open file
  • create file, delete file, list files, directory

Time/Date Tools (time category) - Triggered by:

  • time, date, today, tomorrow, yesterday
  • schedule, calendar, when is, what day

Search/Research Tools (search category) - Triggered by:

  • search, find, look up, research, information about
  • tell me about, what do you know, explain

Conversation Management Tools (conversation category) - Triggered by:

  • clear history, clear conversation, conversation history
  • new conversation, switch conversation, list conversations

Special Cases

  • Simple chat (e.g., "hello", "how are you?") → Returns [] (no tools needed)
  • Multiple categories (e.g., "scrape site.com and calculate the sum") → Returns ["search", "math"]
  • URLs containing operators (e.g., "fetch https://example.com") → Excludes math tools (operators are ignored when URLs detected)
  • Web scraping → Uses "search" category (both web search and scraping tools are in ToolCategory.SEARCH)

Performance Impact

Before (Old Auto Mode)

  • "hello" → 2048 input tokens (all 84 tools loaded)
  • Response time: Slow (all tools in context)
  • Token count: Excessive for simple chat

After (Intelligent Auto Mode)

  • "hello" → ~150 input tokens (0 tools loaded)
  • "scrape joecurlee.com" → ~300 input tokens (only search tools loaded - includes web scraping)
  • "what is 2+2" → ~200 input tokens (only math tools loaded)
  • Response time: Fast (minimal tool overhead)
  • Token count: Optimized for task complexity

Mode Comparison

Mode Tool Loading Use Case
Chat (default) tool_categories=[] - No tools Simple conversations, Q&A without actions
Auto tool_categories=None - Intelligent selection General use, tasks requiring unknown tools
RAG Specific RAG tools Knowledge base queries

Implementation Files

Core Changes

llm_model_manager.py (Lines 260-290, 420-555)

  • Added _classify_prompt_for_tools() method for intelligent category selection
  • Modified handle_request() to detect Auto mode and call classifier
  • Updated tool filtering logic to handle classifier output

llm_request.py (Line 333)

  • Changed APPLICATION_COMMAND to use tool_categories=None (Auto mode)

chat_prompt_widget.py (Lines 73-85)

  • Changed dropdown default from "Auto" to "Chat" for better UX

message_formatting_mixin.py (Lines 103-117)

  • Fixed to only pass tools to chat template when appropriate

Testing

Unit Tests: All 106 LLM component tests pass Integration Test: test_tool_classifier.py - 17/17 test cases pass

Usage

For Users

Simple conversations (recommended):

  1. Keep dropdown on "Chat" (default)
  2. Type "hello", "how are you?", etc.
  3. Fast responses with no tool overhead

Commands requiring tools:

  1. Switch dropdown to "Auto"
  2. Type "scrape joecurlee.com" or "what is 2+2"
  3. Classifier selects relevant tools automatically (e.g., ["search"] for scraping, ["math"] for calculations)
  4. Fast responses with only necessary tools loaded

Specific tool use:

  1. Use programmatic LLMRequest with explicit tool_categories
  2. Example: LLMRequest(tool_categories=["search", "math"])
  3. Valid categories: author, code, research, qa, chat, image, system, file, math, conversation, mood, analysis, workflow, search, rag, user

For Developers

Adding new tool categories:

  1. Register tools with the appropriate category in ToolManager
  2. Add keyword patterns to _classify_prompt_for_tools() in llm_model_manager.py
  3. Update this documentation

Debugging classification:

from airunner.components.llm.managers.llm_model_manager import LLMModelManager

manager = LLMModelManager()
categories = manager._classify_prompt_for_tools("your prompt here")
print(f"Selected categories: {categories}")

Future Enhancements

Potential improvements to the classifier:

  1. Machine Learning Classifier: Replace keyword matching with a lightweight ML model
  2. User Feedback Loop: Learn from user corrections (e.g., "I didn't need those tools")
  3. Context-Aware: Consider conversation history when selecting tools
  4. Performance Metrics: Track tool usage patterns and optimize keyword lists
  5. Custom Categories: Allow users to define their own tool categories

Troubleshooting

Problem: Auto mode loads unnecessary tools

Solution: Check keyword patterns in _classify_prompt_for_tools() - may have false positive matches

Problem: Auto mode missing required tools

Solution: Add more keyword patterns for that category or use explicit tool_categories in request

Problem: Simple chat still slow

Solution: Verify you're using "Chat" mode (default), not "Auto" mode

Related Documentation

Clone this wiki locally