Fix token tracking race condition in threading-based async execution

[devin-ai-integration]

Summary

This PR fixes the race condition described in issue #4168 where token tracking was inaccurate when multiple async tasks with async_execution=True from the same agent ran concurrently.

The root cause was that tokens_before was captured when tasks were queued (in the main thread), but tokens_after was captured sequentially in _process_async_tasks after calling future.result(). This caused later tasks to be credited with tokens from earlier tasks that ran in parallel.

The fix introduces:

• Per-agent locks to serialize async task execution for the same agent, ensuring token deltas can be accurately attributed
• Token capture callback that captures both tokens_before and tokens_after inside the worker thread (after acquiring the lock), not when the task is queued
• New data models (AgentTokenMetrics, TaskTokenMetrics, WorkflowTokenMetrics) for detailed per-task token tracking
• Updated TaskOutput to include per-task usage_metrics

Updates since last revision

• Fixed lint errors across multiple files:
  • B023 (late-binding closure issues) in _aexecute_tasks using default arguments
  • W293 (whitespace in blank lines) in docstrings across crew.py, task.py, and usage_metrics.py
  • B007/PERF102 in calculate_usage_metrics by using .values() instead of .items()

Review & Testing Checklist for Human

• Verify backward compatibility: The execute_async method signature changed to accept optional token_capture_callback and agent_execution_lock parameters. The return type also changed to potentially return a tuple. Verify this doesn't break existing code that calls execute_async directly.
• Review the late-binding closure pattern in _execute_tasks (line ~1194): def create_token_callback(agent: Any = exec_data.agent) - ensure the default argument correctly captures the agent value in the loop
• Review the async wrapper pattern in _aexecute_tasks (line ~964): async def _wrapped_task_execution(_task=task, _agent=agent, ...) - verify default args correctly bind loop variables
• Test with real LLM calls: Run a crew with multiple async tasks from the same agent and verify token metrics are accurate (not duplicated across tasks)
• Check base branch: This PR is based on feat/per-user-token-tracing from PR Feature: CrewAI Token Tracking Enhancement #4132. Verify this is the intended target branch.

Recommended test plan:

1. Create a crew with 2+ async tasks assigned to the same agent
2. Run crew.kickoff() and inspect result.tasks_output[i].usage_metrics for each task
3. Verify each task shows distinct token counts (not cumulative/duplicated values)

Notes

• The per-agent lock intentionally reduces parallelism for tasks from the same agent to ensure correctness. Tasks from different agents still run concurrently.
• There's a known limitation in calculate_usage_metrics where agents with the same role cannot be distinguished (see TODO comment)
• One test job (tests 3.13) was cancelled in CI but all other checks passed

Link to Devin run: https://app.devin.ai/sessions/80968e6ecc774e45ad35f833cf8d2ea0
Requested by: João (joao@crewai.com)

[cursor]

You have run out of free Bugbot PR reviews for this billing cycle. This will reset on January 28.

To receive reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring