Simplify LLM implementation by consolidating LLM and BaseLLM classes

[devin-ai-integration]

Simplify LLM Implementation by Consolidating Classes

This PR addresses issue #2276 by implementing custom LLM support in CrewAI and simplifying the LLM implementation by consolidating the LLM and BaseLLM classes as suggested by @joaomdmoura.

Changes

1. Converted LLM to be the abstract base class that defines the interface for all LLM implementations
2. Created a new DefaultLLM class that inherits from LLM and implements the litellm-based functionality
3. Added backward compatibility through __new__ method in LLM to handle direct instantiation
4. Updated the create_llm utility function to work with the new class structure
5. Added comprehensive documentation in docs/custom_llm.md
6. Created extensive test coverage in tests/custom_llm_test.py

Key Features

• Simplified Class Structure: Single LLM abstract base class with DefaultLLM implementation
• Backward Compatibility: Existing code that instantiates LLM directly will continue to work
• Custom LLM Support: Users can now extend the LLM class to create custom implementations
• JWT Authentication: Added support for JWT-based authentication as requested in issue [FEATURE] Provide Custom LLM Support #2276
• Error Handling: Improved error handling with retry logic and timeout handling
• Documentation: Comprehensive documentation with examples and best practices

Testing

• Added tests for custom LLM implementations
• Added tests for JWT authentication
• Added tests for error handling and timeout scenarios
• Verified backward compatibility with existing code

Link to Devin run: https://app.devin.ai/sessions/c45e76c4de5a45d2af4c486bed8044f1
Requested by: user

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[joaomdmoura]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment: Custom LLM Implementation Support

Overview

The proposed changes in this pull request introduce a robust architecture for supporting custom Large Language Models (LLMs) within CrewAI, utilizing an abstract base class (BaseLLM). The implementation enhances modularity and offers a structured approach for extending LLM functionalities.

---

Key Observations and Recommendations

1. Architecture Design

• Positive Aspects: The introduction of the BaseLLM class is commendable, establishing a consistent interface for LLM implementations. The inheritance structure enhances readability and maintainability.
• Improvement Suggestion: Ensure thorough documentation of the inheritance and usage patterns for future reference.

2. Documentation Quality

• Current State: The docs/custom_llm.md file provides a solid foundation for both usage and implementation of custom LLMs.
• Recommendation: Please add a section on Error Handling Best Practices. For instance:

  ## Error Handling Best Practices
  Always validate input parameters to prevent runtime errors and ensure graceful failure modes. 
  Consider including examples such as handling connection and timeout issues during API calls.

3. Type Annotations

• Identified Issue: Inconsistent type annotations were found in crew.py:

  manager_llm: Optional[Any] = Field(description="Language model that will run the agent.", default=None)
• Proposed Change: Improve type consistency:

  manager_llm: Optional[Union[str, InstanceOf[BaseLLM], Any]] = Field(description="Language model that will run the agent.", default=None)

4. Error Handling Improvements

• Current Practices: The error handling in the custom LLM implementation needs enhancement. As an example:

  class CustomLLM(BaseLLM):
      def __init__(self, api_key: str, endpoint: str):
          super().__init__()
          if not api_key:
              raise ValueError("API key must not be empty")
          # Additional validations for endpoint can be implemented

5. Testing Coverage

• Current Status: While testing coverage is decent, it can be expanded to cover more edge cases.
• Recommended Test Cases: Implement tests focusing on error handling for invalid inputs and timeouts handling in the CustomLLM class:

  def test_custom_llm_error_handling():
      llm = CustomLLM(api_key="invalid", endpoint="https://api.example.com")
      
      with pytest.raises(ValueError, match="Invalid API key"):
          CustomLLM(api_key="", endpoint="https://api.example.com")
      
      with pytest.raises(TimeoutError):
          llm.call("Test message", timeout=0.1)

6. Integration Points

• Integration with CrewAI: The integration of the LLM component within existing agents seems to be well-handled. To improve robustness, encapsulate the LLM creation in a try-except block to handle initialization errors gracefully:

  try:
      self.llm = create_llm(self.llm)
  except Exception as e:
      raise RuntimeError(f"Failed to initialize LLM: {str(e)}")

7. Security and Performance Considerations

• Security Improvements: Although JWT handling is well thought through, consider implementing token expiration checks to enhance security integrity.

  def _validate_token(self):
      if 'exp' in decoded and decoded['exp'] < time.time():
          raise ValueError("JWT token has expired")

• Performance Measurement: Implementing rate-limiting mechanisms will help mitigate the risk of API throttling and ensure efficient resource management:

  class RateLimitedLLM(BaseLLM):
      def _check_rate_limit(self):
          # Logic to manage request times should be added here

Conclusion

Overall, the implementation demonstrates a profound understanding of abstractions and modular design principles, enhancing the extensibility of the CrewAI framework. The highlighted areas for improvement primarily focus on strengthening type annotations, enhancing error handling, expanding testing coverage, and ensuring dynamic security measures. Addressing these recommendations will further solidify the robustness and resilience of the custom LLM implementations.

Let's ensure we reflect on the earlier PRs as this can provide valuable insights on avoiding any pitfalls already discussed in previous reviews.

Looking forward to the revisions!

[devin-ai-integration]

Closing due to inactivity for more than 7 days.