Skip to content

Conversation

@trebedea
Copy link
Member

Description

Add support for reasoning models that output reasoning traces in the response (e.g. DeepSeek-R1).
Current implementation strips out the reasoning traces in the output parser for each executed Guardrails task, but different output parsers can be added to use the information in the reasoning trace as well (e.g. implement a custom rail that analyzes the reasoning traces).

Sample model config for DeepSeek-R1 is:

models:
  - type: main
    engine: deepseek
    model: deepseek-reasoner
    reasoning_config:
      remove_thinking_traces: True
      start_token: "<think>"
      end_token: "</think>"

Using other platforms for serving the model, such as NVIDIA AI Endpoints, is supported as long as the model name contains deepseek in the name.

MR includes:

  • Documentation updates
  • DeepSeek-R1 prompts and sample config
  • Some refactoring in llmrails and self-check rails related to updates for using reasoning models
  • Support for deepseek and google-genai as LLM providers

Related Issue(s)

Checklist

  • I've read the CONTRIBUTING guidelines.
  • I've updated the documentation if applicable.
  • I've added tests if applicable.
  • @mentions of the person or team responsible for reviewing proposed changes.

@trebedea trebedea added documentation Improvements or additions to documentation enhancement New feature or request labels Feb 14, 2025
@trebedea trebedea requested a review from Pouyanpi February 14, 2025 12:34
@trebedea trebedea self-assigned this Feb 14, 2025
@github-actions
Copy link
Contributor

Documentation preview

https://nvidia.github.io/NeMo-Guardrails/review/pr-996

@trebedea
Copy link
Member Author

@mikemckiernan , if you have any feedback about additional documentation changes (including examples).
Should we mention anywhere in the docs that reasoning models induce a serious latency overhead?

@trebedea trebedea requested a review from cparisien February 14, 2025 12:36
@Pouyanpi Pouyanpi added this to the v0.13.0 milestone Feb 24, 2025
@Pouyanpi Pouyanpi requested a review from tgasser-nv March 3, 2025 10:22
Copy link
Collaborator

@cparisien cparisien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, though I'd appreciate a second reviewer as well. I expect we won't have big issues since you've done plenty of "adapt to this LLM" before.



@pytest.mark.asyncio
async def test_1(rails_config):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
async def test_1(rails_config):
async def test_reasoning_traces_removed_from_responses(rails_config):

Copy link
Collaborator

@Pouyanpi Pouyanpi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you Traian!

@Pouyanpi Pouyanpi merged commit 4344daf into develop Mar 5, 2025
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants