Skip to content

feature - policy tools #471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
drorIvry merged 2 commits into from
Feb 10, 2026
Merged

feature - policy tools #471

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
baf7dcb
policy tools
drorIvry Feb 9, 2026
9abe480
hotfix - polic tools
drorIvry Feb 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "qualifire"
version = "0.16.0"
version = "0.17.0"
description = "Qualifire Python SDK"
authors = [{ name = "qualifire-dev", email = "dror@qualifire.ai" }]
requires-python = ">=3.8,<4"
Expand Down Expand Up @@ -97,7 +97,7 @@ color_output = true

[tool.mypy]
# https://mypy.readthedocs.io/en/latest/config_file.html#using-a-pyproject-toml-file
python_version = 3.8
python_version = "3.8"
pretty = true
show_traceback = true
color_output = true
Expand Down
3 changes: 3 additions & 0 deletions qualifire/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ def evaluate(
hallucinations_mode: ModelMode = ModelMode.BALANCED,
grounding_multi_turn_mode: bool = False,
policy_multi_turn_mode: bool = False,
policy_include_tools: bool = False,
policy_target: PolicyTarget = PolicyTarget.BOTH,
topic_scoping_mode: Optional[ModelMode] = None,
topic_scoping_multi_turn_mode: bool = False,
Expand Down Expand Up @@ -109,6 +110,7 @@ def evaluate(
:param hallucinations_mode: Model mode for hallucinations check (speed/balanced/quality).
:param grounding_multi_turn_mode: Enable multi-turn mode for grounding check.
:param policy_multi_turn_mode: Enable multi-turn mode for policy check.
:param policy_include_tools: Include tool definitions and tool calls in policy assertion context.
:param policy_target: Target for policy checks (input/output/both).
:param topic_scoping_mode: Model mode for topic scoping check (speed/balanced/quality).
:param topic_scoping_multi_turn_mode: Enable multi-turn mode for topic scoping check.
Expand Down Expand Up @@ -233,6 +235,7 @@ def evaluate(
hallucinations_mode=hallucinations_mode,
grounding_multi_turn_mode=grounding_multi_turn_mode,
policy_multi_turn_mode=policy_multi_turn_mode,
policy_include_tools=policy_include_tools,
policy_target=policy_target,
topic_scoping_mode=topic_scoping_mode,
topic_scoping_multi_turn_mode=topic_scoping_multi_turn_mode,
Expand Down
1 change: 1 addition & 0 deletions qualifire/types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ class EvaluationRequest(BaseModel):
hallucinations_mode: ModelMode = ModelMode.BALANCED
grounding_multi_turn_mode: bool = False
policy_multi_turn_mode: bool = False
policy_include_tools: bool = False
policy_target: PolicyTarget = PolicyTarget.BOTH
topic_scoping_mode: Optional[ModelMode] = None
topic_scoping_multi_turn_mode: bool = False
Expand Down
8 changes: 8 additions & 0 deletions tests/test_types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,3 +111,11 @@ def test_validate_tsq_requirements(
available_tools=available_tools,
tool_use_quality_check=tsq_check,
)

def test_policy_include_tools_defaults_false(self):
req = EvaluationRequest(input="test")
assert req.policy_include_tools is False

def test_policy_include_tools_can_be_set(self):
req = EvaluationRequest(input="test", policy_include_tools=True)
assert req.policy_include_tools is True
Comment on lines +115 to +121
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Parametrize the new policy_include_tools tests.

The two standalone tests can be combined into a single parametrized case to follow test guidelines and reduce duplication.

♻️ Suggested refactor 
-    def test_policy_include_tools_defaults_false(self):
-        req = EvaluationRequest(input="test")
-        assert req.policy_include_tools is False
-
-    def test_policy_include_tools_can_be_set(self):
-        req = EvaluationRequest(input="test", policy_include_tools=True)
-        assert req.policy_include_tools is True
+    `@pytest.mark.parametrize`(
+        "kwargs,expected",
+        [
+            ({}, False),
+            ({"policy_include_tools": True}, True),
+        ],
+    )
+    def test_policy_include_tools(self, kwargs, expected):
+        req = EvaluationRequest(input="test", **kwargs)
+        assert req.policy_include_tools is expected

As per coding guidelines, "tests/**/*.py: Use parametrized pytest test cases in the tests/ directory for comprehensive test coverage".

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def test_policy_include_tools_defaults_false(self):
req = EvaluationRequest(input="test")
assert req.policy_include_tools is False
def test_policy_include_tools_can_be_set(self):
req = EvaluationRequest(input="test", policy_include_tools=True)
assert req.policy_include_tools is True
`@pytest.mark.parametrize`(
"kwargs,expected",
[
({}, False),
({"policy_include_tools": True}, True),
],
)
def test_policy_include_tools(self, kwargs, expected):
req = EvaluationRequest(input="test", **kwargs)
assert req.policy_include_tools is expected
🤖 Prompt for AI Agents 
In `@tests/test_types.py` around lines 115 - 121, Combine the two tests into one
parametrized pytest case: create a single test (e.g., test_policy_include_tools)
decorated with `@pytest.mark.parametrize`("value,expected", [(None, False), (True,
True)]) or with explicit values [False, True] and instantiate EvaluationRequest
accordingly (for None use default ctor) and assert req.policy_include_tools ==
expected; ensure pytest is imported and reference EvaluationRequest in the test
body to replace the two existing functions
test_policy_include_tools_defaults_false and
test_policy_include_tools_can_be_set.