Skip to content

Gemini CLI should have an integration test that fails on previously unseen contradictions in the prompt #17809

@gundermanc

Description

@gundermanc

#17263 (comment) demonstrates using the LLM to review the prompt for contradictory language.

Contradictions are notable because they lead to inconsistent or not working prompted behavior an are often a sign of deficiencies in the prompt.

This issue tracks adding a simple integration test that reads the prompt, enumerates all ambiguities or contradictions, and checks an allow list for them. Any that aren't present in the allow list, as judged by the LLM, should cause the agent to return a failure message and fail the test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/platformIssues related to Build infra, Release mgmt, Testing, Eval infra, Capacity, Quota mgmtstatus/need-triageIssues that need to be triaged by the triage automation.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions