fix(security)!: drop support for templating in conformity rules#2382
fix(security)!: drop support for templating in conformity rules#2382kevinmessiaen merged 3 commits intomainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request simplifies the Conformity check by removing Jinja2 templating from the rule attribute and passing the full Trace object directly to the LLM prompt. It also updates dependencies and refines the evaluation prompt to focus on outputs and metadata. Feedback includes correcting a non-existent model name in documentation examples, using generic types for consistency in method signatures, and fixing inconsistent tag spacing in the prompt template.
| ... rule="The response should contain the keywords '{{ trace.last.inputs.keywords }}' and be polite.", | ||
| ... generator=Generator(model="openai/gpt-4o") | ||
| ... rule="The last response should be polite.", | ||
| ... generator=Generator(model="openai/gpt-5-mini") |
There was a problem hiding this comment.
The example uses gpt-5-mini, which is not a currently available model from OpenAI. It is better to use a real model like gpt-4o-mini to avoid confusion for users following the documentation.
| ... generator=Generator(model="openai/gpt-5-mini") | |
| ... generator=Generator(model="openai/gpt-4o-mini") |
|
|
||
| @override | ||
| async def get_inputs(self, trace: Trace[InputType, OutputType]) -> dict[str, str]: | ||
| async def get_inputs(self, trace: Trace[InputType, OutputType]) -> dict[str, Any]: |
There was a problem hiding this comment.
For consistency with the class definition and the base class BaseLLMCheck, the get_inputs method should use the generic TraceType instead of the concrete Trace[InputType, OutputType].
| async def get_inputs(self, trace: Trace[InputType, OutputType]) -> dict[str, Any]: | |
| async def get_inputs(self, trace: TraceType) -> dict[str, Any]: |
| < TRACE > | ||
| {{ trace }} | ||
| </ TRACE > |
There was a problem hiding this comment.
The markers in the prompt instructions (line 41) are defined as <TRACE>...</TRACE>, but the actual tags used in the template have extra spaces: < TRACE > and </ TRACE >. Removing these spaces ensures consistency and adheres to standard XML/HTML-like tagging that LLMs are typically trained on.
<TRACE>
{{ trace }}
</TRACE>
Description
This PR updates the conformity check in two ways: