Scorer composition (PR B): TrueFalse pure-composition API + CallableResponseHandler

[romanlutz]

PR B of 3 — Scorer-owned composition ("Option 5")

This is the middle layer of a 3-PR scorer-architecture refactor. It builds on PR A (#2125, merged) and is the precedent PR C follows when rolling the pattern out to the remaining LLM scorers. This is a clean v1.0 breaking change — no deprecation shims, no legacy kwargs.

What changed

1. CallableResponseHandler (concern-D escape hatch). A ResponseHandler that wraps a user-supplied Callable[[str], dict] mapping raw target text → the score dict. This is the clean replacement for the old response_parser hook and unblocks plain-text scoring targets (e.g. LlamaGuard). Its response_format is None, so plain-text targets are not forced into JSON.

2. TrueFalseQuestion value type (Pydantic). Owns category, true_description, false_description, metadata="", with from_yaml(path) and to_render_dict(). Templates and params are decoupled: a templated system-prompt SeedPrompt can be rendered with a TrueFalseQuestion loaded from its own separate YAML. TrueFalseQuestionPaths and render_true_false_system_prompt(question=...) are exported alongside it.

3. SelfAskTrueFalseScorer — pure composition constructor. No legacy kwargs, no deprecation:

def __init__(
    self,
    *,
    chat_target: PromptTarget | None = None,
    system_prompt: SeedPrompt | str | None = None,
    response_handler: ResponseHandler | None = None,
    score_category: Sequence[str] | str | None = None,
    validator: ScorerPromptValidator | None = None,
    score_aggregator: TrueFalseAggregatorFunc = TrueFalseScoreAggregator.OR,
) -> None: ...

Two construction modes work: (a) a static system prompt (str or SeedPrompt, no params — e.g. LlamaGuard), and (b) a templated SeedPrompt + TrueFalseQuestion params. The scorer stores self._response_handler (the caller's, or a default JsonSchemaResponseHandler) and reads the schema identifier off it.

4. from_question factory (sanctioned "preset drives more than the prompt" pattern). Because a TrueFalseQuestion drives both the system prompt and the category, a non-deprecated classmethod renders the prompt via render_true_false_system_prompt(question=...) and sets score_category = question.category:

@classmethod
def from_question(cls, *, chat_target, question, response_handler=None,
                  validator=None, score_aggregator=TrueFalseScoreAggregator.OR): ...

5. LlamaGuard pilot. An end-to-end unit test (mocked target) wires SelfAskTrueFalseScorer with a static LlamaGuard system prompt (str) + a CallableResponseHandler that parses LlamaGuard's safe / unsafe\nS1,S2,... output into the score dict — demonstrating the escape hatch replacing the response_parser hook.

Consumers migrated to the new API

tree_of_attacks.py, scenario.py, setup/initializers/scorers.py, and the ai_recruiter integration test now use the composition API. SelfAskQuestionAnswerScorer (subclass) is made consistent with the slim API.

Response contract (from PR A, preserved)

The round-trip helper reads the schema/format off the handler: JsonSchemaResponseHandler.response_format → "json", CallableResponseHandler.response_format → None. Existing JSON scorers are unaffected.

Verification

• tests/unit/score: 1311 passed / 16 skipped
• ruff check, ruff format --check, ty check, full pre-commit on changed files: clean

Notes for reviewers

• This branch was merged (not rebased) with latest main after PR A landed, so the diff is scoped to B's deltas only.
• The remaining LLM scorers + consumers are finalized in PR C, which stacks on this branch.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com