Skip to content

Commit 9f9ed89

Browse files
chuenchen309claude
andauthored
fix(scan): GOAT/Crescendo generators crash on empty languages list (#2602)
Both GOATAttackScenarioGenerator._select_objectives and CrescendoAttackScenarioGenerator._select_objectives pick a random language via `languages[int(rng.integers(len(languages)))]` with no guard for an empty `languages` list. vulnerability_scan() accepts languages: list[str] with no validation, so an empty list raises `ValueError: high <= 0` from numpy instead of producing scenarios. The sibling KnowledgeBaseScenarioGenerator._sample_languages already handles this exact case correctly: `if not languages: return ["en"] * scenario_count`. GOAT/Crescendo look like copy-pasted siblings (same objectives-dict structure, same _select_objectives body) that never received this guard. Fix: `languages = languages or ["en"]` in both generators, mirroring the existing correct fallback. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 1213b7e commit 9f9ed89

4 files changed

Lines changed: 28 additions & 0 deletions

File tree

libs/giskard-scan/src/giskard/scan/generators/crescendo.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ def _select_objectives(
8888
assignments = list(DEFAULT_CRESCENDO_OBJECTIVES.items())
8989
selected_assignments: list[tuple[str, str]]
9090
rng = rng or np.random.default_rng()
91+
languages = languages or ["en"]
9192

9293
if max_scenarios is None or max_scenarios >= len(assignments):
9394
selected_assignments = assignments

libs/giskard-scan/src/giskard/scan/generators/goat.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,7 @@ def _select_objectives(
177177
assignments = list(DEFAULT_GOAT_OBJECTIVES.items())
178178
selected_assignments: list[tuple[str, str]]
179179
rng = rng or np.random.default_rng()
180+
languages = languages or ["en"]
180181

181182
if max_scenarios is None or max_scenarios >= len(assignments):
182183
selected_assignments = assignments

libs/giskard-scan/tests/generators/test_crescendo.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,3 +93,16 @@ async def test_crescendo_multiturn_still_returns_scenarios():
9393
target_mode="multiturn",
9494
)
9595
assert len(scenarios) == len(DEFAULT_CRESCENDO_OBJECTIVES)
96+
97+
98+
async def test_crescendo_generator_falls_back_to_english_with_empty_languages():
99+
"""An empty `languages` list should fall back to English, mirroring
100+
KnowledgeBaseScenarioGenerator._sample_languages, instead of crashing in
101+
`rng.integers(0)`."""
102+
gen = CrescendoAttackScenarioGenerator()
103+
scenarios = await gen.generate_scenario(
104+
ScenarioContext(description="A safety chatbot", languages=[])
105+
)
106+
107+
assert len(scenarios) == len(DEFAULT_CRESCENDO_OBJECTIVES)
108+
assert all(scenario.annotations["language"] == "en" for scenario in scenarios)

libs/giskard-scan/tests/generators/test_goat.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,3 +92,16 @@ async def test_goat_multiturn_still_returns_scenarios():
9292
target_mode="multiturn",
9393
)
9494
assert len(scenarios) == len(DEFAULT_GOAT_OBJECTIVES)
95+
96+
97+
async def test_goat_generator_falls_back_to_english_with_empty_languages():
98+
"""An empty `languages` list should fall back to English, mirroring
99+
KnowledgeBaseScenarioGenerator._sample_languages, instead of crashing in
100+
`rng.integers(0)`."""
101+
gen = GOATAttackScenarioGenerator()
102+
scenarios = await gen.generate_scenario(
103+
ScenarioContext(description="A safety chatbot", languages=[])
104+
)
105+
106+
assert len(scenarios) == len(DEFAULT_GOAT_OBJECTIVES)
107+
assert all(scenario.annotations["language"] == "en" for scenario in scenarios)

0 commit comments

Comments
 (0)