feat(scan): DeepTeam third-party scan integration

[kevinmessiaen]

Summary

Integrates DeepTeam as a third-party red-teaming engine for third_party_scan(target, tool="deepteam", ...), alongside the existing garak and lidar integrations.

A giskard Target (sync/async, str -> str or BaseModel -> BaseModel) is presented to DeepTeam as an async model_callback; DeepTeam's simulator/evaluator LLMs are driven through the giskard default generator wrapped as a DeepEvalBaseLLM. Results are translated back into a SuiteResult (one ScenarioResult per vulnerability/type/attack).

What's included

• Package scaffold + optional dep (giskard-scan[deepteam]), name resolver for vulnerabilities/attacks, adapter with result translation, model_callback bridge with a uuid-keyed typed-Trace cache, and typed third_party_scan overloads dispatching tool="deepteam".
• Generator wrapper (_judge_generator.py) exposing a giskard BaseGenerator as DeepEval's simulator/evaluation model.
• Schema-honoring fix (final commit): DeepEval calls the wrapper as a non-native model — (a_)generate(prompt, schema=SomePydanticModel) — and reads attributes off the returned instance (e.g. res.data on SyntheticDataList). The wrapper previously ignored the schema kwarg and returned plain text, so every scenario errored with 'str' object has no attribute 'data' (and, earlier in the chain, 'str' object has no attribute 'complete'). It now routes the schema through the generator's structured-output workflow (chat(prompt).with_output(schema).run()), giving the underlying LLM a real response_format and validating/retrying into the schema. No-schema calls are unchanged.
• uv.lock: locks the deepteam dependency tree (pyproject already declared the extra; the lockfile hadn't been regenerated).
• Regression test asserting a_generate(prompt, schema=SyntheticDataList) returns a validated instance whose .data is accessible.

Testing

• uv run pytest libs/giskard-scan/tests/integrations/deepteam/test_deepteam_judge_generator.py → 6 passed.
• Verified end-to-end against a live Azure model: a curated single-vulnerability scan (Bias / PromptInjection) over an echo target returns 12 scenarios, 0 errors, with real Bias metrics and coherent judge reasoning. The full curated multi-turn scan (57 scenarios × 2 target shapes) runs without the .data/.complete errors.

Notes

• DeepTeam tests are gated with pytest.importorskip("deepteam") and the deepteam integration dir is excluded from the default doctest run.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces third-party scanner integrations (garak, lidar, and deepteam) into the giskard-scan package, enabling external security scanners to run against Giskard targets and normalize their outputs into SuiteResult objects. The review feedback highlights critical thread-safety and event loop mismatch issues when invoking target and generator methods from DeepTeam's worker thread, recommending delegation to the main loop via asyncio.run_coroutine_threadsafe. Additionally, the feedback suggests preserving custom Trace subclasses by utilizing Trace.for_target instead of directly instantiating the base Trace class during trace reconstruction in both the DeepTeam and Lidar adapters.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Code Review

This pull request integrates the deepteam security scanner into giskard-scan by adding a dedicated adapter, a callback bridge for target interaction, and a Giskard-backed LLM wrapper for DeepTeam's simulator and evaluator. The review feedback is highly valuable and focuses on enhancing robustness and thread safety. Key recommendations include routing target interactions to the main event loop when invoked from DeepTeam's internal worker thread, defensively verifying metadata types, raising a clear error if no default generator is configured, and checking if the event loop is running before scheduling coroutines to prevent potential hangs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

When DeepTeam runs its red-teaming engine, it manages its own async event loop internally on a worker thread. Since ScanTargetCallback.__call__ is called from that worker thread, executing await trace.with_interaction(interaction) directly will run the user's target on DeepTeam's event loop. If the user's target is asynchronous and uses resources (like HTTP clients or locks) bound to Giskard's main event loop, this will raise a RuntimeError due to cross-loop access.

To prevent this, we should check if we are running on a different event loop and, if so, schedule the with_interaction call on the main event loop (self._loop) using asyncio.run_coroutine_threadsafe and wrap the future to await it safely.

Suggested change

	# Trace is frozen: with_interaction drives the target and returns a NEW
	# trace, which we store back under the same uuid for the next turn.
	trace = await trace.with_interaction(interaction)
	self.traces[conversation_uuid] = trace
	# Trace is frozen: with_interaction drives the target and returns a NEW
	# trace, which we store back under the same uuid for the next turn.
	# If we are running on a different event loop (e.g. DeepTeam's internal loop),
	# route the call to the main scan loop to avoid cross-loop resource issues.
	import asyncio
	try:
	current_loop = asyncio.get_running_loop()
	except RuntimeError:
	current_loop = None
	if self._loop is not None and self._loop.is_running() and current_loop is not self._loop:
	future = asyncio.run_coroutine_threadsafe(
	trace.with_interaction(interaction), self._loop
	)
	trace = await asyncio.wrap_future(future)
	else:
	trace = await trace.with_interaction(interaction)
	self.traces[conversation_uuid] = trace

[gemini-code-assist]

To prevent potential AttributeErrors if a non-standard or mocked turn object has a metadata attribute that is not a dictionary, we should explicitly verify that metadata is an instance of dict before calling .get().

Suggested change

	metadata = getattr(turn, "metadata", None)
	if metadata and metadata.get(self.METADATA_UUID_KEY):
	return metadata[self.METADATA_UUID_KEY]
	metadata = getattr(turn, "metadata", None)
	if isinstance(metadata, dict) and metadata.get(self.METADATA_UUID_KEY):
	return metadata[self.METADATA_UUID_KEY]

[gemini-code-assist]

If no default generator has been configured in Giskard, get_default_generator() will return None. Passing None to make_deepeval_llm will cause cryptic AttributeErrors later when DeepEval attempts to generate prompts or evaluate results.

We should check if the generator is None and raise a clear, actionable ValueError to guide the user to configure a default generator.

        loop = asyncio.get_running_loop()
        callback = ScanTargetCallback(target=target, loop=loop)
        generator = get_default_generator()
        if generator is None:
            raise ValueError(
                "No default generator configured. Please set a default generator "
                "using set_default_generator() before running the scan."
            )
        llm = make_deepeval_llm(generator, loop)

[gemini-code-assist]

If the provided loop is not running (e.g., in a test environment or if the scan loop has already finished), calling asyncio.run_coroutine_threadsafe will block indefinitely on future.result().

To prevent hanging, we should check if loop.is_running() is True before scheduling the coroutine, and fall back to asyncio.run(coro) if it is not.

Suggested change

	if loop is None:
	return asyncio.run(coro)
	future = asyncio.run_coroutine_threadsafe(coro, loop)
	return future.result()
	if loop is None or not loop.is_running():
	return asyncio.run(coro)
	future = asyncio.run_coroutine_threadsafe(coro, loop)
	return future.result()