Studio: stream reasoning tokens in the tool-loop generator (fixes DeepSeek thinking not streaming with a pill on)#6947
Conversation
…pSeek thinking not streaming with a pill on)
for more information, see https://pre-commit.ci
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the llama_cpp backend to stream reasoning tokens incrementally (token-by-token) even when tools are active, aligning its behavior with the no-tool path. It introduces logic to properly close live-streamed <think> blocks with </think> before structured or bare-JSON tool calls drain, ensuring balanced blocks for consumers. Additionally, several unit tests have been added to verify incremental streaming, correct tag closure, and parity with the no-tool path. I have no feedback to provide as there are no review comments.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 23709558cc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
1 participant
With any tool pill active (code, MCP, RAG, canvas), DeepSeek-V4 reasoning stops streaming: the whole thinking phase arrives as one block at the same instant as the first answer token. Without tools it streams token by token.
Problem
enable_tools: trueroutes the request into the tool-loop generator, which holds content in a BUFFERING state until it knows the output isn't a tool call, and only yieldedreasoning_contentwhile STREAMING. Nothing leaves BUFFERING until the first content token, so reason-first models (DeepSeek-V4 isreasoning_always_on; Qwen3-thinking and GLM hit it too) buffer their entire reasoning preamble every turn. Verified live on DeepSeek-V4-Flash: 62 incremental reasoning deltas without tools, one chunk with a pill on.Fix
Yield reasoning whenever
detect_state != _S_DRAINING, so it streams during BUFFERING too. Reasoning is orthogonal to tool detection, which only inspects the content buffer, and the route resets its cumulative cursor ontool_start, so the live<think>block stays a clean monotonic prefix, same as the no-tool path.Two supporting changes:
_flush_reasoning_and_buffercloses an already-streamed<think>instead of re-emitting the whole block.<think>beforetool_start. The XML path already did; the structureddelta.tool_callsand bare-JSON drains now share_close_streamed_think(). Without this, consumers with no reasoning extractor (Anthropic/v1/messages) would get an unclosed tag on a reasoning-then-tool turn.Reasoning-only replies keep resolving to bare text, matching the no-tool path, so the non-streaming drain still returns them as
content.Verification
Tested live with DeepSeek-V4-Flash (UD-Q4_K_XL GGUF): thinking now streams token-by-token with a tool pill active, matching the no-tools path.
New regression tests drive the generator with synthetic SSE deltas: incremental streaming, the
<think>close before both drain types, and reasoning-only parity with the no-tool path (a route-level test replays both generators through the real extractor and asserts identical client output). Each fails on pre-fix code.test_llama_cpp_tool_loop.py61 pass; broader tool/reasoning/Anthropic suites 913 together.