server, webui: support continue generation on reasoning models by ServeurpersoCom · Pull Request #22727 · ggml-org/llama.cpp

ServeurpersoCom · 2026-05-05T17:15:31Z

Overview

Reasoning models can now use the Continue button. Stopping mid thought saves the partial chain of thought, F5 keeps it, and clicking Continue resumes inside the thinking block instead of restarting from scratch. Same behavior for stops after the thinking ends. Plain content prefill is unchanged.

21754.reasoning-continue-prefill.mp4

Additional information

Backend resolves the old TODO in oaicompat_chat_params_parse: removes the throw blocking assistant prefill on reasoning models and the forced reasoning_format = NONE workaround, then orchestrates thinking_start_tag, thinking_end_tag and generation_prompt around the prefilled message so the prompt is rebuilt correctly and the parser introduced in PR #20424 routes the next stream chunks to reasoning_content or content depending on whether the prefill is plain content, mid reasoning, or post reasoning. Bridges the API field from #21036, the parser routing from #20424 and the webui storage from #21249.

Frontend drops the reasoning_content guard on the Continue button, sends reasoning_content with the prefilled assistant message in continueAssistantMessage, persists partial reasoningContent on stop so the CoT survives F5 and Continue, and marks the streaming state on reasoning chunks so savePartialResponseIfNeeded does not early return when stop happens before any content token. Setting hint updated.

First step toward #21754: covers voluntary stop and reload, full network resilience (SSE resume) is left for a follow up.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Opus 4.7 + local MCP server with rootless pod

ServeurpersoCom · 2026-05-05T17:23:57Z

I add a video

ServeurpersoCom · 2026-05-05T18:02:50Z

It's possible that a space or line break might be missing during retrieval. This could be due to an idempotence issue with the tokenizer ? It's not critical, but it slightly skews the distribution. (like a micro-error in the KV Cache, which the model could not have generated on its own)

ServeurpersoCom · 2026-05-05T18:04:48Z

It makes me want to write a Python hammer script that does lots of pauses/resumes to see what happens and isolate a front / back bug

aldehir · 2026-05-05T18:09:46Z

+        const bool thinking_active = chat_params.supports_thinking && !chat_params.thinking_end_tag.empty();
+        const bool has_reasoning   = !last_message.reasoning_content.empty();
+        const bool has_content     = !last_message.content.empty() || !last_message.content_parts.empty();
+        const bool mid_reasoning   = has_reasoning && !has_content;
+
+        // some templates inject thinking_start in generation_prompt, others let the model emit it
+        const bool gp_has_think = thinking_active
+            && chat_params.generation_prompt.find(chat_params.thinking_start_tag) != std::string::npos;
+
+        // open the thinking block when reasoning is present and the template did not inject it
+        if (has_reasoning) {
+            if (thinking_active && !gp_has_think) {
+                chat_params.prompt += chat_params.thinking_start_tag;
+            }
+            chat_params.prompt += last_message.reasoning_content;
+        }
+
+        if (thinking_active) {
+            if (mid_reasoning) {
+                // model continues inside the thinking block, keep generation_prompt open on think
+                if (!gp_has_think) {
+                    chat_params.generation_prompt += chat_params.thinking_start_tag;
+                }
+            } else {
+                // close thinking block when reasoning is followed by content, or when the template forced it open
+                if (has_reasoning || gp_has_think) {
+                    chat_params.prompt += chat_params.thinking_end_tag;
+                }
+                // strip thinking_start from generation_prompt so the parser routes model output as content
+                auto pos = chat_params.generation_prompt.rfind(chat_params.thinking_start_tag);
+                if (pos != std::string::npos) {
+                    chat_params.generation_prompt = chat_params.generation_prompt.substr(0, pos);
+                }
+            }
+        }
+


This is not universal. Some models, like gpt-oss, wrap the assistant content as well. This logic must be delegated to the chat handler for the given template. That is where the knowledge required to properly rebuild the assistant message exists.

I am working on the plumbing in common required to support this.

Great! I'm iterating in the pod with Opus: a Python client that stress tests all this on different models; GPT is going to fail. I will rewrite the code later based on your new API.

allozaur · 2026-05-06T07:24:34Z

Gr8 stuff, @ServeurpersoCom @aldehir let's push further to have it working in production asap :)

ServeurpersoCom · 2026-05-06T08:29:53Z

We will have the missing layer of abstraction! the frontend will pose no problem (at most, a small commit to add a state backup in the callback for network outages), and the C++ will have a clean abstracted OAI chunks redirector!

aldehir · 2026-05-06T21:08:56Z

@allozaur Understood. I'll focus on the common API and open a PR soon.

allozaur · 2026-05-12T11:31:05Z

Hey @aldehir @ServeurpersoCom any updates on that one?

aldehir · 2026-05-12T13:12:47Z

I haven't had time to implement the changes in common. If you want, you can proceed with this PR and I will add it later.

ServeurpersoCom · 2026-05-12T15:46:30Z

As you wish, I can rebase. We can make it work, because the frontend won't change, just the backend function needs updating so that the interrupt also works with GPT-OSS.

…-org#22727) Remove the throw blocking assistant prefill on reasoning models and orchestrate thinking tags around the prefilled message so the parser routes the next stream chunks correctly. WebUI drops the reasoning guard on the Continue button, sends reasoning_content with the prefilled message and persists partial reasoning on stop so the CoT survives reload and resume. Scope : templates with a simple thinking_start_tag / thinking_end_tag pair. Channel-based templates like GPT-OSS are out of scope, pending a per-template prefill API in common/chat. First step toward ggml-org#21754.

ServeurpersoCom · 2026-05-12T18:36:05Z

Rebased and deployed on my server to test the post-rebase version while CI works, need re-approving please:)

aldehir

Approving for just the server backend portion of this. It seems OK to me. Hopefully I get some free time soon to shift this to common.

…-org#22727) Remove the throw blocking assistant prefill on reasoning models and orchestrate thinking tags around the prefilled message so the parser routes the next stream chunks correctly. WebUI drops the reasoning guard on the Continue button, sends reasoning_content with the prefilled message and persists partial reasoning on stop so the CoT survives reload and resume. Scope : templates with a simple thinking_start_tag / thinking_end_tag pair. Channel-based templates like GPT-OSS are out of scope, pending a per-template prefill API in common/chat. First step toward ggml-org#21754.

ServeurpersoCom · 2026-05-12T20:09:05Z

Tested on GPT-OSS, which is now out of scope: stop/continue desyncs the WebUI from the inference. The server keeps generating but the frontend stops receiving content. I'll rework it a bit.

ServeurpersoCom · 2026-05-12T20:34:55Z

Frontend is already OK. Added a small backend guard that rejects unsupported templates (for resume only) with a 400. Once the per-template prefill API lands, just one backend function to refactor in a follow up.

aldehir · 2026-05-13T02:19:19Z

I went down the rabbit hole to see how vLLM supports this feature. They have a separate flag, continue_final_message, instead of assuming continuation from the last message: https://docs.vllm.ai/en/stable/serving/openai_compatible_server/

    continue_final_message: bool = Field(
        default=False,
        description=(
            "If this is set, the chat will be formatted so that the final "
            "message in the chat is open-ended, without any EOS tokens. The "
            "model will continue this message rather than starting a new one. "
            'This allows you to "prefill" part of the model\'s response for it. '
            "Cannot be used at the same time as `add_generation_prompt`."
        ),
    )

The implementation is in the transformers library: https://github.com/huggingface/transformers/blob/0d476f2cc725bdfbde425d355aa1605ece7a963f/src/transformers/utils/chat_template_utils.py#L539-L596

It renders with add_generation_prompt = false, injects a sentinel value in the final assistant message, and then strips it out.

To maintain parity for the sake of client support, how hard do you think it is to send a request with add_generation_prompt = false and continue_final_message = true on continuation?

allozaur · 2026-05-13T09:09:46Z

@aldehir i am merging this PR as it's mostly webui changes. For any further changes on server-side let's have a follow-up PR

ServeurpersoCom · 2026-05-13T12:35:00Z

I went down the rabbit hole to see how vLLM supports this feature. They have a separate flag, continue_final_message, instead of assuming continuation from the last message: https://docs.vllm.ai/en/stable/serving/openai_compatible_server/
    continue_final_message: bool = Field(
        default=False,
        description=(
            "If this is set, the chat will be formatted so that the final "
            "message in the chat is open-ended, without any EOS tokens. The "
            "model will continue this message rather than starting a new one. "
            'This allows you to "prefill" part of the model\'s response for it. '
            "Cannot be used at the same time as `add_generation_prompt`."
        ),
    )
The implementation is in the transformers library: https://github.com/huggingface/transformers/blob/0d476f2cc725bdfbde425d355aa1605ece7a963f/src/transformers/utils/chat_template_utils.py#L539-L596

It renders with add_generation_prompt = false, injects a sentinel value in the final assistant message, and then strips it out.

To maintain parity for the sake of client support, how hard do you think it is to send a request with add_generation_prompt = false and continue_final_message = true on continuation?

Brilliant find, the sentinel approach look much cleaner and aligns with vLLM, I'm up for tackling it as a follow up PR.

ServeurpersoCom · 2026-05-13T12:57:21Z

Problem: the vLLM method delegates the entire responsibility to the Jinja template, so it silently breaks on templates that strip the last assistant message, force a turn end, ignore reasoning_content or trim whitespace, and many older GGUFs embed exactly these broken templates.

Question: do we accept these limitations and only support compliant templates (with a clear error message on the others), or do we also keep the legacy path as an automatic fallback to catch broken GGUFs at the cost of carrying technical debt?

(This only applies to the stop/continue feature) @aldehir WDYT ?

ServeurpersoCom · 2026-05-13T13:02:28Z

I'm going to start with a minimalist approach, without any legacy, to see how it works in practice with my zoo.

aldehir · 2026-05-13T14:13:11Z

Problem: the vLLM method delegates the entire responsibility to the Jinja template, so it silently breaks on templates that strip the last assistant message, force a turn end, ignore reasoning_content or trim whitespace, and many older GGUFs embed exactly these broken templates.

The implementation in the transformers library is template agnostic. Regardless, I'm only suggesting an API change between the frontend and backend. Right now we assume assistant prefill if the final message is an assistant.

I started implementing this in common. Will follow up in a PR.

(This only applies to the stop/continue feature) @aldehir WDYT ?

The continue flag would only apply on the continue feature, yes. That should be the scope of the frontend changes.

ServeurpersoCom · 2026-05-13T14:21:25Z

I finally realized it when I looked at the code.
Right now I'm only updating the API to be compatible with vLLM, and the rest will follow.

…org#22727) * server, webui : support continue generation on reasoning models (ggml-org#22727) Remove the throw blocking assistant prefill on reasoning models and orchestrate thinking tags around the prefilled message so the parser routes the next stream chunks correctly. WebUI drops the reasoning guard on the Continue button, sends reasoning_content with the prefilled message and persists partial reasoning on stop so the CoT survives reload and resume. Scope : templates with a simple thinking_start_tag / thinking_end_tag pair. Channel-based templates like GPT-OSS are out of scope, pending a per-template prefill API in common/chat. First step toward ggml-org#21754. * chore: update webui build output * server: reject reasoning prefill on channel based templates

ServeurpersoCom requested review from a team as code owners May 5, 2026 17:15

aldehir reviewed May 5, 2026

View reviewed changes

github-actions Bot added server/webui examples server labels May 5, 2026

ServeurpersoCom force-pushed the reasoning-continue-prefill branch from 9958111 to 85400a8 Compare May 12, 2026 18:30

chore: update webui build output

1d48840

aldehir approved these changes May 12, 2026

View reviewed changes

server: reject reasoning prefill on channel based templates

27f4d1f

allozaur approved these changes May 13, 2026

View reviewed changes

allozaur merged commit 5d44db6 into ggml-org:master May 13, 2026
47 checks passed

ServeurpersoCom mentioned this pull request May 13, 2026

server, webui: accept continue_final_message flag for vLLM API compat #23012

Merged

aldehir mentioned this pull request May 15, 2026

common : delegate assistant continuation to underlying template handlers #23089

Merged

Uh oh!

Conversation

ServeurpersoCom commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

First step toward #21754: covers voluntary stop and reload, full network resilience (SSE resume) is left for a follow up.

Requirements

Uh oh!

ServeurpersoCom commented May 5, 2026

Uh oh!

ServeurpersoCom commented May 5, 2026

Uh oh!

ServeurpersoCom commented May 5, 2026

Uh oh!

aldehir May 5, 2026

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom May 5, 2026

Choose a reason for hiding this comment

Uh oh!

allozaur commented May 6, 2026

Uh oh!

ServeurpersoCom commented May 6, 2026

Uh oh!

aldehir commented May 6, 2026

Uh oh!

allozaur commented May 12, 2026

Uh oh!

aldehir commented May 12, 2026

Uh oh!

ServeurpersoCom commented May 12, 2026

Uh oh!

ServeurpersoCom commented May 12, 2026

Uh oh!

aldehir left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom commented May 12, 2026

Uh oh!

ServeurpersoCom commented May 12, 2026

Uh oh!

aldehir commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

allozaur commented May 13, 2026

Uh oh!

Uh oh!

ServeurpersoCom commented May 13, 2026

Uh oh!

ServeurpersoCom commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ServeurpersoCom commented May 13, 2026

Uh oh!

aldehir commented May 13, 2026

Uh oh!

ServeurpersoCom commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ServeurpersoCom commented May 5, 2026 •

edited

Loading

aldehir left a comment •

edited

Loading

aldehir commented May 13, 2026 •

edited

Loading

ServeurpersoCom commented May 13, 2026 •

edited

Loading