fix(controller): add timeout to RemoteMCPServer registration by AnantKumar17 · Pull Request #1805 · kagent-dev/kagent

AnantKumar17 · 2026-05-06T11:49:37Z

What

Fixes a bug where a single hung RemoteMCPServer registration blocks all subsequent RemoteMCPServer reconciliations for the lifetime of the controller process.

Closes #1785

Why

newHTTPClient() was returning http.DefaultClient (no timeout) when there were no custom headers, and listTools() was passing the context directly to client.Connect() and session.ListTools() with no deadline. A hung or unreachable endpoint — for example, an IPv6 address resolved in an IPv4-only cluster — would block the goroutine indefinitely, preventing any subsequent RemoteMCPServer from registering until the controller pod was restarted.

Changes

Single file changed: go/core/internal/controller/reconciler/reconciler.go

newHTTPClient(): always returns a *http.Client with Timeout: mcpRegistrationTimeout set, in both the no-headers and with-headers branches. Removes the use of http.DefaultClient.
upsertToolServerForRemoteMCPServer(): wraps the transport creation and tool listing in context.WithTimeout(ctx, mcpRegistrationTimeout) so the entire registration sequence is bounded. The database call (RefreshToolsForServer) intentionally uses the original ctx.
ReconcileKagentRemoteMCPServer(): logs registration start (url, protocol), success (url, tool count, duration), and failure (error, duration) using structured logr key-value pairs, matching existing codebase conventions.
mcpRegistrationTimeout: new unexported constant set to 30 * time.Second, matching DefaultTimeout used in discoverer.go.

Testing

go test -race -skip 'TestE2E.*' ./...

All 26 test packages pass. go vet clean. go build clean.

Copilot

Pull request overview

This PR fixes a controller reliability bug where a hung RemoteMCPServer registration could block all subsequent RemoteMCPServer reconciliations by ensuring the registration workflow is time-bounded and observable via structured logs.

Changes:

Adds a 30s registration timeout and applies it to MCP transport creation + tool listing.
Ensures newHTTPClient() always returns an *http.Client with a timeout (no longer returns http.DefaultClient).
Adds structured logr entries for registration start, success, and failure with duration.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

AnantKumar17 · 2026-05-06T12:43:10Z

+	// mcpRegistrationTimeout is the deadline applied to each RemoteMCPServer
+	// registration attempt (header resolution + MCP connect + tool listing).
+	// A hung or unreachable endpoint is bounded to this duration, ensuring the
+	// reconciler goroutine is always released and does not block subsequent
+	// RemoteMCPServer reconciliations.
+	mcpRegistrationTimeout = 30 * time.Second
 )



added a remoteMCPRegistrationTimeout() helper that returns spec.timeout when set and falls back to the 30s default when the field is nil. The constant is now just the fallback value.

AnantKumar17 · 2026-05-06T12:43:42Z

+	// Bound the entire registration sequence (header resolution + MCP connect +
+	// tool listing) to mcpRegistrationTimeout so that a hung or unreachable
+	// endpoint cannot block this goroutine — and therefore all subsequent
+	// RemoteMCPServer reconciliations — indefinitely.
+	tCtx, cancel := context.WithTimeout(ctx, mcpRegistrationTimeout)
+	defer cancel()
+
+	tsp, err := a.createMcpTransport(tCtx, remoteMcpServer)


Fixed - upsertToolServerForRemoteMCPServer now calls context.WithTimeout(ctx, remoteMCPRegistrationTimeout(remoteMcpServer)) so operators can tune the deadline per resource via .spec.timeout without any code changes.

AnantKumar17 · 2026-05-06T12:44:01Z

 // go-sdk does not have a WithHeaders option when initializing transport
 // so we need to create a custom HTTP client that adds headers to all requests.
 func newHTTPClient(headers map[string]string) *http.Client {
 	if len(headers) == 0 {
-		return http.DefaultClient
+		return &http.Client{
+			Timeout: mcpRegistrationTimeout,
+		}
 	}
 	return &http.Client{
+		Timeout: mcpRegistrationTimeout,
 		Transport: &headerTransport{


Updated newHTTPClient to accept an explicit timeout time.Duration parameter. createMcpTransport derives it from remoteMCPRegistrationTimeout(s) and passes it in, so the HTTP client timeout matches the CRD configuration in both branches.

AnantKumar17 · 2026-05-06T12:44:29Z

+	// Bound the entire registration sequence (header resolution + MCP connect +
+	// tool listing) to mcpRegistrationTimeout so that a hung or unreachable
+	// endpoint cannot block this goroutine — and therefore all subsequent
+	// RemoteMCPServer reconciliations — indefinitely.
+	tCtx, cancel := context.WithTimeout(ctx, mcpRegistrationTimeout)
+	defer cancel()
+
+	tsp, err := a.createMcpTransport(tCtx, remoteMcpServer)
 	if err != nil {
 		return nil, fmt.Errorf("failed to create client for toolServer %s: %w", toolServer.Name, err)
 	}

-	tools, err := a.listTools(ctx, tsp, toolServer)
+	tools, err := a.listTools(tCtx, tsp, toolServer)


Added TestRemoteMCPRegistrationTimeout (covers nil server, nil spec.timeout, and a custom value) and TestNewHTTPClient (covers nil/empty/with-headers - all assert the correct timeout and transport type). These are in reconciler_test.go in the same package.

jmhbh

Changes LGTM overall, just have on enit

jmhbh · 2026-05-06T21:59:07Z

+	// .spec.timeout is not set. A hung or unreachable endpoint is bounded to this
+	// duration, ensuring the reconciler goroutine is always released and does not
+	// block subsequent RemoteMCPServer reconciliations.
+	mcpRegistrationTimeout = 30 * time.Second


nit: I almost prefer setting this as a default value in remotemcpserver_types.go instead in the spec.Timeout field so its more explicit.

Done - added // +kubebuilder:default="30s" on Timeout in remotemcpserver_types.go and regenerated both the base CRD and the Helm CRD template.
The Go fallback in remoteMCPRegistrationTimeout() is kept for existing resources that were created before this default takes effect (nil Timeout at admission time).

Hi @jmhbh ,

I’ve addressed the comments put by you earlier, and also got approval from Eltanya.
Could you please have a look as well, when you get a chance? Since merging is currently blocked pending code owner approval.

Thanks!

Fixes the issue 1785 Signed-off-by: AnantKumar17 <anant3011k@gmail.com>

Signed-off-by: AnantKumar17 <anant3011k@gmail.com>

…ient Signed-off-by: AnantKumar17 <anant3011k@gmail.com>

Signed-off-by: AnantKumar17 <anant3011k@gmail.com>

…dev#1805) ## What Fixes a bug where a single hung `RemoteMCPServer` registration blocks all subsequent `RemoteMCPServer` reconciliations for the lifetime of the controller process. Closes kagent-dev#1785 ## Why `newHTTPClient()` was returning `http.DefaultClient` (no timeout) when there were no custom headers, and `listTools()` was passing the context directly to `client.Connect()` and `session.ListTools()` with no deadline. A hung or unreachable endpoint — for example, an IPv6 address resolved in an IPv4-only cluster — would block the goroutine indefinitely, preventing any subsequent `RemoteMCPServer` from registering until the controller pod was restarted. ## Changes Single file changed: `go/core/internal/controller/reconciler/reconciler.go` - **`newHTTPClient()`**: always returns a `*http.Client` with `Timeout: mcpRegistrationTimeout` set, in both the no-headers and with-headers branches. Removes the use of `http.DefaultClient`. - **`upsertToolServerForRemoteMCPServer()`**: wraps the transport creation and tool listing in `context.WithTimeout(ctx, mcpRegistrationTimeout)` so the entire registration sequence is bounded. The database call (`RefreshToolsForServer`) intentionally uses the original `ctx`. - **`ReconcileKagentRemoteMCPServer()`**: logs registration start (url, protocol), success (url, tool count, duration), and failure (error, duration) using structured `logr` key-value pairs, matching existing codebase conventions. - **`mcpRegistrationTimeout`**: new unexported constant set to `30 * time.Second`, matching `DefaultTimeout` used in `discoverer.go`. ## Testing ``` go test -race -skip 'TestE2E.*' ./... ``` All 26 test packages pass. `go vet` clean. `go build` clean. --------- Signed-off-by: AnantKumar17 <anant3011k@gmail.com> Co-authored-by: Eitan Yarmush <eitan.yarmush@solo.io>

Copilot AI review requested due to automatic review settings May 6, 2026 11:49

AnantKumar17 requested review from iplay88keys, jmhbh and supreme-gg-gg as code owners May 6, 2026 11:49

github-actions Bot added the bug Something isn't working label May 6, 2026

Copilot started reviewing on behalf of AnantKumar17 May 6, 2026 11:50 View session

Copilot AI reviewed May 6, 2026

View reviewed changes

jmhbh reviewed May 6, 2026

View reviewed changes

AnantKumar17 added 4 commits May 7, 2026 11:14

fix(controller): add timeout to RemoteMCPServer registration

71ed02b

Fixes the issue 1785 Signed-off-by: AnantKumar17 <anant3011k@gmail.com>

fix(controller): use spec.timeout per RemoteMCPServer when set

7323396

Signed-off-by: AnantKumar17 <anant3011k@gmail.com>

test(controller): add unit tests for registration timeout and HTTP cl…

4512e13

…ient Signed-off-by: AnantKumar17 <anant3011k@gmail.com>

fix(api): set kubebuilder default of 30s on spec.timeout

f3f1282

Signed-off-by: AnantKumar17 <anant3011k@gmail.com>

AnantKumar17 force-pushed the fix/remotemcpserver-registration-timeout branch from b868f98 to f3f1282 Compare May 7, 2026 06:03

AnantKumar17 requested review from EItanya, ilackarms and yuval-k as code owners May 7, 2026 06:03

EItanya approved these changes May 7, 2026

View reviewed changes

Merge branch 'main' into fix/remotemcpserver-registration-timeout

ce8e1d2

AnantKumar17 requested a review from jmhbh May 8, 2026 04:13

Merge branch 'main' into fix/remotemcpserver-registration-timeout

b88670e

EItanya merged commit a056369 into kagent-dev:main May 11, 2026
23 checks passed

BrewTestBot mentioned this pull request May 11, 2026

kagent 0.9.3 Homebrew/homebrew-core#282066

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(controller): add timeout to RemoteMCPServer registration#1805

fix(controller): add timeout to RemoteMCPServer registration#1805
EItanya merged 6 commits into
kagent-dev:mainfrom
AnantKumar17:fix/remotemcpserver-registration-timeout

AnantKumar17 commented May 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

AnantKumar17 May 6, 2026

Uh oh!

AnantKumar17 May 6, 2026

Uh oh!

AnantKumar17 May 6, 2026

Uh oh!

AnantKumar17 May 6, 2026

Uh oh!

jmhbh left a comment

Uh oh!

jmhbh May 6, 2026

Uh oh!

AnantKumar17 May 7, 2026

Uh oh!

AnantKumar17 May 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

AnantKumar17 commented May 6, 2026

What

Why

Changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

AnantKumar17 May 6, 2026

Choose a reason for hiding this comment

Uh oh!

AnantKumar17 May 6, 2026

Choose a reason for hiding this comment

Uh oh!

AnantKumar17 May 6, 2026

Choose a reason for hiding this comment

Uh oh!

AnantKumar17 May 6, 2026

Choose a reason for hiding this comment

Uh oh!

jmhbh left a comment

Choose a reason for hiding this comment

Uh oh!

jmhbh May 6, 2026

Choose a reason for hiding this comment

Uh oh!

AnantKumar17 May 7, 2026

Choose a reason for hiding this comment

Uh oh!

AnantKumar17 May 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants