Adjustable sleep duration when SIGTERM received

[harwck]

Is your feature request related to a problem? Please describe.

In Kubernetes deployments, when new version of a deployment being rolled out, Kubernetes will continue sending traffic to pods in a TERMINATING state, so we will need to set terminationGracePeriod to a period, accordingly the application running in the container should also sleep and continue to handle incoming traffic until Kubernetes decided to stop sending traffic to it, the duration it takes for Kubernetes to stop sending traffic to pods in TERMINATING state sometimes depends on Cloud Providers or kube versions, so having a configurable sleep period in the termination signal handler might be a good idea.

Describe the solution you'd like

having a configurable sleep period in the termination signal handler might be a good idea.

Describe alternatives you've considered
I have tried:

• Side car container sleeping
• Copying sleep binary into /mnt and run that in the preStop hook

both aren't working well because it won't stop tyk itself from terminating

Additional context
Add any other context or screenshots about the feature request here.

[buger]

/probe what do you think? Is it possible to achive it with what we already have now? I know there were a recent changes related to graceful shutdown.

[github-actions]

Current Implementation Analysis

After examining the Tyk Gateway codebase, I can confirm that there is currently no configuration option for adding a sleep delay between receiving a SIGTERM signal and initiating the graceful shutdown process.

The current implementation works as follows:

1. In gateway/server.go, a signal handler is set up to catch SIGTERM, SIGQUIT, and SIGINT signals.
2. When a signal is received, it immediately calls cancel() on the context, which triggers the shutdown process.
3. Then it creates a timeout context with GracefulShutdownTimeoutDuration and calls gw.gracefulShutdown(shutdownCtx).
4. The gracefulShutdown method shuts down all HTTP servers and TCP proxies, waits for in-flight requests to complete, and closes resources.

The existing GracefulShutdownTimeoutDuration parameter (default: 30 seconds) controls how long to wait for in-flight requests to complete, not how long to delay before starting the shutdown process.

Proposed Solution

To implement the requested feature, a new configuration parameter called GracefulShutdownDelay would need to be added to the Config struct in config/config.go. This would specify the number of seconds to wait after receiving a SIGTERM signal before initiating the graceful shutdown process.

The implementation would require modifying the signal handler in gateway/server.go to add a sleep period after receiving the signal but before calling cancel() and starting the graceful shutdown process.

Here's a sketch of the changes needed:

1. Add a new configuration parameter in config/config.go:

// GracefulShutdownDelay sets how many seconds the gateway should wait after receiving SIGTERM
// before initiating the graceful shutdown process. Useful in Kubernetes environments
// where pods in TERMINATING state still receive traffic. Defaults to 0 seconds (no delay).
GracefulShutdownDelay int `json:"graceful_shutdown_delay"`

2. Modify the signal handler in gateway/server.go to use this parameter:

go func() {
    sig := <-sigChan
    mainLog.Infof("Shutdown signal received: %v.", sig)
    
    // Add delay if configured
    if gwConfig.GracefulShutdownDelay > 0 {
        mainLog.Infof("Delaying shutdown for %d seconds...", gwConfig.GracefulShutdownDelay)
        time.Sleep(time.Duration(gwConfig.GracefulShutdownDelay) * time.Second)
    }
    
    mainLog.Info("Initiating graceful shutdown...")
    cancel()
    
    // Rest of the shutdown process remains the same
    // ...
}()

This implementation would allow Tyk to continue handling traffic for the configured delay period after receiving the SIGTERM signal, which would be useful in Kubernetes environments where pods in TERMINATING state still receive traffic until the termination grace period expires.

Conclusion

The requested feature is not currently available in Tyk Gateway, but it would be a valuable addition for Kubernetes deployments. The implementation would be relatively straightforward and would not require significant changes to the existing codebase.

---

Tip: Mention me again using /probe <request>.
Powered by Probe AI