-
Notifications
You must be signed in to change notification settings - Fork 0
Description
We currently use the Thanos mixing Jsonnet library to define alerts per instance of the component. However, because we don't customize the alert selectors in component/alerts.libsonnet, each instance's alerts will pick up metrics from all instances of the deployed Thanos component (e.g. receive
).
This makes it currently impossible to have different sets of alerts for different instances of the same Thanos component in a single cluster. Since we advertise multi-instance support, we should make sure that alert rules are configured to only match their corresponding component instances (e.g. by adding 'namespace=%s'%instanceNs
to the alert selector(s).
Steps to Reproduce the Problem
- Create two instances of the component which deploy the same Thanos component (e.g.
receive
) - Disable alert rules for one of the instances
- Observe that remaining alert rules will pick up both instances
Actual Behavior
Alert rules pick up all Thanos component instances on the cluster instead of only the ones associated with the Commodore component instance.
Expected Behavior
Alert rules are constrained to only cover Thanos instances associated with the Component instance