Skip to content

Alert rules don't use correct selectors #46

@simu

Description

@simu

We currently use the Thanos mixing Jsonnet library to define alerts per instance of the component. However, because we don't customize the alert selectors in component/alerts.libsonnet, each instance's alerts will pick up metrics from all instances of the deployed Thanos component (e.g. receive).

This makes it currently impossible to have different sets of alerts for different instances of the same Thanos component in a single cluster. Since we advertise multi-instance support, we should make sure that alert rules are configured to only match their corresponding component instances (e.g. by adding 'namespace=%s'%instanceNs to the alert selector(s).

Steps to Reproduce the Problem

  1. Create two instances of the component which deploy the same Thanos component (e.g. receive)
  2. Disable alert rules for one of the instances
  3. Observe that remaining alert rules will pick up both instances

Actual Behavior

Alert rules pick up all Thanos component instances on the cluster instead of only the ones associated with the Commodore component instance.

Expected Behavior

Alert rules are constrained to only cover Thanos instances associated with the Component instance

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions