Add ruler tenant federation

CHANGELOG.md

@@ -17,6 +17,7 @@
 * [CHANGE] Querier: Make query time range configurations per-tenant: `query_ingesters_within`, `query_store_after`, and `shuffle_sharding_ingesters_lookback_period`. Uses `model.Duration` instead of `time.Duration` to support serialization but has minimum unit of 1ms (nanoseconds/microseconds not supported). #7160
 * [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446
 * [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481
+* [FEATURE] Ruler: Add experimental support for federated rule groups. A rule group listing tenants in its `src_tenants` field is evaluated against those tenants while the resulting series and alerts are written to the tenant owning the rule group. Enabled with `-ruler.enable-federated-rules` (requires `-tenant-federation.enabled`), and restricted to selected tenants with `-ruler.allowed-federated-tenants` and `-ruler.disallowed-federated-tenants`. #7828
 * [FEATURE] Parquet: Support sharded parquet file conversion and querying. #7610
 * [FEATURE] Parquet Converter: Add experimental `-parquet-converter.max-num-columns` flag to automatically shard parquet files when the number of columns exceeds the configured limit. This prevents failures when a TSDB block has more unique label names than the parquet library's column limit (32767). #7624
 * [FEATURE] Distributor: Add experimental `-distributor.num-query-workers` flag to use a goroutine worker pool for query fan-out calls to ingesters. Reuses pre-grown goroutine stacks to eliminate the `runtime.copystack` overhead (~8% CPU) observed on rulers with wide ingester fan-out. Falls back to spawning a new goroutine when no worker is available. #7623

docs/configuration/config-file-reference.md

@@ -6243,6 +6243,25 @@ ring:
 # CLI flag: -ruler.disabled-tenants
 [disabled_tenants: <string> | default = ""]
 
+# [Experimental] Enable federated rule groups. A federated rule group lists the
+# tenants to query in its `src_tenants` field, while the resulting series and
+# alerts always belong to the tenant owning the rule group. Requires
+# -tenant-federation.enabled=true.
+# CLI flag: -ruler.enable-federated-rules
+[enable_federated_rules: <boolean> | default = false]
+
+# [Experimental] Comma separated list of tenants allowed to create federated
+# rule groups. If specified, only these tenants can create federated rule
+# groups, otherwise all tenants can.
+# CLI flag: -ruler.allowed-federated-tenants
+[allowed_federated_tenants: <string> | default = ""]
+
+# [Experimental] Comma separated list of tenants that cannot create federated
+# rule groups. If specified, a tenant that would normally be allowed to create
+# federated rule groups is denied instead.
+# CLI flag: -ruler.disallowed-federated-tenants
+[disallowed_federated_tenants: <string> | default = ""]
+
 # Report query statistics for ruler queries to complete as a per user metric and
 # as an info level log message.
 # CLI flag: -ruler.query-stats-enabled

docs/configuration/v1-guarantees.md

@@ -38,6 +38,7 @@ Currently experimental features are:
 - Ruler
   - Evaluate rules to query frontend instead of ingesters (enabled via `-ruler.frontend-address`).
   - When `-ruler.frontend-address` is specified, the response format can be specified (via `-ruler.query-response-format`).
+  - Federated rule groups (`-ruler.enable-federated-rules`, `-ruler.allowed-federated-tenants`, `-ruler.disallowed-federated-tenants`).
 - S3 Server Side Encryption (SSE) using KMS (including per-tenant KMS config overrides).
 - Alertmanager:
   - Receiver integrations firewall (configured via `-alertmanager.receivers-firewall.*`)

docs/guides/ruler-tenant-federation.md

@@ -0,0 +1,108 @@
+---
+title: "Ruler tenant federation"
+linkTitle: "Ruler tenant federation"
+weight: 10
+slug: ruler-tenant-federation
+---
+
+This guide explains how to configure the Ruler to evaluate federated rule groups, which query data from several tenants while the resulting series and alerts belong to a single tenant. The feature is experimental and implements the [federated ruler proposal](../proposals/federated-ruler.md).
+
+## How it works
+
+A federated rule group is a regular rule group with an additional `src_tenants` field listing the tenants to query:
+
+```yaml
+name: cortex-admin
+interval: 1m
+src_tenants: [team-a, team-b, team-c]
+rules:
+  - record: tenant:prometheus_rule_evaluation_failures:rate5m
+    expr: sum by (__tenant_id__) (rate(prometheus_rule_evaluation_failures_total[5m]))
+  - alert: TenantRuleEvaluationFailures
+    expr: sum by (__tenant_id__) (rate(prometheus_rule_evaluation_failures_total[5m])) > 0
+    for: 10m
+    labels:
+      severity: warning
+    annotations:
+      summary: "Rule evaluations are failing in tenant {{ $labels.__tenant_id__ }}"
+```
+
+- The rule group is owned by the tenant that created it through the Ruler API, `infra` for example.
+- Every rule in the group is evaluated with the `X-Scope-OrgID` set to `team-a|team-b|team-c`, so the query runs across the source tenants exactly like a federated query, and each series carries the `__tenant_id__` label. As for any federated query, the label is not added when `src_tenants` lists a single tenant.
+- The resulting series, the `ALERTS` and `ALERTS_FOR_STATE` series and the notifications are written to `infra` only. The source tenants are never modified and cannot see the rule group.
+- Rule groups without `src_tenants` are not affected.
+
+Alerting rules work the same way: the `TenantRuleEvaluationFailures` alert above fires once per source tenant with failing rule evaluations, and every alert is sent to the Alertmanager configuration of `infra`. The `__tenant_id__` label is kept on the alert as long as the expression does not aggregate it away, so the Alertmanager configuration of the owning tenant can route the alerts per source tenant:
+
+```yaml
+route:
+  receiver: infra-default
+  routes:
+    - matchers: ['__tenant_id__="team-a"']
+      receiver: team-a-slack
+    - matchers: ['__tenant_id__="team-b"']
+      receiver: team-b-slack
+```
+
+### Chaining rules within a federated rule group
+
+In Prometheus, the rules of a group are evaluated in order, so a rule can use the series recorded by a previous rule of the same group. In a federated rule group this does not work out of the box, because the results of a recording rule are stored in the owning tenant while the following rules still query the source tenants. In the following group owned by `infra`, the alert never fires: `job:requests:rate5m` is written to `infra`, but the alert looks for it in `team-a` and `team-b`.
+
+```yaml
+name: traffic
+src_tenants: [team-a, team-b]
+rules:
+  - record: job:requests:rate5m
+    expr: sum by (job) (rate(http_requests_total[5m]))
+  - alert: HighTraffic
+    expr: job:requests:rate5m > 1000
+```
+
+To reuse the output of a previous rule, add the owning tenant to `src_tenants`. The recorded series is then found in `infra`, carrying the `__tenant_id__="infra"` label:
+
+```yaml
+name: traffic
+src_tenants: [infra, team-a, team-b]
+rules:
+  - record: job:requests:rate5m
+    expr: sum by (job) (rate(http_requests_total[5m]))
+  - alert: HighTraffic
+    expr: job:requests:rate5m > 1000
+```
+
+## Configuration
+
+Federated rule groups require multi-tenant query federation and the ruler flag:
+
+```
+-tenant-federation.enabled=true
+-ruler.enable-federated-rules=true
+```
+
+`-tenant-federation.enabled` must be set on all Cortex services. When the ruler evaluates rules through the query frontend (`-ruler.frontend-address`), the query frontend and the queriers perform the federated query; otherwise the ruler merges the results of the source tenants itself.
+
+When the feature is disabled, the Ruler API rejects rule groups with `src_tenants` and any stored federated rule group is skipped with a warning log.
+
+### Restricting the tenants allowed to create federated rule groups
+
+By default, every tenant can create federated rule groups querying any tenant. The following flags restrict which tenants may own federated rule groups:
+
+```
+-ruler.allowed-federated-tenants=infra,platform
+-ruler.disallowed-federated-tenants=untrusted
+```
+
+- If `-ruler.allowed-federated-tenants` is set, only the listed tenants can create federated rule groups.
+- If `-ruler.disallowed-federated-tenants` is set, the listed tenants cannot create federated rule groups even if they are allowed otherwise.
+
+The checks apply when a rule group is created and again when the ruler loads the rule groups, so changing the flags (and restarting the ruler) also disables the federated rule groups already stored for a tenant. Note that these flags do not restrict which tenants can be listed in `src_tenants`.
+
+### Limits
+
+- `-tenant-federation.max-tenant` also limits the number of tenants listed in `src_tenants`.
+- When `-tenant-federation.regex-matcher-enabled` is set, the joined tenant IDs are resolved as a regular expression against the tenants discovered in the blocks storage. Tenant IDs containing regex metacharacters (`.`, `*`, `(`, `)`) are therefore rejected in `src_tenants`, and a source tenant that has not uploaded any block yet is silently ignored.
+
+## Deployment notes
+
+- Rulers running a version without this feature ignore the `src_tenants` field and evaluate such rule groups against the owning tenant only. Enable the feature and create federated rule groups only once every ruler has been upgraded, and delete them before downgrading.
+- The `local` and `configdb` rule stores load Prometheus rule files, which cannot contain `src_tenants`. Federated rule groups require a rule store backed by an object store.

docs/proposals/federated-ruler.md

@@ -71,7 +71,7 @@ To support this we suggest an additional field `src_tenants` on the rule group c
 
 | Challenge                                                                | Status                                |
 |--------------------------------------------------------------------------|---------------------------------------|
-| Allow federated rules behind feature flag                                | Planned but not yet implemented       |
-| Allow federated rules only for select tenants                            | Planned but not yet implemented       |
-| Where to store resulting series of federated rules                       | Planned but not yet implemented       |
-| Which tenants to query from for federated rules                          | Planned but not yet implemented       |
+| Allow federated rules behind feature flag                                | Implemented                           |
+| Allow federated rules only for select tenants                            | Implemented                           |
+| Where to store resulting series of federated rules                       | Implemented                           |
+| Which tenants to query from for federated rules                          | Implemented                           |

integration/e2ecortex/client.go

@@ -818,12 +818,15 @@ func (c *Client) GetRuleGroups() (map[string][]rulefmt.RuleGroup, error) {
 
 // SetRuleGroup configures the provided rulegroup to the ruler.
 func (c *Client) SetRuleGroup(rulegroup rulefmt.RuleGroup, namespace string) error {
-	// Create write request
 	data, err := yaml.Marshal(rulegroup)
 	if err != nil {
 		return err
 	}
+	return c.SetRuleGroupYAML(data, namespace)
+}
 
+// SetRuleGroupYAML configures the provided YAML encoded rulegroup to the ruler.
+func (c *Client) SetRuleGroupYAML(data []byte, namespace string) error {
 	// Create HTTP request
 	req, err := http.NewRequest("POST", fmt.Sprintf("http://%s/api/prom/rules/%s", c.rulerAddress, url.PathEscape(namespace)), bytes.NewReader(data))
 	if err != nil {