Prometheus relabeling is one of those topics that looks straightforward in the docs and reveals its complexity over months of production use. The official documentation explains the mechanism — relabeling rules, regex, actions like replace, keep, drop, labelmap — but it doesn't tell you which patterns keep your cardinality manageable at scale, or which ones silently create label explosions that you won't notice until query latency degrades on your most important dashboards.
This is a collection of patterns we've settled on after instrumenting services at scale, along with specific mistakes that are easy to make and expensive to clean up.
The cardinality problem, stated plainly
Prometheus stores data as time series. A time series is uniquely identified by a metric name plus a set of label key-value pairs. The number of unique time series is your cardinality. Cardinality determines your memory usage, your ingestion rate, and your query performance.
The danger in microservice environments is that each new service adds labels, and if those labels have high cardinality (many distinct values), the number of time series multiplies rapidly. The classic example is an HTTP request label that includes the full path: path="/api/users/12345/orders" — where 12345 is a user ID. Every user creates a new label value, which creates a new time series. In a system with millions of users and thousands of request paths per service, this pattern can generate hundreds of millions of time series and make Prometheus non-functional.
Relabeling is your primary mechanism for managing this. Understanding the two relabeling phases and where each applies is the first step.
Relabeling vs metric relabeling: know the difference
Prometheus has two relabeling phases that most guides conflate:
relabel_configs applies during target discovery, before scraping. Use this to decide which targets to scrape, and to set labels derived from the target metadata (pod annotations, Kubernetes labels, service names). This is where you set service, namespace, cluster, and other infrastructure labels.
metric_relabel_configs applies after scraping, before storage. Use this to manipulate the metrics themselves — dropping metrics you don't need, aggregating label cardinality, renaming label keys, dropping high-cardinality labels before they reach the TSDB.
The critical rule: drop high-cardinality labels in metric_relabel_configs, not in relabel_configs. By the time you're in metric_relabel_configs, you've already scraped the data — you're just filtering what gets stored. In relabel_configs, you're deciding which targets to even scrape. Mixing these up means you might be dropping targets you wanted to keep, or keeping metrics you wanted to drop.
Pattern 1: normalize service names from Kubernetes pod labels
When Prometheus scrapes Kubernetes pods via service discovery, it exposes a large set of __meta_kubernetes_pod_* labels. You need to turn these into stable, consistent labels for your metrics. The most important is the service name.
The pattern that works for most Kubernetes setups:
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: service
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_label_version]
target_label: version
- regex: __meta_kubernetes_pod_label_(.+)
action: labeldropThe final labeldrop rule removes all remaining Kubernetes pod labels after you've explicitly promoted what you need. Without it, every pod label becomes a Prometheus label, which means every label change in Kubernetes (often done by automation) affects your cardinality. Labeldrop is your fence — only what you explicitly promote with target_label survives into your metrics.
Pattern 2: drop high-cardinality path labels at the boundary
HTTP server metrics from most frameworks expose a path or route label. Frameworks like Express, FastAPI, and Spring typically normalize route templates — /api/users/:id/orders rather than /api/users/12345/orders — but not all do, and even when they do, you need to verify. One service in your fleet that passes raw path values will create a cardinality explosion that gradually degrades Prometheus query performance for all dashboards.
The defensive pattern in metric_relabel_configs:
metric_relabel_configs:
- source_labels: [__name__, path]
regex: 'http_request_duration_seconds;/api/[^/]+/[0-9]+(/.*)?'
target_label: path
replacement: '/api/<resource>/<id>/...'
action: replaceThis is a catch-all that normalizes paths matching a numeric ID pattern. It's not elegant, and ideally your application framework handles this at the source. But at 50+ services maintained by different teams with different frameworks, a Prometheus-level safety net is worth the complexity.
Pattern 3: keep rules before drop rules
A common mistake is writing drop rules first and keep rules after. In Prometheus relabeling, rules are applied sequentially and the action of a drop rule terminates processing for that target or series. If you write a broad drop rule before a specific keep rule, the drop fires first and the keep never runs.
The correct pattern: write all keep rules first, then drop rules. For metric relabeling, the typical structure is: keep the metrics you need (using the keep action on __name__), then drop specific high-cardinality labels, then apply any normalization replacements.
If you need to keep only specific metrics from a high-cardinality exporter (like node_exporter, which exposes several hundred metrics), an explicit keep list reduces storage dramatically:
metric_relabel_configs:
- source_labels: [__name__]
regex: 'node_cpu_seconds_total|node_memory_MemAvailable_bytes|node_filesystem_avail_bytes|node_network_receive_bytes_total|node_network_transmit_bytes_total'
action: keepThis kind of explicit keep list can reduce per-node time series from 800+ (all node_exporter metrics) to under 50, with no meaningful loss in alerting or dashboarding capability for most use cases.
Pattern 4: version label cardinality control
Exposing a version label on your service metrics is useful — it lets you compare performance across deploys, which is exactly the kind of context that makes incident investigation faster. The risk is that version labels need to be stable: if your versioning scheme uses git SHA hashes or build timestamps, you'll get a new time series for every deploy, and the old series will age out but not before inflating cardinality during their retention window.
The pattern: use semantic version strings (v2.4.1, not 8f3a1c2) as your metric version label. If you want to also track the commit SHA for debugging, expose it as an info metric (service_build_info) with a count of 1 and the SHA as a label, rather than embedding it in all metrics. The info metric approach (a Prometheus convention) lets you join against the SHA when needed without inflating cardinality across your operational metrics.
What we actually do with these labels at query time
The reason these patterns matter beyond operational hygiene is that they directly affect what's possible when you're doing automated root cause analysis. When Devloom correlates a metric spike with a deployment event, it does so by querying Prometheus for metrics labeled with the affected service name and version, filtered by the timestamp window around the deploy. If your service label isn't normalized consistently — if half your services use service and half use app or app_name — that correlation breaks silently.
We're not saying every team needs to adopt our labeling scheme. The point is that your labeling scheme needs to be consistent and documented, and the enforcement mechanism is relabeling configs that normalize at the Prometheus layer rather than trusting every service team to instrument the same way.
The relabeling config is infrastructure-as-code. Treat it that way: version it, review it, and audit it when you add new services. It's much easier to maintain consistency when you have 55 services than when you have 120 and the inconsistencies have been accumulating for two years.
