Fixing Prometheus High-Cardinality Metrics
Unbounded labels like user and property IDs inflate Prometheus metrics. I removed them, kept bounded dimensions, and fixed the design instead of changing platforms.
Monitoring can become a problem of its own. On RE/AMP, my Prometheus metrics grew in a way that made dashboards harder to use, and the cause was a design mistake.
What cardinality means
Every unique combination of label values in Prometheus creates its own time series. A label with a few possible values, like HTTP status, adds a few series.
A label with an unbounded number of values, like a user ID, adds a new series for every user. That is high cardinality, and it grows with your data.
What I had done
I had added labels that were useful when debugging a single case: user IDs, property IDs, and cohort dates. Each seemed informative on its own.
Together they produced far more series than any dashboard needed.
What I changed
Remove unbounded labels
I took out the user, property, and cohort labels. They belong in logs and event tools, not in metrics.
Keep bounded dimensions
I kept labels with a limited, known set of values: the route, the response status, and the company. Those answer the questions a dashboard needs to answer.
Rebuild dashboards around aggregation
I rebuilt the Grafana dashboards with aggregated PromQL queries. Functions such as rate, increase, histogram_quantile, sum by, and topk show the busiest routes or companies without drawing every series.
The platform question
During this work I looked at Datadog as an alternative. Its custom metrics pricing is driven by the same cardinality, so a platform switch would have carried the problem with it, and added cost.
The fix belonged in the metric design. I kept Prometheus and Grafana.
A rule for next time
Decide on label rules before instrumenting anything. A short rule helps: a label is allowed only if its values are bounded and a dashboard actually uses it.
Takeaway
When monitoring gets heavy, look at what you are measuring before you look at where you are storing it. High cardinality is a design problem that moves with you.
If you need monitoring that stays useful, see the code rescue service, or read how this fits into the RE/AMP case study.