The Devloom blog.
Causal inference, incident triage patterns, OTEL pipeline design, and the occasional opinion on why most observability tooling optimizes for dashboards instead of answers.
Shipping Devloom 0.9: what we built and what we got wrong
Six months of building, three major architecture pivots, one very long on-call shift. What it took to ship a causal inference engine that actually works at production load.
The on-call engineer experience in 2026: what's changed
We surveyed 140 engineers about their on-call experience. The tooling has gotten better. The mean time to innocence has not.
Kubernetes deployment rollbacks: when to automate and when not to
Automated rollbacks sound great until one fires at 3am on a microservice that has 40 dependents and a shared database schema migration in flight.
Finding latency bottlenecks with Jaeger: a real case study
A payments service was 200ms slower than the SLA. Here's how we found the culprit using trace waterfall analysis — and what the diff looked like when we fixed it.
Remediation runbooks that actually work
Most runbooks are documentation that was wrong six months ago and nobody updated. Here's a structural approach that keeps runbooks useful under pressure.
Loki log aggregation patterns for microservice architectures
Loki is cheap and fast. It's also easy to set up in a way that makes querying during an incident painful. Here are the labelling patterns that actually hold up under load.
Causal inference vs correlation in SRE workflows
Every observability tool does correlation. Almost none do causal inference. Here's what the difference looks like in practice — and why it matters at 2am.
PagerDuty alert fatigue: causes and solutions
Alert fatigue isn't a PagerDuty problem. It's a signal-to-noise problem. Here's how to audit your alert configuration without a complete rewrite.
How to map blast radius in distributed systems
You've found the root cause. Now what? Understanding which downstream services are degraded before they page you is the difference between a 15-minute incident and a 90-minute one.
Prometheus relabeling patterns for 50+ microservices
At a certain service count, Prometheus relabeling config becomes the scariest file in the repo. Here's the pattern that scales without becoming unreadable.
Mean time to innocence: the real incident metric
MTTR tells you when the incident ended. Mean time to innocence tells you how long your on-call spent ruling out services they didn't break. That's the number worth reducing.
Building an OTEL-native observability pipeline
OpenTelemetry is the right long-term bet. Here's how to build a pipeline that uses OTLP end-to-end — without getting locked into a collector topology you'll regret in 18 months.
Why alert correlation is not root cause analysis
Alert correlation groups symptoms. Root cause analysis finds causes. They're different algorithms, different outputs, and different amounts of useful in a live P1. Here's the technical distinction.