Site Reliability Engineer

An SLI is the actual measurement, something like the percentage of requests that return successfully within 300 milliseconds. An SLO is the internal target we set for that SLI, say 99.9% of requests meet that threshold over a rolling 30 day window. An SLA is different in kind, not just degree: it is an external, often contractual commitment to a customer, usually with financial or reputational consequences if missed, and it is typically set looser than the internal SLO to give the team margin. I treat the SLO as the number that actually drives day to day decisions, and I reserve the SLA conversation for what we tell customers and legal, since conflating the two leads teams to either over-invest in reliability nobody is paying for, or under-invest and risk breaching an actual contract.

The error budget is simply 1 minus the SLO, expressed as an amount of acceptable failure over a time window. If our SLO is 99.9% over 30 days, the error budget is 0.1%, which on a high traffic service translates into a concrete number of minutes of downtime or failed requests we can spend. I use it as a decision-making tool, not just a reporting metric: when the budget is healthy, the team has room to ship faster and take on more risk, including riskier releases or infrastructure changes. When the budget is close to exhausted, we shift focus to reliability work and slow down or pause new feature releases until it recovers. The value of this framing is that it turns "is this safe to ship" from a subjective argument into a shared number both engineering and product can look at and agree on.

I treat the error budget as the mechanism that resolves this tension rather than trying to solve it through negotiation meetings. As long as the budget is healthy, product and engineering can ship at full speed, since the data shows the service can absorb that level of risk. Once the budget starts running low, that becomes an automatic, pre-agreed signal to prioritise reliability work: fixing the root causes behind recent incidents, paying down toil, or hardening a fragile dependency, before more feature risk gets added. I also track toil separately, time spent on manual, repetitive operational work, since high toil quietly erodes velocity even when the SLO looks fine on paper. Getting this balance right depends on setting the SLO with rigour in the first place: an SLO set too strict burns the whole team out chasing five nines nobody actually needs, and one set too loose lets real reliability problems build up unnoticed.

I use observability tools with anomaly detection to flag unusual patterns in latency or error rate before they cross an alert threshold, which buys the team lead time on emerging problems. During an active incident I have used AI-assisted log summarisation to pull the relevant lines out of a huge volume of noise quickly, which speeds up the first ten minutes when everyone is trying to understand scope. Some of our runbooks now have AI-suggested next steps based on the alert that fired, though I always treat those as a starting hypothesis, not an instruction to follow blindly. What stays fully human is the incident commander role and the postmortem analysis: deciding what actually happened, what contributed, and what changes are worth making requires judgment about organisational context and trade-offs that the tooling does not have visibility into.

Our primary database experienced a connection pool exhaustion issue that took checkout down for about 22 minutes during a promotional traffic spike. I was the incident commander: my first move was to declare the incident formally and pull in a database specialist and a backend engineer rather than trying to debug it alone. We mitigated first, not root caused first: we scaled the connection pool limit and restarted the affected service instances, which restored checkout within about eight minutes, then spent the rest of the incident confirming stability. I posted a status update every ten minutes in our incident channel, even when there was nothing new, because stakeholders checking silently is worse than a repetitive "still investigating" update. Once service was stable, I handed off root cause investigation to the database specialist and closed the active incident, since keeping an incident open past mitigation just adds process overhead without helping users.

After a deployment pipeline failure caused a two hour delay in a critical security patch, I ran the postmortem and set the ground rule at the start: we are naming the contributing factors in the system and process, not the person who clicked deploy. We built a timeline together as a group rather than me presenting one, which surfaced a detail I would have missed: the alert that should have caught the failure had been muted three weeks earlier during an unrelated debugging session and never unmuted. Every action item got a named owner and a due date in the same meeting, not left as a vague follow-up, and we reviewed the action items from the previous postmortem first to confirm they had actually been completed. The postmortem worked because nobody spent energy defending themselves: the engineer who had muted the alert volunteered that detail himself once it was clear the room was looking for causes, not someone to blame.

Product wanted to ship a major checkout redesign two weeks before our highest traffic period of the year, and our error budget for that service was already down to about 15% remaining for the month after a rough few weeks of incidents. I did not simply say no. I brought the error budget data to the planning meeting and framed it as a shared risk decision: shipping now, with an untested code path, during peak traffic, with almost no budget left, was a specific quantified risk, not a vague worry. I proposed shipping the redesign to a small percentage of traffic immediately, watching the SLO for a week, and rolling it out fully only if it held, rather than a full launch right before the peak. Product agreed once the risk was framed in numbers instead of caution, and the phased rollout caught a memory leak under load that would have caused a real incident during peak traffic if we had shipped fully on the original timeline.

I anchor metrics around the four golden signals: latency, traffic, error rate, and saturation, since together they cover most of what tells you a service is healthy without drowning you in noise. For a single service those come from metrics, but the moment you are dealing with a call chain across multiple services, metrics alone stop being enough, and that is where distributed tracing earns its cost: a trace shows you exactly which hop in a request added the latency instead of just telling you the aggregate got slower. Logs I keep structured, with consistent fields like request ID and service name, specifically so they can be correlated with a trace during an incident instead of grepped through manually. I am deliberate about sampling on high volume services, since capturing 100% of traces gets expensive fast and rarely adds value over a well-designed sample rate combined with full capture of anything that errors.

I define toil as work that is manual, repetitive, tactical, and scales linearly with service growth rather than providing lasting engineering value, things like manually restarting a stuck job or hand-editing a config for every new customer onboarded. The first step is measuring it with precision: I track how much on-call time actually goes to toil versus genuine incident response, since teams consistently underestimate this until they log it for a few weeks. Anything that recurs more than a handful of times gets a clear test: can this be automated with reasonable engineering effort, and does removing it free up enough time to justify that effort. I have automated things like certificate renewal and stale resource cleanup this way, which took real toil off the on-call rotation permanently rather than just making each instance slightly faster. I also protect time on the roadmap specifically for this work, since toil reduction always loses to feature pressure if it is not explicitly planned for.

I start with the actual growth number product is expecting, not a vague 'a lot more traffic,' and I convert that into a specific request rate target. I look at historical growth patterns for similar past events to sanity check whether the projection is realistic, then run load tests against a staging environment configured to match production as closely as possible, since testing against an undersized environment gives false confidence. I set a headroom target, typically planning for meaningfully more than the projected peak, not just enough to exactly meet it, because real traffic is spikier than any smooth projection suggests. I check autoscaling limits specifically, since a service can have plenty of theoretical capacity but still fail if the maximum instance count on the autoscaler or the connection limit on a downstream dependency caps it below what is needed. Cost is part of the conversation too: I present a plan with the headroom I recommend and the cost of that headroom, so the decision to accept more risk for lower cost is made explicitly by the business, not by default.