Building Reliable Systems

Episode Focus: Moving beyond infrastructure to build reliability directly into software, including application code and databases. Discusses cultural shifts, proactive design, and justifying reliability work.

Guests:

Core Challenges Discussed:

  1. Databases as Single Points of Failure (Historically):
    • Relational databases can be complex to make reliable.
    • Historically, DB reliability often fell to a single DBA ("witch in the woods," "human single point of failure"), creating bottlenecks and burnout.
  2. Complexity & Hidden Unreliability:
    • Interfaces (APIs) can obscure underlying system complexity and unreliability. Simply putting an API in front of a mess doesn't solve the mess.
    • Software and the world it runs in are inherently complex; failure modes are not always obvious.
  3. Prioritizing Reliability Work:
    • Justifying reliability improvements ("feature work" for reliability) against new feature development can be difficult.
    • Shifting from reactive (fixing after outage) to proactive investment in reliability.
  4. Cultural Inertia & Mindset:
    • Teams may not always be primed to think about failure modes or the necessity of proactive reliability work ("happy path programming").
    • The idea that "incidents happen to us" vs. "we can design to mitigate incidents."
  5. *Understanding System Behavior:
    • It's hard to predict all failure modes, especially cascading failures.
    • Teams may lack a holistic view of their system and its dependencies.

Key Solutions & Concepts Presented:

  1. Cultural Shift in Engineering (especially for Databases - Silvia):
    • Move from a single "DBA hero" to team ownership of database reliability.
    • Predict failure and plan accordingly (e.g., don't let a single DB failure take down the whole product).
    • Uneven distribution of this shift; managed DB services help but require different thinking (less direct control, more reliance on the service's reliability features).
  2. System Simplification & Complexity Management (Niall):
    • Actively work to reduce system complexity.
    • "Turn things off": Deprecate and remove old, unused, or low-value/high-cost components rather than just hiding them behind an API.
    • Leverage research (e.g., Microsoft's study on feature impact: 1/3 positive, 1/3 neutral, 1/3 negative) to argue for simplification over marginal new features.
  3. Data-Driven Prioritization & Justification (Both):
    • DORA Metrics & "Accelerate" (Silvia): Use metrics like Change Failure Rate to quantify "squishy feelings" (e.g., team fear of deployments) and demonstrate the impact of reliability (or lack thereof) to leadership.
    • Cost/Benefit Analysis (Niall): If the cost of outages (or maintaining a complex, unreliable feature) outweighs the cost/benefit of fixing or removing it, make the data-driven decision.
  4. Proactive Design for Reliability (In-Code & System Level - Both):
    • "Paranoid Planning" (Silvia): Assume failure; think beyond the happy path.
    • Asynchronous Non-Critical Operations: Logging, metrics, and other non-core path operations should not be synchronous and block critical user paths.
    • Effective Traffic Management (Niall):
      • Rate Limiting, Prioritization, Load Shedding (especially client-side): Crucial, often underutilized tools to prevent overload and cascading failures.
      • Client-side load shedding is more efficient than server-side dropping after deserialization.
    • Formal Methods (Niall): Tools like TLA+ (Hillel Wayne's work) for formally reasoning about system states, pre/post-conditions can help cut off entire classes of failures by reducing state space complexity.
    • Tabletop Exercises (Silvia): Explore hypothetical failure scenarios (e.g., "What breaks if traffic 10x's?") to identify weaknesses without live testing.
  5. Understanding and Building Customer Trust (Niall & Silvia):
    • Reliability directly impacts customer trust. Don't wait for customers to leave due to repeated outages.
    • Proactively addressing reliability issues builds and maintains this trust.
  6. Learning from Patterns (Silvia):
    • Learn from your own team's incidents and other people's experiences/drama to identify common failure patterns and proactively address them.
    • Aim to learn from patterns (the "second way") rather than solely through direct, traumatic incident experience (the "first way").

Highlights (Key Takeaways for Engineers):

  1. Reliability is People + Tech + Culture: It's not just about tools or heroic individuals; it's about shared ownership, proactive mindsets, and engineering culture.
  2. Proactive > Reactive: Design for failure, anticipate issues, and plan for mitigation. Don't just wait for outages to drive improvements.
  3. Complexity is an Enemy: Actively simplify systems. Don't just hide complexity behind new interfaces; consider deprecation.
  4. Data Makes the Case: Use metrics (DORA, cost of outages, feature ROI) to justify reliability work to leadership.
  5. Beyond the Happy Path: "Paranoid planning" and considering edge cases/failure modes during design is crucial.
  6. Interfaces Can Deceive: Understand the full stack and external dependencies; an API doesn't magically make an unreliable component reliable.
  7. Learn from Patterns: Study your own incidents and those of others to improve. The goal is to learn from patterns (the "second way") not just from your own painful experiences (the "first way").
  8. Core Reliability Primitives in Code:
    • Check return codes.
    • Make non-critical operations (logging, metrics) asynchronous.
    • Implement robust traffic management (rate limits, client-side load shedding, prioritization).

Resources Mentioned: