Stress Free Change Validation at Netflix

How do you gain confidence that a system modification does what it’s supposed to do? A refactoring should not cause a functional change, whereas a feature modification should cause a specific kind of change. Tests are great for validating assertions one has already thought of, but for sufficiently complex systems it is infeasible or cost-prohibitive to verify the long tail of possible outcomes through testing. Without high confidence in our code changes, it is difficult to move quickly without breaking things.

I built a framework at Netflix that allows our engineers to define basic characteristics of a service API, and then replays pairs of identical requests against two versions of the service looking for differences in the responses. When changes are found, the framework filters out noise using a variety of techniques which effectively bring noise down to zero. I will share examples of how this has helped accelerate development by unlocking stress-free refactoring in my team’s most critical service, and how we’ve used it to perform large technical debt migrations in record time.

Key Takeaways

1. Why we need a high confidence change process for maintainable code bases
2. How zero-noise diffs help close the confidence gap left by tests and canaries
3. Recommended practices for building a diff system that controls for non-idempotent dependencies and (some) side effects