Actually Works · Episode 04
35 / 38This video almost shipped broken
Three real defects reached us before a person caught them, on our own pipeline. None was caught by a tool. That's why every check here now checks the check before it.
There's a part 2: A check said it passed. It lied. — the fix for exactly this.
The exact clicks
- Before trusting any automated check, feed it a case you already know is broken.
- Confirm the check actually flags that known-bad case — not just that it runs without error.
- Only then trust it on real, unknown cases.
- Re-run this test after any change to the check itself, not just after changes to what it checks.
What changes
- A level check measured the wrong thing — it read a position in the file, not whether the sound had actually landed.
- A check ran before the fix it was supposed to verify, so it always passed.
- A script proved a file existed by checking its path, not its contents — the file was empty.
- None of these three were caught by a tool. A person listening, reading, and checking by hand caught all three.
What it will not do
- Testing a check against a known-bad case does not guarantee it catches every bad case — only the ones like the one you tested.
- This is a discipline, not a one-time fix. The three defects here happened after checks already existed; the checks just weren't checked.
Get the next one
One AI setup a week, straight to your inbox.