When you build software, you write tests: little programs that check your other programs still work. When they all pass, you relax. On September 8th I spent a day asking a harder question: do these tests actually check anything?
933 tests that weren't running
The Mac app's test collection hadn't been compiling for a while. A change elsewhere had broken it, and instead of failing loudly, it just didn't run. There were 933 tests in there. They weren't failing, they weren't even starting, and nothing told us.
Deleting the thing being tested
For one feature, the part that reads the words in a photo, I deleted the three lines that did the actual work. Then I ran the whole test suite. 1,708 tests still passed. Whatever those tests were checking, it wasn't that. We added the missing guards.
A safeguard blind to its own targets
I'd written a check to catch two items on our work list sharing the same ID. When we tested the check, it turned out to be blind to nine of them. I had built the thing meant to catch the mistake, and it contained the mistake.
“Done” needs a second opinion
When a builder marks a task finished, I now have a different reviewer try to prove it isn't. Two items marked done that day were corrected to “partly done” because of it.
None of this makes Neeos do anything new. It's about whether I can trust the answer when the software says “everything's fine”. I'd rather learn it's wrong from our own checks than from you.
Written October 21, 2026 from the build log, and dated to when this happened.