Why We Attack Our Own Tools Before We Trust Them With Yours
Before we hand a piece of software to a client -- or build it as a product of its own -- we try to break it ourselves first. Here's what that actually looked like this week.
We spent this week doing something that doesn't show up in a feature list: trying to break our own site-audit engine, on purpose, repeatedly, before trusting it with anyone else's website.
It went through three separate passes, each looking for a different class of problem.
Pass one: can this be attacked?
The engine fetches URLs supplied by whoever's running an audit, which means it inherits every risk that comes with fetching arbitrary URLs on a server. We went looking specifically for SSRF: ways a malicious or careless input could make the server reach somewhere it shouldn't. We found three real gaps (redirects, linked stylesheets, DNS rebinding) and closed them.
Pass two: is the fix actually correct?
Having fixed the security gaps, we went back over that exact fix looking for holes in it, and found one: a way of writing an IPv6 address that our own check didn't recognise, because of how URL parsers silently rewrite it. Confirmed with real tests, not assumptions, and fixed properly.
Pass three: is the output honest?
Security isn't the only way a tool can fail you. We went through the scoring logic itself looking for false positives and false negatives: cases where the report would claim something about a site that isn't true. We found three: two common, everyday phrases being misread as fake trust signals, and one dead check meaning real social media links were being missed entirely.
Why bother writing this up
Every one of these was a real, previously-shipped bug, not a hypothetical. We're publishing the details, not just the fact that we fixed something, because we think it's more useful, and more honest, to show the actual process than to just claim a tool is "rigorously tested." Anyone can say that. Fewer people show the three separate passes it took, the specific bugs each one found, and the tests written to make sure they stay fixed.
This is the same engine behind our Site Health Check: the tool that looks at a prospective client's site and tells them, plainly, what's working and what isn't. It only gets to make that kind of claim about someone else's website because we held it to the same standard first.