signalLobsters2026-09-24
FLAWED's Flaws and What This Means for Industry Research: Suha Sabi Hussain
Suha Sabi Hussain criticizes 1Password's FLAWED report for severe citation and factual errors. The report cites only 19 sources, misses prior work like PatchBench and an NDSS pitfalls paper, and incorrectly describes Patch the Planet involvement. The author calls for a retraction or correction and demands stronger research norms from industry labs.
- for who
- Researchers and industry labs producing AI security research
- what changes
- Industry research must meet academic citation standards and face accountability for errors, preventing sloppy work from shaping roadmaps.
- to do
- Demand retraction or correction of flawed industry reports and enforce rigorous research norms.
key points
- FLAWED report cites only 19 sources, mostly corporate blogs and XKCD
- PatchBench has 73 academic citations; AutoPatchBench and NDSS paper ignored
- 1Password must issue retraction or correction and partner with academic researchers
#ai security#research norms#1Password3 sources · confidence medium
score
score 6 out of 10. 0-10: how dense the facts are, multiplied by how much you can do with them after reading. 8+ means the topic's evidence bar is met: benchmarks and availability for a new model, amount and investors for a funding round, revenue figures for a solo-money story. Below 5 an item does not enter the digest. A press release scores 3 or less, a reprint loses 2, anything older than 14 days loses 1, a headline that misleads loses 3.
read the source