# FLAWED's Research Flaws and Industry Impact > 1Password's flawed security research risks industry standards. - Title: FLAWED's Research Flaws and Industry Impact · Suhacker - Summary: 1Password's flawed security research risks industry standards. 1Password's security research paper contains critical errors that undermine industry research… - Keywords: 1password, off-by-1 labs, ai security, research norms, patch the planet, flawed, industry research, academic research, security research, ai patching, technology, startups - Source: Suhacker — https://suhacker.ai/p/flaweds-flaws-and-what-this-means-for-industry-research - Published: 2026-09-22T00:00:00+00:00 - Read time: 2 min - Topics: 1password, off-by-1 labs, ai security, research norms, patch the planet, flawed, industry research, academic research, security research, ai patching, technology, startups ## FLAWED's research flaws On September 17th, I quoted Trail of Bits's blog post criticizing 1Password's FLAWED security research paper for misleading claims and insufficient citations. > 1Password's AI patching benchmark is misleading ## Citation issues in FLAWED FLAWED cites only 19 sources, mostly corporate blog posts and XKCD, while the concurrent PatchBench paper has 73 academic citations. > FLAWED adopts the form and rhetoric of rigorous research (even claiming there is "very little prior work") ## Missing academic citations FLAWED omitted critical academic papers, including a NDSS paper on LLM security research pitfalls. > FLAWED did not cite and did not read this NDSS paper about pitfalls in LLM security research ## Patch the Planet inaccuracies FLAWED incorrectly attributed vulnerability triage efforts to ToB engineers instead of Calif and HackerOne. > Calif and HackerOne participated in "vulnerability triage, coordinated disclosure, and additional focused vulnerability discovery efforts" ## Research norms at stake The paper's errors threaten research integrity by prioritizing corporate distribution over academic rigor. > Research should not become an influence operation ## Industry research standards FLAWED uses the phrase "peer review" to describe review by three industry peers, not scholarly peer review. > Our assessment does not meet the rigors of scientific research and should not be taken as such ## Consequences of flawed research Dishonest research distorts research agendas, wastes labor, and pollutes the epistemic environment. > It distorts research agendas. Work that seems duplicative or incorrect according to prior research is often not funded ## Call for corrections 1Password should issue a retraction or correction, partner with academic researchers, or hire research experts. > I hope that 1Password issues a retraction or a correction ## Research integrity needed The research community needs stronger norms to prevent corporate influence on security research. > We should care about OSS security ## OSS security funding The author seeks evidence on which interventions improve open-source software security with limited resources. > Given a fixed budget, which interventions work best: hiring more maintainers, providing existing maintainers with token spending ## Research should not be sports Research requires healthy skepticism, not the competitive approach of sports. > Research is not sports. OpenAI is not the Spurs, 1Password is not the Knicks ## Public corrections needed Public claims require public corrections to maintain research integrity. > A public claim should have a public correction ## Industry research risks FLAWED's errors could lead to misleading security claims and reduced research quality. > This opens the door for malicious actors that could repeatedly select conclusions that flatter their institutional or personal interests ## Academic research alternatives EleutherAI and the AI Now Institute provide rigorous security research that 1Password should fund. > EleutherAI regularly publishes precise work in this realm ## Research community response CISOs, academics, and industry researchers noted similar issues but lacked a forum to raise them. > Afterward, CISOs, academics, industry researchers, and practitioners alike messaged me to say they had noticed the same problems ## What happens next 1Password must correct FLAWED or the research community will face weakened security standards. > I hope that 1Password issues a retraction or a correction ## Key takeaway 1Password's security research paper contains critical errors that undermine industry research norms and require corrections.