From Evaluation · · 1m 20s
Prompts Aren’t Real
Agents fail without measurement, not prompts.
In brief
Agents fail without measurement, not prompts. Prompts are ephemeral; self-improving systems require measurement to avoid chaos. Originally reported by Evaluation.
I’m Dan
I’m an engineer in Los Angeles with 25 years experience.
Agents in production
I build agents that users trust to perform tasks.
My goal
I want a reliable agentic experience without embarrassment.
My motivation
I’m driven by shame, not satisfaction with subjective advice.
The fun
Building agents feels magical and addictive.
The risk
Agents can collapse psychologically while being fun.
The problem
LLMs fail at simple tasks despite seeming reliable.
The fix
Renaming fields from title to heading fixes JSON errors.
The pattern
Prompts don’t work well when scaled across contexts.
The solution
Run pass^k tests to measure prompt effectiveness.
The optimizer
Genetic pareto optimizes prompts automatically.
The test suite
Holdout tests prevent overfitting to examples.
The brand voice
Brand voice requires complex judgment, not simple prompts.
The flywheel
Self-improving systems need measurement, not prompts.
The danger
Unmeasured prompts lead to chaos and madness.
The conclusion
Prompts are ephemeral; measurement is the real solution.
The short version
Prompts are ephemeral; self-improving systems require measurement to avoid chaos.
