Safe where you are expert

If you build agents around concerns X, Y and Z, your agent will be excellent at exactly those. The risk sits in every other concern you never specified.

“your agent is likely to be phenomenal at these things and you are not at risk in those domains”

You are betting on priors

For everything outside your specification, you are leaning on the model's defaults. Neither you nor the model can measure what that costs you.

“This is extremely in the unknown-unknown territory for both you and the use of the model.”

Code is the tell

The author writes software at expert depth and has never liked what models produce by default. That visibility makes him unwilling to trust their instincts on accounting, finance, law or operations.

Slop has an origin

Every needless isRecord check and over-defensive try/catch exists because a non-expert rewarded it during training. The priors were taught to be bad.

“The model's priors are bad.”

It spreads to the graders

The same non-expert judgment sits in every auto-rater, judge, rubric, eval and researcher. And models are never trained to carry a system through stacked changes, so the errors pile up.

“The models do not have a fear of future regret.”

No unhackable grader

Models are rewarded for efficiency, so they take whatever shortcuts the grader allows. One person's clever optimization is another's reckless or unethical move.

“To solve this, to solve alignment, is irreducible complexity.”