From Fastcompany · · 1m 40s
When AI fails, can humans still do the job?
As AI systems take over more work, employees may lose the skills to step in when things go wrong.
In brief
As AI systems take over more work, employees may lose the skills to step in when things go wrong. As AI assumes larger roles in organizational decision-making, employees overseeing it risk losing the capabilities they need to intervene when systems fail. Originally reported by Fastcompany.
The reliable AI system fails
An AI system performs reliably for months, its recommendations almost always correct. Then something unusual happens and it makes an error or cannot proceed, needing human intervention.
The employee must step in
The employee reviewing the AI is expected to intervene and remains responsible for the outcome. But the critical question emerges: does the employee actually know how to do the task?
Latent risk explained
A professor of management information systems at Drexel University calls this latent risk: the possibility that people may lose capabilities their organizations still depend on them to exercise.
Skills fade while AI performs
AI can continue to perform well while the people overseeing it become less practiced at exercising independent judgment and less prepared to intervene when needed.
The hidden organizational danger
The risk becomes latent when the organization still depends on that human capability to step in but has little cause to notice that capability may be weakening over time.
Automation bias is not new
The concern builds on decades of research into how people respond to automation. Studies of automation bias show people can miss problems or follow automated recommendations even when evidence suggests otherwise.
What is different now
Today AI is beginning to assume larger parts of the analysis and decision-making that people previously performed, making the stakes of skill loss significantly higher for organizations.
The main idea
As AI assumes larger roles in organizational decision-making, employees overseeing it risk losing the capabilities they need to intervene when systems fail.
