by Karen Feigh, Georgia Institute of Technology, [This article first appeared in The Conversation, republished with permission]
By now, almost everyone has asked an AI agent a question, only to realize that the answer seems not quite right. Something might be missing or wrong or out of date. An executive assistant told me she uses AI tools for drafting itineraries and briefings, but she’s caught enough issues that she now builds time into her workflow to thoroughly check for inaccuracies and biases.
Therein lies the catch: AI responses can improve productivity, but the gains are often countered by time and mental energy spent determining whether the output is accurate and appropriate. This is the verification burden.
At a deeper level, the burden is the cognitive cost of trying to regain authority over work that is partly delegated to a system, while retaining responsibility for the final product. Cognitive engineer David Woods described this problem back in 1985 as the responsibility-authority double bind: Automation exercises practical authority, yet responsibility remains with the human.
I am a cognitive engineer, and I study how people work with automated decision-support systems. Generative AI verification is the latest instance of a long-standing human-factors problem: a new kind of necessary work brought about by use of a new technology.
AI workers are exchanging some skills related to creating and transforming information for separate skills that resemble supervising an extraordinarily fast and polished intern. The difficulty is that this intern delivers both good and bad work in the same confident voice, provides few clues about where scrutiny is needed, and bears no responsibility for the consequences.
Verifying instead of creating
Research shows that verifying new information is a different task than creating it, and it’s not a simpler one. A meta-analyis of 86 studies revealed that people remember what they produced far better than anything they only read. The same is true for understanding information. Research also shows that material that is easy to read and understand seems more credible, and people therefore consider it more likely to be true.
But humans do not always vet AI responses. Instead, they tend to weigh their own internal costs and benefits to decide when to check. Overreliance on AI output is hard to stop, too. One research team found that adding simple explanations of the AI’s reasoning did not eliminate the quandary people felt in committing the time and energy needed to check. People tend to generalize agents as either capable or incapable and nothing in between.
Economics scholars have found real productivity improvements from using generative AI in office work. But a randomized trial of experienced software developers found no gains, and completion times increased when a verification step was included. Surprisingly, the developers believed they had been faster when using AI, yet their completion time increased by 19%.
And a large study of consulting teams found that using AI outside of a person’s area of expertise resulted in more people reaching the wrong conclusion.
These studies do not establish that there is a universal verification “tax,” but they show why the speed alone of generating output is an incomplete measure of productivity.
Why verification is hard
Verification can feel burdensome in terms of cognitive load, time spent and motivation.
People feel a cognitive burden because verification is fundamentally different from creation. You generally know when something you are creating is “done,” or is at least sufficient for the task. When to stop verification is less obvious. A worker must decide what to check and how; they do not know whether an error exists, where it might be or what form it might take.
This is similar to airport security screening, where an unusual threat gets missed more often than a common one because it is easy for inspectors to conclude that rare threats are not present. This burden is greater when AI helps someone generate work that exceeds their ability to evaluate it. A recent study found that AI reasoning helped people catch mistakes on easy questions but not hard ones, when help was needed most.
A time burden arises because instead of ending work, finding an error starts another cycle of correction and then rechecking. Research into how people evaluate AI-generated suggestions has found that the more work needed to fix an identified mistake, the less likely people are to actually fix it.
Finally, verification carries a motivational burden. Creation produces visible progress, but verification may find only that nothing was wrong. When automation usually performs well, checking can feel unproductive, and people may gradually become less vigilant. This pattern was found two decades ago in a study of people working with automated decision-making systems.
Skill swap
Together, these burdens result in a skill swap. In a recent survey, knowledge workers said their critical thinking was moving from production toward verification. Workers also need to learn how to recognize the capabilities and limits of a specific AI agent, including omissions and biases.
These skills are different, but they are not necessarily easier, and they might not even be in the skill set of workers who excel at creation. Companies and organizations therefore cannot treat verification as a natural step that carries no effort. If workers remain accountable for AI-assisted output, I believe that they must also be given the time, training and authority needed to verify it.
The measurement of productivity gains cannot include just how quickly AI produces a first draft. Instead, the measure is whether the entire human-AI workflow produces trustworthy work at a lower total cost.
Karen Feigh, Professor of Cognitive Engineering, Georgia Institute of Technology
This article is republished from The Conversation under a Creative Commons license. Read the original article.
