article
The more organizations rely on AI, the harder it may become to spot mistakes, Concordia research shows
Graphic by Vector Elements on Unsplash
Financial institutions are investing heavily in artificial intelligence (AI) to increase efficiency and scale their operations. But as more work is delegated to machines, a critical challenge emerges: how much can organizations automate before employees lose the expertise required to determine whether AI-generated results are correct?
New research from the John Molson School of Business at Concordia University examines how one Canadian investment management firm manages that tension. The study, a master's thesis by Quang Tung To in the Business Analytics and Technology Management program, was supervised by Arman Sadreddin, associate professor of Business Technology Management. It draws on interviews with employees working in risk, analytics, technology and business functions.
The research found that employees determine what work to delegate to AI not solely based on what the technology is capable of doing, but on whether they can still independently verify the output. The paradox is that the more work is delegated, the harder it becomes to maintain that verification capability.
Automation depends on the ability to verify
The study introduces the concept of a "verification period," defined as the time between receiving an AI-generated output and knowing whether it is correct. Some outputs can be confirmed almost immediately, while others, such as investment decisions, may take months or even years before their quality can be assessed. As a result, the practical limits of automation are shaped not only by technological capability but also by how confidently people can validate results.
Automation does not eliminate work
The findings also challenge common assumptions about productivity gains from AI. In one workflow described by participants as fully automated, automation cut the time the task took by only about half. The remaining time was spent on verification: checking facts, correcting errors, comparing results and deciding whether the work could be trusted.
Employees also reported investing significant effort in building, testing and refining AI tools before realizing any operational benefits. The study suggests that organizations may overestimate productivity improvements when they focus only on technical capabilities and fail to account for the ongoing work required to support and monitor automation.
The ability to assess AI can decline over time
The thesis identifies what it calls "evaluative capacity," the knowledge and judgment needed to determine whether AI-generated outputs are reliable. This capability depends on employees having enough understanding of the underlying task to recognize when something appears incorrect. That expertise is often developed through performing the work directly.
As AI assumes more responsibility for routine activities, employees have fewer opportunities to practice and strengthen those skills. Experienced workers reported becoming less familiar with tasks now handled by AI systems, while newer employees may learn to use AI tools before developing the expertise required to critically assess their outputs.
The result is a feedback loop: an organization's ability to delegate work depends on its employees' ability to verify outputs, yet increased delegation can gradually weaken that very capability.
Implications for organizations
The research suggests that the objective should not simply be greater automation, but protecting the work through which employees learn to judge AI outputs. Tasks often considered routine may serve an important developmental function by helping employees build the judgment required to assess more advanced outputs later in their careers. Eliminating junior tasks or review layers loses that experience along with the work.
The study identifies three organizational practices that can help address the issue. First, organizations should measure net time savings rather than gross automation gains, recognizing that verification work remains essential. Second, oversight frameworks should account for how long errors could go undetected, not only the complexity or potential impact of a task. Finally, responsibility for maintaining the ability to evaluate AI outputs should be formally assigned rather than left to individual initiative.
The research concludes that organizations with the greatest potential to benefit from AI may not be those that automate the most work, but those that successfully preserve the human expertise needed to recognize when automated systems are wrong.