RO / EN
Official Profiles International Partner in Romania, Bulgaria and Moldova office@globaltest.ro · +40 735 558 134

Does AI Make Employees More Competent — or Just More Productive?

New research raises an important question for recruitment, employee assessment, and the future of work: are we measuring what people know, or what they can achieve with AI? Imagine asking an employee to complete a task that requires skills they don't have. Perhaps they need to write Python code, interpret statistical data, or develop a predictive model. They have two options: use traditional online resources, such as Google and Stack Overflow, or turn to ChatGPT.

Does AI Make Employees More Competent — or Just More Productive?

New research raises an important question for recruitment, employee assessment, and the future of work: are we measuring what people know, or what they can achieve with AI?

Imagine asking an employee to complete a task that requires skills they don't have.

Perhaps they need to write Python code, interpret statistical data, or develop a predictive model. They have two options: use traditional online resources, such as Google and Stack Overflow, or turn to ChatGPT.

Which approach would produce better results?

More importantly, would completing the task mean that the employee had actually learned something?

A recent study published in Nature Human Behaviour, titled Generative AI and the Temporary Upskilling of Knowledge Workers, offers some fascinating answers.

The Experiment: ChatGPT vs. Traditional Online Resources

Researchers conducted a randomized controlled experiment involving consultants from Boston Consulting Group (BCG).

Of the 986 consultants initially allocated to the experiment, 573 began it and 487 completed the tasks included in the main analysis.

Participants were asked to solve data science problems that were largely outside their usual professional expertise.

One group received access to ChatGPT and training on how to use it. The control group received training on traditional online resources, including Google and Stack Overflow.

The participants completed tasks involving Python programming, statistical analysis, and predictive modelling.

Importantly, these were not problems ChatGPT could reliably solve independently. Human input was necessary.

The researchers also compared participants' results with those of experienced data scientists.

AI Significantly Improved Performance

The results were striking.

Compared with the control group, participants using ChatGPT achieved:

  • 49 percentage points higher scores on the Python programming task.
  • 20 percentage points higher scores on the statistical analysis task.
  • 17 percentage points better results on the prediction task, measured against the expert benchmark.

These differences refer to the study's task-specific performance measures, not general productivity increases.

AI helped participants perform complex technical tasks that would normally require expertise beyond their existing capabilities.

Some participants with prior coding experience even achieved results statistically indistinguishable from those of professional data scientists on the programming and statistics tasks.

However, the experiment revealed something even more interesting.

What Happened When ChatGPT Was Taken Away?

After completing their tasks, participants were asked to answer technical questions without access to ChatGPT.

If AI-assisted work had resulted in meaningful knowledge acquisition, we might expect the ChatGPT group to perform better.

But that wasn't what researchers found.

There was no statistically significant advantage in the technical knowledge demonstrated by participants who had used ChatGPT.

In other words, the ability to perform a task successfully with AI did not necessarily translate into the ability to perform related tasks independently.

This doesn't mean that AI prevents learning. The experiment was relatively short, and the subsequent knowledge assessment was limited.

It does, however, highlight an important distinction:

Improving someone's performance is not the same as improving their underlying competence.

The researchers describe AI as a kind of cognitive exoskeleton: a technology that allows people to perform tasks beyond their usual capabilities, without necessarily changing what they can do independently.

Another Finding: Greater Confidence, Poorer Judgement

The study also examined how accurately participants understood ChatGPT's capabilities.

Interestingly, participants who had used ChatGPT became more optimistic about what the technology could accomplish.

But they also became less accurate at recognising certain tasks that ChatGPT could not solve correctly.

This part of the research was exploratory, so the findings should be interpreted cautiously.

Nevertheless, it raises an important organisational concern.

An employee who successfully uses AI to complete unfamiliar tasks may become increasingly confident in the technology, without necessarily becoming better at identifying its mistakes.

For organisations, this suggests that AI proficiency should involve more than the ability to generate useful outputs.

It may also require the ability to question those outputs, recognise uncertainty, and know when additional expertise is necessary.

What Does This Mean for Recruitment and Employee Assessment?

For decades, recruitment and assessment practices have focused on identifying individual characteristics that predict future job performance. We evaluate cognitive abilities, professional knowledge, problem-solving skills, judgement, and behavioural competencies.

But AI introduces a new question:

Should we assess what candidates can accomplish independently, or what they can achieve using the tools available in their actual working environment?

Consider two candidates completing a business simulation.

The first candidate demonstrates excellent analytical skills and develops a strong recommendation without AI assistance.

The second candidate has weaker independent analytical skills but uses AI effectively to produce an even better recommendation.

Which candidate is more likely to perform successfully in the role?

There is no universal answer.

For some positions, independent technical knowledge and judgement remain essential. For others, the ability to use AI effectively may become an increasingly important contributor to performance.

This creates an interesting challenge for Assessment & Development Centers.

Traditional business simulations may need to evolve to reflect the environments in which candidates will actually work.

One possible approach would be to evaluate candidates in two different conditions:

Without AI assistance, to understand their independent capabilities.

With AI assistance, to observe how they use technology, evaluate its recommendations, identify potential errors, and make decisions.

Such an approach would require careful design and validation. The study itself did not test recruitment or assessment methods, so these are potential implications rather than established recommendations.

What About Learning and Development?

The findings are equally relevant to employee development.

Organisations invest heavily in helping employees acquire new capabilities.

But if AI enables people to complete unfamiliar tasks successfully, how do we distinguish between genuine learning and technology-assisted performance?

This distinction matters when evaluating training effectiveness.

An employee may produce an excellent financial analysis using AI without fully understanding the underlying methodology.

That may be perfectly acceptable if the objective is to deliver an accurate analysis efficiently.

But it becomes a problem if the organisation assumes the employee has acquired expertise they cannot demonstrate independently.

Training programmes may therefore need to distinguish more clearly between two objectives: improving performance with AI and developing knowledge that employees can retain and apply without it.

Both can be valuable. They are simply not the same outcome.

Are We Measuring the Right Competencies?

Perhaps the most interesting implication of this research is not that AI makes employees more productive.

We already have substantial evidence that AI can improve performance on certain tasks.

The more challenging question is whether our traditional understanding of professional competence remains sufficient in an AI-enabled workplace.

As AI becomes integrated into everyday work, organisations may need to consider not only what employees know, but also how effectively they collaborate with intelligent systems.

The ability to evaluate AI-generated recommendations, recognise limitations, exercise independent judgement, and take responsibility for decisions may become increasingly important.

For industrial-organisational psychology, this opens an important area for future research.

If AI allows people to perform tasks beyond their individual capabilities, how should we define, measure, and predict professional performance?

The answer may reshape the way organisations recruit, assess, and develop talent.

And perhaps the biggest challenge will not be evaluating how intelligent people are, but understanding how effectively they can work alongside increasingly intelligent technology.


Research reference

Wiles, E., et al. (2026). Generative AI and the Temporary Upskilling of Knowledge Workers. Nature Human Behaviour.

https://www.nature.com/articles/s41562-026-02566-6

← BACK TO ALL ARTICLES