Research Assessment Research
How valid is our AI skills assessment? 6 studies from our technical report
In short
This article summarizes Codility’s AI for Business technical report, version 2.0, which sets out 6 studies on the validity, reliability and fairness of its AI skills assessment for business roles. The full report, with every method, table and limitation, is free to download. In live hiring, scores rose with an expert’s ratings at every level. No meaningful gender differences were found. The evidence supports using scores from 2 or more tasks to rank candidates for a shortlist, with a person reviewing mid-range scores.
- Candidates use an AI Assistant to work through a realistic business problem and hand in a short written deliverable. Fixed rules score it against a rubric written for each task, so the same answer always gets the same score, and no generative AI model is involved.
- In live hiring, a trained expert rated 96 applicants from 1 to 5. The engine’s average score rose with every step up that scale, from 37% for applicants rated 1 to 87% for those rated 5.
- A first-pass check of all 53 longer tasks, the ones that take 20 to 50 minutes, found defects in 28. Tasks that an external AI tool such as ChatGPT could answer from the description alone are scheduled for redesign and retesting.
- Decisions should rest on at least 2 tasks, and 3 or more when the stakes are high.
- No meaningful score differences were found between women and men. Racial and ethnic differences were small on average, and at selective passing scores some groups fell below four-fifths of the highest group’s pass rate, so each employer should set its own passing score and monitor pass rates.
- No score built from a candidate’s conversation with the AI is used in hiring decisions.
- Evidence that scores predict job performance has not yet been established.
Since May, our team has talked with talent leaders, people-analytics teams and industrial-organizational psychologists at customer organizations about assessing AI skills. The request we hear most often is to see a candidate use AI on a realistic task. The psychologists and legal reviewers then go further. They want to know exactly how the score is produced, and they ask for evidence on whether it predicts performance and whether it disadvantages any group of candidates.
That is a reasonable thing to ask of any selection procedure, and a hard one to answer in a category this new. In October our team reviewed about 30 vendors selling assessments of AI skills. None of them published evidence on reliability, on whether scores predict job performance, or on adverse impact for an assessment in which a live AI takes part in the task.
Our answer is the AI for Business technical report, version 2.0, which covers 6 studies run between April and October 2026. This article summarizes it study by study. The full report runs to 59 pages, with every method, table and limitation, and it is free to download. The report is organized as a validity argument, so reviewers can check it against the professional standards for testing. Validity belongs to how a score is interpreted and used, so the report tests each link in that chain in turn. Is the score derived accurately? Does it generalize beyond the one task a candidate took? Does it reflect the skill as it is used on the job? Are decisions based on it fair?
What does the assessment measure?
It measures how well a candidate uses AI to get useful and accurate results on a realistic job task. The report calls this AI-mediated problem solving, and defines it as four behaviors.
- Asking questions that draw out the information the problem needs.
- Probing the evidence through the AI, and questioning the first summary it offers.
- Checking what the AI returns against the source data and the logic of the problem.
- Communicating a conclusion the evidence supports.
Each task gives the candidate a role, a business problem, an evidence base of data extracts, reports and correspondence, and the AI Assistant. The materials carry real signals and misleading details planted on purpose. The correct conclusion depends on facts the candidate has to draw out and verify. Grammar and spelling do not affect scoring. Role-agnostic tasks test core AI skills in everyday work scenarios that need no job-specific knowledge, such as writing an email, messaging a colleague or working with a chat agent.
Work samples, tests in which candidates do a piece of the actual job, have a long record as predictors of job performance. You will still see .54 quoted for them, from Schmidt and Hunter in 1998. The current estimate, from Sackett and colleagues’ 2022 review of the research, is a correlation of .33 with later job performance, and that is the figure we use. Work samples also tend to show smaller differences between demographic groups than cognitive ability tests, although a 2008 review by Roth and colleagues found those differences larger than earlier estimates suggested. Candidates rank them among the selection methods they view most favorably, and they give both the candidate and the hiring team a realistic picture of the job.
The AI Assistant is part of the test itself, and it is calibrated task by task. Each task ships with a behavior specification for it: what it knows, what it reveals and when, and what it declines to do. That standardizes conditions across candidates, so differences in outcomes come from how candidates used the assistant. The full conversation is saved to the candidate report, and reviewers can read it at any point after submission. The score evaluates the result the candidate obtained and verified.
The standard library holds 71 work samples. 51 are longer tasks of 20 to 50 minutes across 9 role families, from sales and finance to customer support and product management. The other 20 are short role-agnostic tasks. They run inside Codility Screen, Interview and Skills Intelligence, in six formats.
| Format | What the candidate does |
|---|---|
| Decision briefing | Weighs competing courses of action, each backed by a plausible stakeholder, and recommends the one the evidence supports |
| AI output audit | Checks a report the AI wrote against the source data and finds the errors planted in it |
| Research drilldown | Pulls a defined set of findings from several sources under time pressure |
| Prompt engineering | Writes a prompt that must reliably produce an accurate, audit-ready piece of work |
| Lean, about 5 minutes | Writes one prompt, or corrects one flawed AI draft |
| Inbox, about 10 minutes | Answers a workplace email from two sources, where the assistant’s first answer is designed to mislead |
Every scoring criterion is linked to skills in the Codility Engineering Skills Model, the framework our team maintains. Across the 626 criteria in the June 2026 library’s 56 longer tasks, 69% carry at least one AI skill. Those links concentrate on checking the AI’s output and on giving the AI the context it needs, because those are the skills a written answer shows most clearly. How a candidate delegated work to the AI, and how often they refined its answers, shows up in the conversation itself. Reviewers can read every conversation today, and measuring it directly is the subject of the last study below.
The model was built largely with engineering experts, so its relevance to business work has to be shown task by task. The studies below are where we show it.
How is the written work scored?
A deterministic natural language processing (NLP) engine scores it against a rubric of behavioral and technical criteria written for each task, and the engine contains no generative AI model. The same answer gets the same score every time. Each criterion can be inspected, reproduced and challenged, and the reviewer sees a plain-language account of which criteria a response met.
Some criteria look for required conclusions, figures and concepts, with checks for negation and for the scope of a claim, so quoting a flawed draft earns nothing for correcting it. Others compare the meaning of a response with 4 to 6 exemplar answers, often called golden answers, written to cover the distinct valid ways of reaching the right conclusion.
We also tested the obvious alternative, a large language model judging each response as a whole. In a small internal comparison during 2026, the model gave almost the same judgment every time it saw a response, yet its judgments bore little relation to a subject-matter expert’s ratings of the same work. The comparison was limited, so we treat it as preliminary, but it points the same way as the rest of our research: a scorer can be consistent and still be wrong. Deterministic criteria let us check accuracy one criterion at a time, which is what the next two studies do.
The deterministic approach carries its own risk, and we want to name it. A correct answer phrased very differently from the exemplars can earn less credit than it deserves. We guard against that with several exemplars per task, and a task is released only when concise but correct answers score at least 80%. We have not yet estimated the size of this risk for candidates who are not native English speakers.
Does the engine score the way an expert would?
On the scenario we tested, the engine agreed with the consensus of 3 expert raters at .77, on a scale where 1 means perfect agreement, a level conventionally described as good to substantial.
In April 2026 we tested the engine on 200 synthetic responses to a single scenario: a chief executive’s dashboard shows an 18% drop in a key metric after a pricing change, and the candidate recommends one of four courses of action. The correct answer recognizes that most of the drop is a measurement artifact, because the metric’s definition changed and automated traffic was filtered out.
The 3 raters agreed with one another at .91, so their consensus made a stable benchmark. The engine’s .77 is an intraclass correlation, the standard measure of agreement for this kind of study. It placed a response more than one point away from the experts on the 3-point scale in 1% of cases, and it put all 46 clearly weak responses in the weak category. Partly correct answers were the hardest to place, with the engine matching the experts on 21 of 44. Passing scores often sit in that middle range, so that is where we have focused the fixes described below, and it is why we recommend that a person reviews mid-range scores before any decision.
This study used synthetic responses to one scenario outside the current library, rated by members of the team that built the assessment. The next step is a replication on real candidate answers, rated by experts independent of development.
In a follow-up, we generated 50 new synthetic responses to the same scenario and compared the engine with an AI rater whose ratings had been checked against the human ones. Agreement fell to .49. Several criteria had passed every check against exemplar text and then failed on newly written answers. One had a threshold that new text could never reach, so it failed every response, including the best. Another failed all 10 of the strongest responses. After 7 corrections, agreement reached .87, an optimistic figure because the same 50 responses were used to find the faults and to measure the fix.
A criterion that looks perfect against the answers it was written from can still fail on the answers candidates actually write, and that lesson changed how we build tasks. We also moved the criteria that look for required conclusions, figures and concepts to graded credit, so a response can earn nothing, half or full credit on each one. That helped most in the middle of the scale, against the AI rater. We have not yet tested it against human raters.
What happened when we tested every task in the live product?
On the first pass, 25 of 53 longer tasks passed all 5 tests. The most serious failure, an external AI model answering from the description alone, was concentrated in 2 task formats.
Between June 17 and 25, 2026, 8 reviewers from assessment science, content and product ran 5 tests on every longer task in the live candidate environment.
| Test | What should happen | Tasks that behaved as designed |
|---|---|---|
| Exemplar answer | Scores at the top of the range | 49 of 53 |
| Unedited starter text | Scores zero | 46 of 53 |
| Exemplar with required parts removed | Score falls in proportion to what is removed | 52 of 53 |
| Off-task text generated by AI | Earns no credit | 42 of 53 |
| External AI answer written from the description alone | Stays below a passing-level score | 31 of the 52 tasks tested |
In 52 of 53 tasks, scores fell in proportion to the substance removed, the result we take most comfort from, because it shows the engine rewards reasoning and gives little for isolated keywords.
Writing from the task description alone, an external AI model rarely reached a passing-level score on research drilldowns, 1 of 14, or on AI output audits, 2 of 14. It often did on decision briefings, 11 of 15, and on prompt engineering tasks, 7 of 9. The likely reason is in the design. A decision briefing lists its options in the description, and the framing often hints at the answer. A prompt engineering task states what the prompt has to achieve. In both, the description gives away what the data was supposed to hold back.
As a result, prompt injection, refusals and unedited placeholders now score zero, and starter text must score 0%. Under a program the report schedules from October 2026, every live task is to be tested against the answer an external AI model writes from its description, and each has to score below an agreed threshold. The fixes under consideration include holding decisive facts in the evidence base and designing the AI Assistant’s first answer to sound fluent while missing something, as the inbox tasks already do. Until a decision briefing or prompt engineering task has been rebuilt, we recommend proctoring it. Re-test results are not yet published.
Are the short tasks ready for high-stakes decisions?
They passed expert review, and today they work best as part of a set of tasks or for development use. Using one on its own for a high-stakes decision comes after the improvements described below.
Short tasks exist because talent teams asked for them: one customer told us it lost candidates beyond about 5 minutes per task, and others asked that short tasks still look like real work. Short tasks are scored on 5 to 8 criteria, and longer tasks on around 12.
Before release, a design review found that a typical inbox task spent roughly 40% of its score on email etiquette and empathy, with little to do with AI. We rewrote the scoring of every inbox task so that AI-skill criteria carry between 50% and 75% of the score.
Between August 21 and September 1, 2026, 3 assessment scientists reviewed all 20 short tasks under decision rules we fixed before collecting any data. Two of them rated 16 tasks, naming the skills each task required before seeing our tags. The third had built the tasks and tested them in the platform, and the remaining 4 tasks were cleared on measurement evidence alone. Of 22 ratings of whether a task measures the skill it claims to, 20 were 4 or 5 out of 5, and all 20 tasks were cleared, 8 of them after scoring changes.
A later calibration round, scored against an AI judge on synthetic answers, showed the short tasks still let through more weak answers than our standard allows. We are tightening their scoring toward a target of fewer than 1 in 10, work the report schedules from October 2026.
Do scores match expert judgment in real hiring?
In a field study, scores on a custom AI task rose with the expert’s judgment at every level.
Study 3 ran with the selection and assessment team of a large US telecommunications company, for a senior analyst position resolving escalated customer complaints, a role that carries reputational and regulatory consequences. Before working with us, the team assessed AI skills by hand, with one moderator watching each candidate use a general-purpose AI tool on a proctored video call, which the partner found slow and hard to standardize.
Their team ran a formal job analysis, and we built a custom task from it using their documents and scenarios. Candidates have 75 minutes to query the AI Assistant across 6 workspace files and write a recommendation that resolves a complex billing dispute. The response is scored on 12 criteria covering the decision, the financial reconciliation and adherence to policy. Their experts piloted the task and reviewed the model answer, and the partner asked its legal team to review the scoring method.
In summer 2026, 96 applicants completed the task in a live hiring process, and the partner made 29 hires from that round. A trained expert at the partner rated every response independently on a 5-point scale, where 3 or higher counts as a pass.
| Expert’s overall rating | Candidates | Average engine score |
|---|---|---|
| 1, lowest | 12 | 37% |
| 2 | 28 | 60% |
| 3 | 24 | 76% |
| 4 | 19 | 83% |
| 5, highest | 13 | 87% |
The rank-order correlation between the expert’s rating and the engine was .72, which means the engine put candidates in much the same order as the expert did. Against the partner’s pass mark, a rating of 3 or higher, the engine matched the expert’s pass or fail call for 79% of candidates, at either of two engine thresholds we examined after the fact.
Of the 28 responses the expert rated 2, the engine placed 14 above 60%, the largest group of disagreements. The partner’s analysts found the expert had also weighed grammar, spelling and tone, which the 12 criteria leave out. The partner already assesses writing in its interview, so whether to add it to the task is their design question. Their raters described the task as representative of the work and of where the role is heading.
One expert rated every response, so there was no second expert to compare against, and we could not run the planned test of whether engine and expert scores are interchangeable. It covers one custom task. The engine’s percentages and the expert’s 5 levels do not line up one to one, so a passing score for this task has to be set on the engine’s own scale. A second hiring round is under way. Because the partner holds interview outcomes for these candidates, a first analysis of scores against later hiring outcomes is now possible.
How many tasks does a full picture of a candidate take?
Each task adds to the picture: 1 task gives a signal, 2 give a clearer one, and 3 or more start to build a full view of what a candidate can do. In the operational data, 716 candidates completed both a decision briefing and an AI output audit, and their scores correlated at .41. That fits a shared underlying skill, along with the large task-specific variation that research on performance assessment has documented for decades.
The Spearman-Brown formula, the standard estimate of how consistency grows as you add comparable tasks, projects .58 for 2 tasks, .67 for 3 and .73 for 4. Read those as approximate. They are the basis for our recommendation: at least 2 tasks per decision, and 3 or more when the stakes are high. Avoid pairing two tasks that turn on the same key insight, because the second partly measures exposure to the first.
Does the assessment treat groups of candidates fairly?
For women, the evidence shows no disadvantage on any measure we have. For racial and ethnic groups, average differences were small, and pass rates at selective passing scores need watching. We want to be precise here, because this is the result most likely to be quoted out of context.
Study 5 covers 1,006 Codility Screen sessions from July to October 2026 that included AI for Business tasks, 330 of them with demographics from an optional survey. No employer in the sample had set a passing score, which psychologists call a cut score. We therefore compared pass rates between groups at 4 hypothetical passing scores, 40, 50, 60 and 70. The Uniform Guidelines on Employee Selection Procedures use a four-fifths rule: when a group passes at less than 80% of the rate of the highest-passing group, that is generally treated as evidence of adverse impact that warrants review.
Average score differences between women and men were negligible on every outcome, and none was statistically significant. Two of 12 comparisons fell below four-fifths, both at a passing score of 70 and both with lower pass rates for men. One would reverse if one more man passed. The other, on a single task, was significant on one of two tests.
Three racial and ethnic groups were large enough to analyze: Asian, White, and Hispanic or Latino candidates. Average differences among them were at most 0.37 of a standard deviation, small by conventional standards, and the largest, on a single task, was statistically significant. At passing scores of 40 and 50, every comparison met the four-fifths rule or fell within one candidate of it. At 60 and 70, Asian candidates had the highest pass rates. In 6 of 12 analyses, White candidates passed at less than four-fifths of the Asian rate, and 4 of those were statistically significant. Hispanic or Latino candidates fell below four-fifths of the Asian rate in 6 analyses, none statistically significant. The Uniform Guidelines compare every group with the highest-passing group, whichever group that is, so these results would warrant attention if an employer set a passing score in that range.
The passing scores are hypothetical, and adverse impact is properly judged at the passing score an employer actually uses. Survey respondents scored slightly higher than non-respondents. Most of the demographic sample, 94%, came from one employer’s multinational applicant pool, with US EEO-1 categories applied and no information on where candidates live, so differences may partly reflect regional composition and English proficiency. Age and disability were not collected. All tasks are in English, and a country-level analysis has been proposed.
Our advice to any employer using this at volume is the advice Tony gave after the Department of Justice opinion on disparate impact in June. Set passing scores locally, and monitor pass rates in your own applicant flow from the first day. Job-relatedness and adverse impact are separate questions. Our team can help design that monitoring.
What is the candidate experience when AI is part of the test?
Candidates were mostly positive. Of those who answered the post-test survey, 81% rated the AI Assistant favorably and 64% agreed the task reflected real work. The least favorable item asked whether a person who scored well would be a good candidate for the job: 37% agreed and 42% chose the neutral midpoint. Judging the job-relatedness of a new kind of assessment from the candidate’s chair is hard, and we read that midpoint mostly as uncertainty. A third of candidates answered the survey, so these results may not represent everyone.
Women and men did not differ in how fair they found the assessment. One finding surprised us. Women estimated their own scores lower than men did, although their actual scores did not differ. Both groups overestimated, men by about 21 points and women by about 12. That gap matters for how we design the feedback candidates receive.
As someone not so familiar with IT and more exposed on Finance, I really appreciate the test on how to use the AI tools.
Candidate, August 2026
Candidates asked to open the attached files themselves, and we added a read-only file viewer in August 2026. The most frequent suggestions now concern the time limit and copy and paste. The copy restriction is deliberate, because it protects the task content, and a recalibration of time limits from observed completion times has been proposed.
Can you score how a candidate uses the AI?
Not yet with the accuracy a hiring decision needs, and the research toward it is under way.
Today the score comes from the written work alone, while the conversation shows parts of working with AI that a written answer hides, such as how a candidate delegated and how often they iterated. Scoring it defensibly is harder, because there is no single correct conversation.
Study 6 is a series of interim studies, run from July to October 2026 on transcripts of AI-assisted software engineering. It focused on context engineering, meaning how well a person gives the AI the objective, context, constraints and success criteria it needs. We compared two approaches. In one, an AI model reads the whole transcript and gives a single rating. In the other, it judges a bank of specific behaviors one at a time, and rules combine the results.
Both were highly repeatable, with run-to-run agreement between .85 and .95. In October we completed the first comparison with an expert, who rated 70 candidates, 61 of whom had enough conversation to analyze. The single overall rating agreed with the expert moderately, at .63. Every method built from specific behaviors agreed poorly, between .11 and .26.
No score built from a transcript will be used in hiring decisions until a method agrees with independent experts about as well as experts agree with each other. The next step is a structured overall judgment that quotes its evidence, tested against fresh ratings from 2 experts and then extended to AI for Business conversations.
What can you use the scores for today?
You can use scores from a set of tasks to rank candidates for a shortlist, with a person reviewing the middle of the range.
| Use | What the evidence supports today |
|---|---|
| Ranking candidates on 2 or more tasks, 3 or more when stakes are high | Supported |
| Human review of mid-range scores | Recommended, because agreement with experts is weakest on partly correct answers |
| Decision briefing and prompt engineering tasks before they are rebuilt | Use with proctoring |
| Short tasks within a broader set, or for development | Supported |
| Short tasks as a stand-alone hurdle in a high-stakes decision | Wait for human ratings of candidate answers, and for weak answers to pass less than 10% of the time |
| The assessment as an independent hiring hurdle | First build local evidence linking the tasks to the role, as the field study did |
| Measuring change before and after training | Wait for matched versions of each task, in development from October 2026 under the report’s schedule |
| Benchmarking against a wider candidate population | No category-wide benchmark exists yet. Interpret scores against each task’s criteria |
| Conversation transcripts | Supplementary information for a reviewer to read. They carry no score |
Where an employer needs a passing score, set it locally on the engine’s own scale. That can happen before testing, with a structured method such as a modified Angoff procedure, in which subject-matter experts judge how a minimally qualified candidate would score. It can also happen after a first round, by reviewing score distributions, pass rates and adverse impact together. In practice, our team works through these choices with each customer: which tasks to combine, where to set the passing score and what to monitor once hiring starts.
What comes next?
The next studies answer the open questions, starting with whether scores predict job performance. The report lists every one under way, scheduled or proposed, and these are the ones we are watching most closely.
- The second hiring round with the telecommunications partner, and an analysis of scores against the partner’s interview outcomes.
- A two-rater study on real answers, with the target that engine-to-human agreement reaches at least .70 and comes within .05 of agreement between the two humans.
- A replication of the scoring study on real candidate answers, rated by experts independent of development.
- Short tasks on which weak answers pass less than 10% of the time.
- Proposed analyses of reliability, on more pairs of tasks and on candidates who retake a task, and analyses by country and by age.
If you deploy AI for Business in high-stakes hiring, treat the deployment as a chance to build local evidence. Our team can support content validation against your job analysis, benchmarking with current employees, and studies that link scores to later performance. We will revise the report as each limitation is addressed.
When we review someone else’s assessment, the limitations section is where we start. It tells us whether the people who built the test understand what they built. Much of what is written about AI skills assessment today describes what a product can observe. A technical report has to say what a score means, and how far that meaning reaches. That is what we have tried to publish.
Sources
- Codility. 2026. AI for Business Technical Report, version 2.0, October 2026.
- American Educational Research Association, American Psychological Association and National Council on Measurement in Education. 2014. Standards for Educational and Psychological Testing.
- Hausknecht, J. P., Day, D. V. and Thomas, S. C. 2004. Applicant reactions to selection procedures: an updated model and meta-analysis. Personnel Psychology, 57, 639 to 683.
- Kane, M. T. 2013. Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1 to 73.
- Roth, P. L., Bobko, P., McFarland, L. A. and Buster, M. 2008. Work sample tests in personnel selection: a meta-analysis of Black-White differences in overall and exercise scores. Personnel Psychology, 61, 637 to 661.
- Sackett, P. R., Zhang, C., Berry, C. M. and Lievens, F. 2022. Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107, 2040 to 2068.
- Schmidt, F. L. and Hunter, J. E. 1998. The validity and utility of selection methods in personnel psychology. Psychological Bulletin, 124, 262 to 274.
- Shavelson, R. J., Baxter, G. P. and Gao, X. 1993. Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215 to 232.
- Uniform Guidelines on Employee Selection Procedures. 1978. 29 C.F.R. Part 1607.
AI for Business technical report, version 2.0
The full report: 6 studies, every table and every limitation.
Download PDF
About the authors
Tony Mellek, PhD, is Head of Assessment Science at Codility. He holds a PhD in industrial-organizational psychology and has spent his career building assessments across industries, including selection tests for police and fire departments. At Codility he leads the assessment science team, which maintains the Engineering Skills Model and designs the measures behind Codility’s engineering assessments and AI for Business.
Shane McKnight, PhD, is an Assessment Scientist at Codility. He holds a PhD in industrial-organizational psychology from Louisiana Tech University. He works with enterprise customers to design skills-based assessment strategies, and with product and engineering teams to build assessment science into the platform. He co-wrote this piece.
Frequently asked questions
Codility gives each candidate a realistic business problem, its source data and an AI Assistant, and scores the short written result against a rubric written for the task. The score reflects how well the candidate draws out information, checks what the AI returns and supports a conclusion with evidence. The assessment runs inside Codility Screen, Interview and Skills Intelligence.