Article AI and Hiring
How to Hire AI-Native Engineers: What to Test, and What to Score
In short
To hire AI-native engineers, decide what the role should hand to AI agents, ask for proof of AI-assisted work they have shipped, and test them on a realistic task with the agents they use at work, with an AI mistake planted for them to catch. Score the result the same way for every candidate. Then review six things in how they got there: how they frame and delegate the work, the context they give the agent, how they iterate, whether they validate the output, how they recover when it fails, and whether they understand the result.
Anthropic rebuilt its own engineering take-home after its model caught up with the candidates. More than 1,000 people had completed the test, and by the two-hour mark Claude Opus 4.5 matched the best human score, according to Anthropic engineer Tristan Hume, writing on January 21, 2026. If the company that builds the model cannot keep a take-home ahead of it, a hiring loop that scores only the final code is measuring the AI as much as the candidate.
I wrote this guide for engineering leaders and talent teams who run technical hiring. Talent teams need consistency across teams and a decision that holds up when a candidate asks why. Engineering leaders need a better signal and fewer interview hours lost to candidates who cannot do the job.
Each side usually owns part of the loop. The talent team runs Stages 1, 2 and 6: the AI policy, the application question and a consistent decision. Engineers run Stages 3 to 5: the task, the live review and the understanding check.
I read more than 30 studies and standards from the US National Institute of Standards and Technology, NIST, and from Anthropic, METR, Stack Overflow and others, and used Codility’s own assessment data. I lead product marketing at Codility, which sells technical assessments, including the AI-Native tasks this guide describes.
This guide gives you:
- a six-stage hiring loop, from defining the role to the final decision
- a test plan with the format and the AI rules for each stage
- a review guide for what to score and what to review, which you can copy
The guide does not cover job descriptions, compensation or the offer.
Key takeaways
The job now runs on AI.
METR reported in February 2026 that a growing number of developers declined to join its productivity study because they did not want to work without AI. A loop that bans AI tests a different job from the one you are hiring for.
Take-homes are losing signal.
Claude Opus 4.5 matched the best human score on Anthropic’s own performance take-home by the two-hour mark.
Output and understanding come apart.
In Anthropic’s randomized study of 52 mostly junior engineers, the engineers who learned with AI scored 50% on a comprehension quiz, against 67% for those who coded by hand.
Score the result, and review how they got there.
Correct code is not always efficient or maintainable, and people check polished AI output less often, so a hiring loop needs both a score and a review of the process.
Structure still predicts best.
Structured interviews rank first in the 2022 meta-analysis by Sackett and colleagues, at a validity of 0.42 against 0.19 for unstructured interviews.
Keep AI out of the score.
Let AI help the candidate, and score the work with a deterministic rubric that gives the same result for the same work, so the decision can be explained to a candidate or, in the EU, a regulator.
What is an AI-native engineer?
An AI-native engineer, also called an AI-native developer, is a software engineer whose normal way of working is to direct AI coding agents, check what they produce and take responsibility for the result. Anthropic’s 2026 Agentic Coding Trends Report describes the shift as one from writing code to reviewing, directing and validating code that AI generates.
In an Anthropic survey of 132 of its engineers and researchers, published on December 2, 2025, staff reported using Claude in 59% of their work, and most said they could fully delegate only 0 to 20% of it. Heavy use with little full delegation makes supervising the agent the core daily skill.
What makes an engineer AI-native?
An engineer is AI-native when they show the four competencies of the AI Fluency Framework by Rick Dakan and Joseph Feller, developed with Anthropic, and back them with engineering fundamentals:
- DelegationDeciding whether, when and how to hand work to the agent.
- DescriptionGiving the agent the goal, the context and the constraints.
- DiscernmentJudging whether the output is useful and correct.
- DiligenceTaking responsibility for what ships.
| AI-native engineer | AI engineer | Engineer who doesn’t use AI | |
|---|---|---|---|
| Daily work | Directs coding agents, reviews and owns their output | Builds AI features and systems, such as models, retrieval and agents | Writes and reviews code by hand |
| What to test | Delegation, context, validation and fundamentals on a realistic task with agents | AI-system design, evaluation and data handling | Fundamentals on a realistic task |
| Where to go next | This guide | Hiring AI engineers | This guide, with AI turned off where the role requires it |
Why hiring AI-native engineers is harder than it looks
A hiring loop that scores only the final code now measures the AI as much as the candidate, and a loop that bans AI tests a different job from the one you are hiring for. Both failures come from the same change: the agent now does much of the typing.
The take-home is the clearest case. Anthropic’s Tristan Hume wrote that the original test “worked because it resembled real work,” and that the replacement “works because it simulates novel work.” A task the model has seen, or can solve alone, no longer separates candidates. Talent teams see candidates who pass the screen and then cannot answer basic questions about the same problem in the interview, after an engineer has spent the hour.
Banning AI does not fix the problem. METR, which runs controlled studies of developer productivity, is changing its study design because developers declined to work without AI. A company that tells candidates it works AI-first and then bans AI in the test contradicts its own description of the job.
Allowing AI without checking the output fails too. In the 2025 Stack Overflow Developer Survey, 66% of developers named “AI solutions that are almost right, but not quite” as a frustration. An engineer who accepts almost-right code ships defects, and a loop that never shows them almost-right code cannot tell you which engineers will.
Identity adds one more problem. In a Gartner survey of 3,000 job candidates published on July 31, 2025, 6% admitted to interview fraud, and Gartner predicts that by 2028 one in four candidate profiles worldwide will be fake.
How to hire AI-native engineers: a six-stage hiring loop
Hire AI-native engineers with a six-stage loop that tests the job as it is done today and decides the same way for everyone:
Codility runs Stages 3 to 6Stage 1. Define the role, and the AI rules for each stage
Goal. Split the role’s work between AI agents and the engineer, then choose the hiring stages that allow AI and tell candidates in writing.
How to do it. With the hiring manager, list the tasks the engineer will do in the first six months and mark each one: the agent drafts it, the engineer and the agent build it together, or the engineer does it alone. Most of Anthropic’s engineers say they can fully delegate only a small share of their work, so expect most rows in the middle. Map each row to the four competencies you will test.
Then write the AI policy for the loop. For each stage, state whether AI is allowed, which tools, and whether the activity is recorded. If it is, agree the policy with your legal and data protection team before it reaches candidates.
Anthropic publishes its own policy stage by stage, with AI off in take-homes and live interviews unless the team says otherwise, and its performance take-home is one of those exceptions. Canva Engineering wrote in June 2025 that it now expects backend, machine learning and frontend candidates to use AI tools in technical interviews.
Strong versus weak
| Strong | Weak |
|---|---|
| Each stage names the tools allowed and says the activity is recorded | “AI is allowed” with nothing else |
| Candidates receive the policy before the first stage | Candidates learn the rules in the interview |
Worked example, constructed for this guide
A three-line AI policy: “Application: write it yourself, and use AI to edit if you like. Screen: use your usual coding agent; every prompt and change is recorded. Interview: AI stays on, and one question at the end is answered without it.”
Common mistake
Announcing “AI is allowed” and leaving candidates to guess the tools, the logging and the limits.
Proof
In control GOVERN 3.2 of its AI Risk Management Framework, NIST asks organizations to “define and differentiate roles and responsibilities for human-AI configurations.” NIST wrote it for AI systems, and I apply it here by analogy: a role spec should say the same thing about a person.
Stage 2. Source on proof of work
Goal. Ask for one AI-assisted change the candidate shipped and the AI’s mistake in it, and give more weight to signals that are hard to fake.
How to do it. Add one question to the application: “Link or describe a change you shipped with an AI agent. What did the agent get wrong, and how did you find it?” The answer shows delegation and discernment before anyone spends an interview hour. Then read public work with a skeptical eye: a contribution that a maintainer reviewed and merged is hard to fake, and stars and forks are easy to inflate.
Expect early-career candidates to answer from experience. GitHub’s Octoverse 2025 reports that 80% of new developers on GitHub use Copilot in their first week.
Strong versus weak
| Strong | Weak |
|---|---|
| Names a specific AI error and how they caught it | “I use AI every day” |
| Contributions merged by other people’s review | High star counts on personal repositories |
Worked example, constructed
A candidate links a merged pull request where their review caught an agent-proposed cache that ignored user permissions, and the fix added a test. That one link shows validation and ownership of the result.
Common mistake
Treating “uses Cursor” or “uses Claude Code” on a résumé as evidence of skill. A tool name shows exposure, and the loop still has to test the skill.
Proof
In a 2013 interview study, Jennifer Marlow and Laura Dabbish found that employers trusted contributions accepted to high-status projects and saw popularity counts as a signal developers could game. The study rests on 13 interviews, so I use it as qualitative support only. Check identity later in the loop, given Gartner’s fake-profile forecast; the AI technical assessment page covers integrity.
Codility covers this stage
Stage 3. Screen with a realistic task, the agents they use at work and an AI mistake to catch
Goal. Test in a real integrated development environment (IDE) with Claude Code, Anthropic’s coding agent, Codex, OpenAI’s coding agent, or GitHub Copilot, on a task the model hasn’t seen, built so the agent’s most likely suggestion is wrong.
How to do it.
- Use an existing repository with tests, a realistic ticket and the agents the candidate uses at work.
- Make the task novel, so it simulates work the model has not seen.
- Plant one AI mistake, such as a deprecated internal helper the agent is likely to reach for. Confirm in a pilot run that the agent suggests it.
- Ask for a short context file for the agent,
AGENTS.mdorCLAUDE.md, as evidence for review dimension 2. - Fix one time box for every candidate. Anthropic cut its own take-home from four hours to two, partly because scheduling caused multi-week delays. Set your own limit from pilot runs, and check it against your completion rate in the first month.
Strong versus weak
| Strong | Weak |
|---|---|
| Reads the diff, the agent’s proposed change, before accepting it, then runs and adds tests | Accepts the agent’s first diff and submits |
| Explains why the agent’s suggestion was wrong | Fixes the symptom without noticing the cause |
Worked example, constructed
The ticket asks for rate limiting on an API endpoint in a small service with tests. The repository already has a flawed rate-limit helper, and the agent will usually reach for it. A strong candidate notices that the helper fails an existing test, tells the agent to avoid it in the context file, and adds an edge-case test.
Common mistake
Using a puzzle the model has memorized, or a toy repository with no tests. Both turn the screen into a test of the agent.
Proof
NIST’s Generative AI Profile defines confabulation as “confidently stated but erroneous or false content” and lists “Review generated code” among its recommended actions. Veracode’s 2025 vendor research found that AI-generated code introduced security vulnerabilities in 45% of cases across 80 coding tasks.
In Codility
Eight AI-Native tasks run in native VS Code with Claude Code and Codex, and several are built so the assistant’s most common suggestions are among the wrong answers. More on how to hire AI-native engineers with Codility.
rateLimit.tsCodility covers this stage
Stage 4. Interview by watching them work with the AI, from a full record
Goal. Score the finished work the same way for every candidate, then review how the candidate directed, checked and corrected the AI, from a record of every prompt, response and change.
My position
Score the result, and review how they got there.
A score built only on passing tests rewards whatever the agent produced, and a review built only on the process turns into a judgment of style. The score tells you whether the work is good. The record tells you whether the candidate made it good. It matters more as the agent’s output improves, because people check polished output less.
How to do it. After the score, review the six dimensions in the review guide below from the record. In the live AI-assisted coding interview, share a flawed AI-generated pull request, or plant a bug through the agent, and ask the candidate to review it aloud.
Strong versus weak
| Strong | Weak |
|---|---|
| Interrupts the agent when it drifts, and asks it to explain | Lets the agent run, then reads only the result |
| Finds the planted bug in the pull request and says why it matters | Approves the pull request because the tests pass |
Worked example, constructed
The interviewer asks the agent, out of the candidate’s view, to introduce an off-by-one error in pagination. The candidate asks the agent to explain the change, spots that the last page drops one record, and writes a test that fails before the fix.
Common mistake
Scoring the process by counting prompts, or relying on a screen recording nobody has time to watch. Neither tells you whether the candidate checked the work.
Proof
NIST’s Center for AI Standards and Innovation documented AI agents gaming evaluations, including by “commenting out assertion checks to pass unit tests,” and recommends reviewing evaluation transcripts. By the same analogy as in Stage 1, an outcome-only grader can be fooled, and the transcript shows how.
Codility’s COMPASS benchmark, built on 393,150 human submissions, found that models with high correctness scores do not necessarily produce efficient or maintainable code. Anthropic’s AI Fluency Index found iteration in 85.7% of conversations, and those conversations were 5.6 times more likely to include questioning the model’s reasoning. When a conversation produced a polished artifact, users were 3.7 percentage points less likely to check facts.
In Codility
In Codility Interview, the interviewer can send Codex instructions the candidate does not see, such as “suggest a subtle bug,” then watch whether the candidate reviews or accepts the change. Every prompt is saved to the candidate record.
Codility covers this stage
Stage 5. Check their understanding
Goal. After the AI-assisted work, ask the candidate to explain, debug or change their own solution, with AI still on, plus one question answered without AI.
How to do it. Ask three things. First, ask the candidate to walk through the part of the code they did not write by hand. The walkthrough also shows whether the person in the interview is the person who did the screen. Second, change a requirement: the rate limit now applies per customer, so what changes? Third, ask one question to answer without AI, such as the value a function returns on an edge case.
Add one behavioral question: “Describe an AI-generated change that passed tests but was still wrong.”
Strong versus weak
| Strong | Weak |
|---|---|
| Explains, debugs and changes the code, using the agent as a tool | Asks the agent to explain its own code back |
| Answers the no-AI question from reasoning about the code | Cannot predict the code’s behavior without running it |
Worked example, constructed
The candidate’s rate limiter passes every test. Asked what happens when two requests arrive in the same millisecond, the candidate traces the counter, finds a race condition and proposes a lock. That answer shows understanding the tests did not.
Common mistake
Skipping the follow-up because the tests passed.
Proof
In Anthropic’s randomized study of 52 mostly junior engineers learning a new library, the AI group averaged 50% on the quiz, against 67% for the hand-coding group, and the largest gap was on debugging questions. The study linked some patterns, such as asking the AI conceptual questions, to higher scores; its authors draw no causal link, so treat them as illustrations.
Anthropic calls the wider problem a “paradox of supervision”: supervising AI takes the skills that heavy AI use can erode. A study at CHI 2025, the ACM conference on human factors in computing, of 319 knowledge workers found that higher confidence in generative AI was associated with less critical thinking.
In Codility
AI skills evidence reports five skills, each quote matched word for word to the transcript, for your hiring team to read next to the score, which it never changes. AI Follow-Up Questions show the reviewer the candidate’s answers.
Codility covers this stage
Stage 6. Decide the same way for everyone, and keep AI out of the score
Goal. Give every candidate for a role the same task, tools, time and rubric, across every team that hires for it, and score them deterministically. Use references to confirm the candidate’s claims.
My position
Keep AI out of the score.
A deterministic rubric gives the same result for the same work, so the decision can be explained to the candidate, the hiring manager and, in the EU, a regulator.
The strongest evidence against this position deserves a fair hearing. In a 2026 study in the Journal of Applied Psychology, Kayden Stockdale, Louis Hickman and Siyi Liu found that ensembles of larger, newer language models “hold potential” for scoring employment interviews about as well as, or better than, a single human rater. The authors also set out reasons to be cautious before using language models for high-stakes assessments.
I accept the reliability point, and reliability is where a hiring score starts. The score also has to be repeatable and explainable to the person it rejects, and a rubric with fixed rules is both by construction.
The law
The law raises the stakes. Annex III, point 4, of the EU AI Act, Regulation (EU) 2024/1689, lists AI systems intended to be used for the recruitment or selection of people, including to evaluate candidates, as high-risk. The Digital Omnibus on AI, Regulation (EU) 2026/1744, has entered into force, and its obligations for those systems apply from December 2, 2027.
The record that lets you review the process in Stage 4 is also the record you show when a candidate or an auditor asks how a decision was made.
How to do it. Fix the task, the tools, the time and the rubric per role before the first candidate, and use more than one task where the role allows it. Ask references about specific claims, for example: “Did reviewers trust this person’s AI-assisted changes?”
Strong versus weak
| Strong | Weak |
|---|---|
| Every candidate gets the same task, tools, time and rubric | Interviewers improvise questions per candidate |
| References confirm specific claims from the loop | References asked only whether they would rehire |
Worked example, constructed
A rejected candidate asks the hiring team why. The hiring team shows the hidden tests the solution failed, the rubric line for each, and the record of the planted defect accepted without review. Scoring the same submission again gives the same result.
Common mistake
Letting an AI grade the free-text answers because it is fast. A score that can change on a second run is hard to defend.
Proof
The 2022 meta-analysis by Paul Sackett, Charlene Zhang, Christopher Berry and Filip Lievens ranks structured interviews first, at 0.42, ahead of work samples at 0.33 and unstructured interviews at 0.19. The meta-analysis lists reference checks among the procedures excluded for insufficient information, so use references to confirm claims.
NIST warns against extrapolating from “narrow, nonsystematic, and anecdotal assessments” in its Generative AI Profile, and NIST AI 800-3 separates benchmark accuracy from generalized accuracy. Both were written for AI systems. Applied to hiring, one task is a narrow sample.
In Codility
Scoring is rule-based and deterministic, against test cases, and no AI sits in the pass or fail decision. Codility gives your hiring team the evidence, and your team makes the final decision.
What to test: the test plan for each stage
Test five things across the loop: a realistic task with AI agents, a written context file, a planted AI mistake, a live review of a flawed AI pull request, and an explain-or-change follow-up. A sixth test, AI-system design, applies only to roles that build AI features.
Each test feeds the score or one of the six review dimensions, D1 to D6, in the next section.
| # | What to test | Stage | Format | AI | Feeds |
|---|---|---|---|---|---|
| T1 | A realistic repository task with the agents they use at work | Screen | Async, timed | On | Score; review D1, D3, D4 |
| T2 | A short context file for the agent, AGENTS.md or CLAUDE.md |
Screen | Part of T1 | On | Review D2 |
| T3 | A planted AI mistake in the agent’s suggestion | Screen or interview | Built into T1, or planted by the interviewer | On | Score, defect fixed; review D4 |
| T4 | Live review of a flawed AI-generated pull request | Interview | Live, shared screen | On | Review D4, D6 |
| T5 | Explain, debug or change their own solution | Interview | Live | On, plus one question without AI | Review D5, D6 |
| T6, optional | AI-system design, only for roles that build AI features | Interview | Live | Depends on the role | Hiring AI engineers |
with no email required.
What to score: the result, and six things to review in how they got there
Score the result the same way for every candidate: whether the work is correct, handles the edge cases, fixes the planted defect, and is efficient and maintainable. Then review six things in how the candidate got there, from the full record: framing and delegation, context engineering, iteration, output validation, failure recovery, and fundamentals and ownership.
What to score
Efficiency and maintainability are on the list because of Codility’s COMPASS benchmark, cited in Stage 4. Apply the same rubric the same way to every submission.
What to review
In the live stages, the interviewer uses the 1, 3 and 5 descriptions below as a structured scorecard. There are no numeric weights. The hiring team weighs the dimensions for the role.
Review guide · scorecard
Six things to review, D1 to D6
Rate each dimension 1, 3 or 5 from the record. No numeric weights.
| # | What to review | Maps to | 1means | 3means | 5means |
|---|---|---|---|---|---|
| D1 | Problem framing and delegation | AI Task Delegation; Delegation | Hands the whole ticket to the agent | Splits the work and delegates the obvious parts | States acceptance criteria first, decides what to delegate and why |
| D2 | Context engineering | Context Engineering; Description | One-line prompts, no repository context | Gives files and constraints | Writes a context file with conventions, tests and boundaries, and updates it |
| D3 | Iteration | AI Iteration | Accepts the first answer | Iterates on errors | Iterates on reasoning and design |
| D4 | Output validation | AI Output Validation; Discernment | Doesn’t read the diff | Runs tests | Finds the planted defect, tests edge cases, rejects bad output |
| D5 | Failure recovery | AI Failure Modes; Diligence | Stuck when the agent fails | Retries the prompt | Changes strategy, works by hand where that is faster |
| D6 | Fundamentals and ownership | Diligence | Can’t explain the code | Explains most of it | Explains, debugs and extends it without the assistant |
The “Maps to” column names the matching skill in Codility’s AI skills evidence first and the AI Fluency Framework competency second.
D1. Problem framing and delegation
The candidate plans the agent’s part of the task before prompting it. The evidence sits in the first prompts and any written plan, where a pasted ticket is the weakest sign.
D2. Context engineering
The candidate gives the agent the context it needs. The evidence is the context file and the constraints attached to prompts. A strong one also names the files the agent must not touch.
D3. Iteration
The candidate improves the work over several rounds, and the evidence is the sequence of prompts and diffs.
D4. Output validation
The candidate checks the output before trusting it. The evidence is diff reads, test runs, added tests and rejected suggestions, and finding the planted defect is the clearest strong signal.
D5. Failure recovery
The candidate gets unstuck when the agent fails. The evidence is the candidate’s next move after a wrong or looping answer, where endless rewording is the weakest sign.
D6. Fundamentals and ownership
The candidate understands and owns the result. The evidence comes from Stage 5: the explanation, the requirement change and the question answered without AI.
What to leave out of the score
- Prompt count or prompt cleverness, which says nothing about whether the work is right.
- Speed on its own. In METR’s 2025 study of 16 experienced open-source developers, the developers believed AI had made them 20% faster when it had made them 19% slower. METR now describes that result as out of date.
- The polish of the final code, which lowers checking.
- Loyalty to one tool.
- Whether the candidate used AI at all, once the stage allows it.
How to check that your hiring loop works
Track three leading indicators from the first month, to see whether the loop runs well, and three lagging indicators from the first hires, to see whether it predicts the job.
| Indicator | Type | How to calculate it |
|---|---|---|
| Completion rate | Leading | Candidates who finish the screen, divided by candidates invited |
| Candidate fairness | Leading | Share of candidates who agree the assessment gave them a fair chance to show their abilities |
| Agreement between reviewers | Leading | Share of D1 to D6 ratings where two reviewers land within one point |
| Pass-through after the screen | Lagging | Candidates who pass the live interview, and offers made, each divided by candidates who passed the screen |
| Performance against the score | Lagging | Correlation of the score with 90-day and 6-month performance ratings |
| Adverse impact by group | Lagging | Each group’s selection rate divided by the highest group’s rate |
The same six numbers give the talent team a report to take to finance and leadership at six months.
9 in 10
Vendor dataFor candidate fairness, Codility’s own figure is a benchmark. In Codility’s post-assessment survey, 9 in 10 candidates agreed that the assessment gave them a fair chance to show their abilities. The figure comes from 46,204 responses to that question between October 2025 and April 2026, and it is vendor data.
For adverse impact, the US Uniform Guidelines treat a selection rate below four-fifths of the highest group’s rate as general evidence of adverse impact, the so-called four-fifths rule.
What the evidence doesn’t tell you yet
No controlled study has validated AI-allowed technical assessments against later job performance. I searched for one while writing this guide and found none. The method rests on evidence that points the same way, and each piece has limits:
- The comprehension study is small. Anthropic’s randomized study covered mostly junior engineers learning one library.
- The AI Fluency Index measures consumer chat. Its 9,830 conversations came from Claude.ai, so engineering behavior inside an IDE may differ.
- NIST frameworks are written for AI systems. This guide applies them to hiring by analogy, and NIST has published no guidance on assessing people who work with AI.
- The proof-of-work evidence is one small 2013 interview study.
- Reference checks have no current validity estimate.
- METR’s 2026 follow-up gives only weak evidence on the size of any speedup from AI.
- Vendor figures are labeled as vendor data, Codility’s included.
- The AI-rater evidence is new. One 2026 paper suggests AI raters can be reliable. Evidence on their fairness in hiring is thin.
Can you run this hiring loop yourself?
Yes, if you hire a few engineers a year and one team runs every interview. As volume grows, the by-hand loop gets harder to keep fair, fresh and on the record.
A small team with one interview panel can run every stage with a shared repository, a screen recording and a spreadsheet. Each new team and role adds fresh tasks, the same rubric and another record to keep. Every new task also needs an engineer to build it, plant the mistake and pilot it before a candidate sees it, and those hours come from the same calendars the loop is meant to protect.
| The loop needs | By hand | With a platform |
|---|---|---|
| A steady supply of new tasks | Engineers write and rotate tasks. A task loses signal fast once a model can solve it, as Anthropic’s take-home showed | A maintained task library |
| A record of every prompt, response and change | Screen recordings, slow to review and hard to search | Saved AI activity on the candidate record |
| The same rubric, applied the same way | Interviewer training and calibration. Unstructured interviews predict about half as well as structured ones | One rubric, deterministic scoring |
| Integrity checks | Manual identity and behavior checks, as fake profiles grow | Integrity signals for your team to review |
| A record you can show a candidate or a regulator | Notes and recordings spread across tools, as the EU’s high-risk rules for hiring AI approach | One defensible record per candidate |
| Engineer time | An engineer runs or reviews every screen | Async screening before the live interview |
The costs in the “By hand” column are reasoning, apart from the cited points. If your volume has outgrown the left column, the next section shows how Codility helps you hire AI-native engineers.
How Codility helps you hire AI-native engineers
Codility runs Stages 3 to 6 of this loop: the realistic screen with coding agents, the live interview, the record of how the candidate worked, and the deterministic score. Codility does not source candidates or write role definitions, so Stages 1 and 2 stay with your team.
Stage 3, Screen
Candidates work on the eight AI-Native tasks in native VS Code with the real Claude Code and Codex, preconfigured, with no account or API key to set up for the company or the candidate.
The AI Copilot and the AI Assistant are also available. AI Assistance switches on or off per task, so a build task can allow the agent while a fundamentals task keeps it off, and one account-level switch sits above every AI feature.
Stage 4, Interview
Codility Interview gives the candidate and the interviewer a shared terminal where both can work with Claude Code and Codex. AI Assistance can be switched on or off per interview.
Stages 4 and 5, evidence
AI skills evidence reports five skills: Context Engineering, AI Iteration, AI Output Validation, AI Task Delegation and AI Failure Modes. Each skill is judged against a fixed list of 10 to 13 behaviors written by Codility’s assessment science team, and the method was tested on 300 real sessions. The evidence sits next to the score without changing it.
Stage 6, scoring
Scoring is rule-based and deterministic, against test cases tagged to the Engineering Skills Model, with no AI in the pass or fail decision. Integrity signals combine into a risk level for your team to review.
Compliance
Codility is ISO/IEC 27001:2022 certified and audited every year under SOC 2 Type II. Customers choose US or EU data hosting in the contract, with the EU cluster in Frankfurt. Codility’s Trust Center states that “no AI system makes or materially shapes a hiring decision without a person in control.” The EU AI Act hiring checklist explains what the law asks of employers.
The limit
AI skills evidence is not yet included in PDF or CSV exports or in applicant tracking system (ATS) reports, so your team reads it in Codility.
See how to hire AI-native engineers with Codility.
FAQ
Yes, in the stages that test the job as engineers now do it, with the tools named in writing and every interaction recorded. Keep one question that the candidate answers without AI, so the loop also checks fundamentals. Anthropic allows AI in its redesigned performance take-home, and Canva expects AI tools in its technical interviews. The AI technical assessment page covers the policy question in more depth.
Further reading
- Hire AI-native engineers with Codility
- Codility AI-Native tasks
- AI skills evidence for Codex and Claude Code
- AI in technical assessments: Codility’s approach
- The case against AI interviews
- The best technical assessment platforms in 2026
- Anthropic, How AI assistance impacts the formation of coding skills
- Dakan and Feller, with Anthropic, The AI Fluency Framework
- NIST, AI Risk Management Framework 1.0
- NIST CAISI, Cheating on AI agent evaluations
See an AI-Native task in a real IDE
Codility’s AI-Native tasks put candidates on a real repository with Claude Code and Codex, a planted AI mistake and a full record of how they worked.
How this guide was researched
I built this guide on more than 30 primary sources, including studies and reports from Anthropic, NIST, METR, Stack Overflow, GitHub, Gartner and the Journal of Applied Psychology, and I opened each one on or after October 26, 2026. I ran the questions hiring teams ask AI assistants through Perplexity on October 26 and 27, 2026. Codility data comes from AI skills evidence, the COMPASS benchmark and Codility’s candidate survey. The data cutoff is October 30, 2026.
About the author
Christopher Greco
Head of Product Marketing, Codility
Christopher Greco leads product marketing at Codility, where he works on how engineering teams assess candidates who build with AI. Before Codility, he led product marketing for AI data and model evaluation at Toloka.