First in category · Live now
Your engineers use AI every day. Your assessment should too
Writing code is becoming a commodity. What separates strong engineers now is judgment of AI output: knowing when to trust the assistant, when to doubt it, and how to direct it. Codility AI-Native tasks assess exactly that, in a real VS Code IDE with an AI assistant in the loop. Eight tasks are live today.
# z-score anomaly check
def detect(series):
mean = stats.mean(series)
sd = stats.stdev(series)
return [x for x in series
if abs((x - mean) / sd) > 3]The job changed. Most assessments did not
An AI assistant can generate a function in seconds. That part of the work is now a commodity. What decides whether the code is correct, safe and fit for purpose is the engineer: how they give context, decompose the problem, and judge what the assistant hands back.
Traditional assessments test the opposite. They drop candidates into a stripped-down editor with AI banned, which is nothing like the job your team does every day. You end up measuring how well someone copes with an artificial constraint, not how well they engineer alongside the tools they actually use.
AI fluency is now a baseline hiring skill. The Pragmatic Engineer’s State of the Job Market 2026 puts it plainly: working effectively with AI is becoming table stakes for engineering candidates. So the question is no longer whether your candidates use AI. It is whether you can see how well they do it.
AI fluency is becoming a baseline hiring skill for engineering candidates in 2026
The Pragmatic Engineer
State of the Job Market 2026
Two things you now need to test
You need to answer two separate questions. Can this candidate build AI products? And can this candidate work alongside an AI assistant, including knowing when to doubt it? AI-Native tasks cover both.
Build AI
The AI or ML system is the product the candidate builds. You see whether they can stand up something that works: retrieval quality and grounding, statistically sound detection, correct persistence.
Work with AI
The AI assistant is the co-worker. You assess how well the candidate drives it and, the part that matters most, whether they catch when it is wrong.
Eight AI-Native tasks, live today
This is shipped product, not a roadmap promise. Eight tasks are live now, each tagged against the Codility Engineering Skills Model.
Build AI · Converse
Legal Document Q&A Assistant
Implement a LangChain and FAISS RAG pipeline that answers natural-language questions about a legal document archive, returning a generated answer plus the retrieved source chunks.
Work with AI · Debug
Meeting Planner
Fix subtle bugs in an existing meeting planner, then implement a new group time-slot suggestion feature. The assistant’s first answer is often the wrong one, and the whole signal is whether the candidate catches it.
Build AI · Predict
DataSentinel
Build an anomaly-detection pipeline that ingests daily KPI readings, persists history to PostgreSQL, detects statistical anomalies with z-score, and emits a structured JSON report.
Work with AI · Investigate
Log Analyzer
Build a Python log-analysis tool that extracts incidents and patterns from noisy, semi-structured logs and produces structured diagnostics.
Work with AI · Investigate
Chat Conversation Analyzer
Build a command-line tool that explores chat log files and produces a report, extracting intent, topics and participants from messy dialogue.
Work with AI · Create
Task Scheduler API
Implement a FastAPI task scheduler with dependency management, cycle detection, topological ordering and critical-path scheduling.
Work with AI · Create
Gallery Wall Curator
An optimisation and packing problem: choose paintings to hang on a wall to maximise total value, designing and iterating heuristics with AI toward value thresholds.
Work with AI · Create
React Carousel Music Player
Build a React carousel music player with audio playback: carousel layout, track metadata, cover art, play and pause, next track and progress scrubbing.
We score the judgment, not the keystrokes
Letting a candidate use AI is easy. Seeing whether they should have trusted it is the hard part, and the part that matters.
Some AI-Native tasks are built deliberately so that the most common AI copilot suggestions are among the wrong answers. A candidate who prompts the assistant, takes the first answer and moves on loses points. A candidate who reads it, doubts it and corrects course scores. The Debug tasks are the sharpest example: the assistant’s first answer is often the wrong one, and the whole signal is whether the candidate catches it.
That is the difference between watching someone use AI and measuring how well they use it.
The signal is not whether they use AI. It is whether they catch when it is wrong
Mapped to the Engineering Skills Model
Every AI-Native task is tagged against the Codility Engineering Skills Model, our research-backed framework for the skills modern engineering jobs require. It spans technical skills, problem-solving, and the professional skills engineering work depends on, and it carries a dedicated Artificial Intelligence category:
That AI category is why AI-Native tasks map straight onto the model rather than bolting on beside it. Tagging happens at the test-case level, so each task reports against the specific skills its checks actually exercise, each at a defined proficiency level. The Build AI tasks report on AI Engineering skills such as LLM Development and AI Evaluation, alongside Context Engineering. The Debug and Investigate tasks report on AI Output Validation and Critical Thinking.
You get the same skills-based reporting for AI-Native tasks as for the rest of the Codility library, so results slot into the role definitions you already use.
A real IDE, with a real AI assistant
AI-Native tasks run in the same environment your engineers work in: a full VS Code IDE with terminal access, package managers, debugging, multi-file projects, and an AI assistant in the loop. Not a stripped-down editor with a Run Tests button. The environment is part of the assessment, because you cannot measure judgment of AI output unless the candidate actually has an AI assistant in a real toolchain.
AI access is configurable per assessment:
AI disabled
For roles where you want foundational coding ability without assistance.
AI enabled
Candidates work with the AI assistant, and every interaction they prompt, accept, modify and reject is captured as reviewable AI activity, so you evaluate the quality of the output and the decisions behind it.
Frequently asked questions
What is an AI-Native task?
An AI-Native task assesses engineers the way they work now: in a real VS Code IDE with an AI assistant available. Some tasks ask the candidate to build an AI or ML system as the deliverable. Others give them an AI assistant as a co-worker on a realistic engineering job and assess how well they direct it and when they override it. Either way, the score reflects engineering judgment, not whether they reached for AI.