Request a free trial

AI engineer interview with a live AI build

Codility Interview is a live technical interview platform where candidates build AI components in a shared VS Code workspace. Your interviewers observe a task, such as building a retrieval-augmented generation (RAG) pipeline, and score the session against your rubric.

Request a free trial

The Codility team contacts you to set up trial access.

A candidate building a RAG pipeline in a live Codility interview. AI engineer interview
00:34:12
CCandidate
IInterviewer
End interview
Tasks
‹1›Legal Document Q&A…
Add task
System design
Whiteboard
Tools
AI Assistant
Skills evaluation
Notes & Questions
Explorer
chain.py
retriever.py
▾ legal_qa
__init__.py
chain.py
ingest.py
retriever.py
▸ data
▸ tests
requirements.txt
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
from langchain_core.prompts import ChatPromptTemplate
from .retriever import load_index, retrieve
PROMPT = ChatPromptTemplate.from_template(
“Answer only from the context. Cite sources.\n”
“Context: {context}\nQuestion: {question}”
)
defanswer(question: str, k: int = 4) -> dict:
index = load_index(“data/faiss_index”)
docs = retrieve(index, question, k=k)
context = “\n\n”.join(d.page_content for d in docs)
reply = llm.invoke(PROMPT.format(context=context, question=question))
return {“answer”: reply.content, “sources”: [d.metadata for d in docs]}
TerminalProblemsOutput
candidate@interview:~/legal_qa$ python -m legal_qa.ingest data/archive
Loaded 212 documents, split into 3904 chunks
Built FAISS index: data/faiss_index
candidate@interview:~/legal_qa$ python -m legal_qa.ask “Who signs off on renewals?”
Retrieving 4 chunks…

Engineering teams at companies including GitHub, SAP, Barclays, American Express, EY and Unity use Codility.

G2 Leader Enterprise, Fall 2026 G2 Leader Mid-Market, Fall 2026 G2 Best Est. ROI Enterprise, Fall 2026 G2 Most Implementable Enterprise, Fall 2026

Codility is rated 4.6 out of 5 on G2 and ranked #1 for Enterprise Technical Skills Screening. Rankings come from verified customer reviews.

Today’s failing approach: no live build in the AI engineer interview

Challenge 1: Algorithm rounds miss AI engineering skills

A generic coding round asks the candidate to solve an algorithm problem. AI coding tools can solve this type of problem. The round also gives no signal on whether the candidate can build a retrieval pipeline.

Challenge 2: The design round shows no implementation

In an AI system design round, the candidate explains how a system such as a chatbot with guardrails would work. Your interviewers hear the design but do not see whether the candidate can implement any part of it.

Challenge 3: Take-home projects hide the build process

When a take-home process captures only the submitted code, your team cannot review how the candidate reached the result. Preparing and sending custom tasks also takes an engineer’s time.

Hire AI engineers based on a live build in Codility Interview

Your team can run the AI build and the system design discussion in one Codility Interview session. The candidate discusses the architecture on a shared whiteboard alongside the code. You can choose a Codility AI-native task or have your engineers create a custom task for the role.

Choose an AI build task for the interview

  • Build AI tasks in the Codility AI-native task library make an AI system the deliverable. For example, in the Legal Document Q&A Assistant task, the candidate implements a LangChain and FAISS RAG pipeline. The pipeline answers questions about a legal document archive and returns its sources.
  • The DataSentinel task suits AI engineering roles closer to machine learning. In this task, the candidate builds an anomaly-detection pipeline for daily KPI readings and stores the reading history in PostgreSQL.
  • Your engineers can also create a multi-file task with the Codility Model Context Protocol (MCP) connector. An engineer describes the task to their own AI coding assistant, reviews the generated task and publishes it in Codility.

The difference: The candidate builds the kind of AI system your team ships.

TerminalProblemsOutput bash · legal_qa
candidate@interview:~/legal_qa$ python -m legal_qa.ask “What notice period applies to ending the supplier agreement?”
Retrieving 4 chunks from data/faiss_index…
Answer
Either party can end the supplier agreement with 90 days’ written notice. Notice must go to the registered address named in the agreement. [1][2]
Sources
[1] contracts/supplier_agreement_2023.pdf p. 14 §12.2 0.87
[2] contracts/supplier_agreement_2023.pdf p. 22 §18.1 0.81
[3] amendments/supplier_amendment_02.pdf p. 3 §4 0.74
[4] policies/vendor_management.pdf p. 9 §3.6 0.62
candidate@interview:~/legal_qa$
Explorer
chain.py
retriever.py
test_retriever.py
▾ legal_qa
__init__.py
ask.py
chain.py
ingest.py
retriever.py
▸ data
▾ tests
test_chain.py
test_retriever.py
docker-compose.yml
requirements.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
from langchain_community.vectorstores import FAISS
from .embeddings import embed
MAX_DISTANCE = 0.9
defload_index(path: str) -> FAISS:
return FAISS.load_local(path, embed)
defretrieve(index: FAISS, question: str, k: int = 4):
if not question.strip():Candidateraise ValueError(“Empty question”)
hits = index.similarity_search_with_score(question, k=k)
return [d for d, s in hits if s < MAX_DISTANCE]Interviewer
TerminalProblemsOutput
candidate@interview:~/legal_qa$ pytest tests -q
……F [100%]
FAILED tests/test_retriever.py::test_empty_question
6 passed, 1 failed in 2.41s

Watch the candidate build in a realistic VS Code environment

  • The candidate works in VS Code with a terminal, package managers, debugging and multi-file projects.
  • Your interviewers can attach a database, including PostgreSQL, MySQL, MongoDB and Redis, when the task needs stored data.
  • Your interviewers can observe the code as the candidate writes it, or edit the same code alongside the candidate.

The difference: You can see whether the candidate can implement the AI system they describe.

Score the AI build against the skills the role needs

  • Codility supports automated test-case scoring on selected library tasks.
  • Before the interview, your team agrees on a rubric, for example on retrieval quality and failure handling. Your interviewers use the same criteria to score each candidate’s session.
  • When configured, recording, transcript and playback let your hiring committee review the candidate’s reasoning after the session.

The difference: Your hiring committee can compare AI engineer candidates on the same rubric and review the session record before deciding.

Request a free trial
InterviewsAI engineer interview
Share report
Session recordRecorded
def retrieve(index, q, k=4):
if not q.strip():
raise ValueError
hits = index.similarity…
21:4058:12
Whiteboard Build Discussion
Transcript
21:40Interviewer: How would you evaluate answer quality?
21:52Candidate: I’d keep a labelled question set and check that each cited source contains the answer.
Rubric scoresScale 1–4
CriterionInt. 1Int. 2
Retrieval quality 4 3
Failure handling 2 3
Answer evaluation 3 3
System design 3 4
Communication 3 3
Interviewer notes Handled empty input after the first test run. Ask about malformed model output in the debrief.

Questions about AI engineer interviews

An AI engineer interview is a technical interview for engineers who build applications on large language models (LLMs), such as RAG pipelines, agents and evaluation systems. Teams also call it an LLM engineer interview. The interview loop often includes a technical screen, a build round or take-home project, and an AI system design round.

Run your next AI engineer interview in Codility

Ask to see the Legal Document Q&A Assistant task and its scoring criteria before choosing a task for your interview loop.

Request a free trial

Request a free trial

Ask to see the Legal Document Q&A Assistant task and its scoring criteria.