Be the first to know
Sign up for a weekly dose of tech hiring news and updates.
Following the release of ChatGPT-4 earlier this week, AI and its potential to game technical assessments have never been more relevant. And we’re sure you’re wondering – can candidates use ChatGPT to pass technical assessments like Codility? As innovators (and inventors) of the technical skills assessments category, we’re leading the way in research on this topic.
Since 2020, Codility has worked with OpenAI, exchanging code and insights on generative AI and its relation to coding assessments. We were among the first to participate in beta testing for relevant AI technologies, including GPT-2, GPT-3, Codex, and, you guessed it, ChatGPT.
Our I/O psychology and content development teams have also been ahead of the curve in actively researching AI’s applicability, impact, and benefits to technical assessment strategies and are currently developing tasks that require candidates to collaborate with AI to embrace this shift. We also conducted extensive testing of our task library to identify what makes tasks more or less solvable by ChatGPT. The results?
Codility offers the strongest and only ChatGPT-resistant solution in the market. Our three-pronged approach to addressing generative AI with prevention, detection, and innovation bolsters our initiatives even more.
- Prevention: We are committed to preventing cheating by providing deterrents to candidates and maintaining a task library resistant to ChatGPT in all its evolutions.
- Detection: We are expanding the available mechanisms to give our customers visibility into likely indicators of ChatGPT use in assessments.
- Innovation: We are embracing generative AI and its promise to unlock the world’s engineering potential.
In this article, we’ll share more about how we ensure our customers can create ChatGPT-resistant assessments and why Codility customers hold the strongest position in the market in maintaining the integrity of their technical assessment programs.
The First & Only ChatGPT-Resistant Task Library
We view generative AI and ChatGPT as partners in our mission to unlock the world’s engineering potential. But understandably, there’s also some interest in preventing candidates from using ChatGPT in coding tests.
Since 2009, our assessment science experts have carefully shaped each element of our proprietary Evaluation Engine to ensure fair and accurate assessment of candidates’ technical skills. We’ve leveraged our nearly 15 years of experience to integrate robust features to deter, detect, and prevent candidates from cheating or leaking test content.
For example, when pairing CodeCheck results with CodeLive interviewing, candidates can be asked to demonstrate their thought process in a live format. Our Similarity Check feature also detects if someone submits a solution that was previously submitted to Codility or that can be found online. This means each new solution we receive is compared to the millions of solutions already submitted by candidates or Codility testers, some of which already include elements generated by ChatGPT.
You might also like: Is Your Technical Hiring Tool “Valid” for Your Recruitment Process?
We also know that the best way to prevent cheating of any kind is to have a high cadence of fresh, expertly developed content – so we have completely changed how we develop content to make it even more resistant to cheating with AI. Codility has the most tenured internal task development team in the market, and from Q4’22 to Q1’23, we doubled production and release of new tasks.
Equipped with findings from our extensive research and testing on how ChatGPT handles different types of tasks and problems, we made changes to our task development process and have increased the production of highly ChatGPT-resistant task types.
Finally, with our three-pronged approach to addressing the generative AI of the future, we will help our customers hire with confidence – no matter what new tech is on the horizon.
Adapting to AI: Prevention, Detection & Innovation
As of March 2023, we are the only leading assessment vendor actively enabling our customers to create AI-resistant assessments. We’ve conducted extensive testing across our task library to determine what tasks and problems it solves well and what trips it up.
We’ve tagged tasks according to ChatGPT solvability, which means that our customers now have the option to create tests that are ChatGPT-resistant or easily identify and filter out ChatGPT-solvable tasks in our entire task library. We are also implementing a robust remote proctoring integration with an industry-leading partner – because we know that lightweight homegrown proctoring solutions will not prevent (or, in many cases, even detect) ChatGPT use.
Extra reading: Hiring and Retaining Tech Talent in 2023
In addition, Codility’s product and engineering teams are laser-focused on researching the impact and exploring new solutions to generative AI tools like ChatGPT. Our newly-formed Evaluations Integrity Team owns the topic of ChatGPT, as well as future emerging technologies. Their job? Continue to monitor trends in this space and engage in proactive experimentation and research that ensures our customers are always prepared and supported.
We also provide our customers with anti-cheating measures recommended by our experienced in-house I/O psychology team, new ChatGPT-resistant task types, robust monitoring for leaked tasks, and similarity and plagiarism detection. If more support is needed, our global Customer Success and Support teams are equipped to make informed recommendations to maintain the integrity of our assessments.
Setting You Up for Success
Codility’s standpoint is that, instead of focusing on how we overcome or prevent AI’s use, we should focus on how we can best collaborate with these tools to unlock human potential.
ChatGPT will likely raise the bar in some ways for what defines a “minimally qualified” software engineer. Generative AI gives engineers new abilities to solve more robust problems and requirements, thus pushing the desired skills for software engineering roles in new and exciting directions. This will require new assessment strategy designs and a recalibration of what proficiency levels are required on day one of a job.
Fortunately, Codility offers the most sophisticated and cheating-resistant assessment content in the market today. We will continue to do what we do best and actively partner with our customers to help them understand and evaluate the new job skill requirements in the future.
Book a demo with us to see our ChatGPT-resistant library at work, as well as the rest of the products in the Codility Evaluation Engine. Current customers can contact their Codility representative to learn more about our ChatGPT-resistant features today.