Be the first to know
Sign up for a weekly dose of tech hiring news and updates.
In the rapidly evolving landscape of software engineering, generative artificial intelligence (AI) has sparked a paradigm shift. From streamlined code generation to automated tests, generative AI is transforming how engineers develop and maintain software. A question on everybody’s mind is, “Will generative AI replace programmers?” — so far, the resounding conclusion is that it won’t.
Generative AI has revolutionized engineering by offering powerful tools that boost developer productivity and engagement. As a result, AI has accelerated the software development process, allowing engineers to focus on the tasks they enjoy the most – tasks that require complex problem-solving and innovation.
It’s these intrinsically human skills that will both prevent generative AI from “replacing programmers” and empower engineers to guide AI tools to deliver a competitive advantage. But what exactly are these human skills? And how do we know if someone has them?
Over the past three years, Codility’s I/O psychology team researched the skills that are critical for effective software engineering across a range of roles, job families, and industries. Recently, we turned our focus to studying which skills are particularly essential given the rise of generative AI. We conducted interviews with experienced engineers, gathered training curricula, and scoured emerging academic literature across several disciplines.
We’re excited to launch the industry’s first-ever assessment of AI-Assisted Engineering skills. Fueled by assessment science, with a strong focus on validity and measurement precision, this assessment represents an important first step in accurately evaluating the skills that humans need to successfully collaborate with generative AI.
You might also like: Codility on ChatGPT and the Future of Technical Assessments
What engineering skills are we measuring, and why?
Codility’s 12+ years of experience evaluating the skills of over 13 million engineers combined with our expertise in how generative AI-assisted engineering works means that we have a clear view of the skills that will underlie effective engineering practice — for today, and into the future.
Supporting this, our I/O psychology team’s expertise in work analysis enables us to systematically break jobs down into their component parts to address critical questions such as:
- What tasks are engineers doing?
- How do they perform these tasks?
- What skills do they need to perform these tasks well?
Based on this experience and expertise, we can identify the most critical skills for job performance, on a specific team, and for a specific company. For jobs requiring AI-assisted engineering, we identified around a dozen particularly important skills.
Three of these skills are necessary for effectively working with the current versions of generative AI. These skills include:
- Computational thinking
- Prompt writing
- AI proofreading (i.e., evaluating an AI-generated output).
Codility’s first iteration of our AI-assisted engineering assessment focuses on measuring these skills, which we describe in more detail below.
1. Computational Thinking
Computational thinking is a critical skill to have when engaging in AI-assisted programming because it bridges the gap between human engineers and AI systems’ powerful capabilities. It involves formulating problems in a way that can be solved by a computer—i.e., breaking down problems into smaller, manageable steps, identifying patterns, devising and automating processes to find solutions, and evaluating and iterating to improve those solutions. Computational thinking enables engineers to steer generative AI toward producing effective and efficient outputs and troubleshoot and adapt when unforeseen scenarios are encountered.
As the algorithms and technology behind generative AI continue to advance, interacting with computers will become increasingly abstracted away from the nuances and complexities of coding in a computer language and toward a more human-centered natural language. This will enable engineers to spend less time writing and troubleshooting code and more time designing and delivering solutions. However, the full potential of generative AI will only be realized by an engineer who can devise an appropriate plan and set of instructions that the AI can follow – in other words, by an engineer who has a high proficiency in computational thinking.
2. Prompt Writing
Prompt writing involves crafting prompts or commands to guide the AI’s generation process, making them a critical aspect of how users interact with AI. The way a user formulates prompts can significantly influence the AI’s output. For instance, providing more context in the prompt or asking the AI to work step-by-step can often lead to more detailed and accurate outputs. Conversely, vague or ambiguous prompts can lead to inadequate or incorrect outputs.
We note that generative AI’s future sophistication is very speculative. Some propose that these models might reach a level of sophistication where they can understand and fulfill a user’s objectives, even from rudimentary and unsophisticated prompts. This advancement could diminish the importance of prompt writing as a skill. However, in the current landscape of AI technology, mastering prompt writing remains an essential skill for effectively interacting with generative AI.
3. AI Proofreading
AI proofreading focuses on critically evaluating AI-generated output. This multifaceted skill helps software engineers evaluate AI-generated output’s accuracy, reliability, and compliance with project requirements and industry standards. While generative AI models are powerful tools, their outputs are often far from error-free—even the most state-of-the-art LLMs “hallucinate” or generate wrong or buggy code. Skill in AI proofreading enables software engineers to identify errors, bugs, inconsistencies, or irrelevant results. In addition, this skill helps engineers evaluate whether AI-generated code or text aligns with specific project requirements and coding conventions. This is crucial to maintaining code consistency and coherence within a software project.
Finally, by applying skilled AI proofreading, engineers can provide valuable feedback to the generative AI model and improve its performance, resulting in more suitable and reliable output. As such, this skill fosters an iterative feedback loop between engineers and AI, ensuring the AI system becomes a more dynamic, integrated, and increasingly valuable assistant to the software development process.
The skills that are paramount today may be replaced as AI technology and our capabilities to collaborate with it change. Codility will continue monitoring for developments and innovating how we accurately and fairly assess these skills while giving candidates the high-quality, job-relevant assessment experience Codility is known for.
Conclusion
Generative AI gives engineers exciting new abilities to solve more challenging and business-critical problems and pushes the skills required for software engineering into new and exciting directions. These new opportunities come with the need for the right expertise to understand what these skills are, how they manifest, and how we can accurately measure them.
Codility’s new AI-Assisted Engineering assessment is the next step in our journey to providing customers with the most innovative and robust solutions on the market for embracing generative AI. We are releasing new stand-alone tasks tagged to these skills, so be sure to check those out as well!
Book a demo with us to check out our new AI-Assisted Engineering assessment, as well as the rest of the products in the Codility Evaluation Engine. Current customers can contact their Codility representative to learn more about our Generative AI assessment features today.
Meet The Team:
