Get started

Developing features for products that programmers use is always a challenge–not only do they notice quickly whether the interface works properly or not, but they usually have a pretty good idea of how it could be done better. That is why at Codility we put great emphasis on making sure we develop the right things the right way.

There’s also the fact that we’re developers ourselves, and we know that programmers’ time is precious. We know from our own (painful) experience that coding new features and then hastily pushing them to production just to get them out the door will come back to bite us. Doing this inevitably forces us to backtrack to enact a laundry list of product improvements to address the negative feedback end users surface.

And to top it all off, like many other SaaS startups we face certain limitations. We can’t invest too much money on user research and we definitely can’t afford to spend too much time on it either. The quicker we develop things, the quicker we can improve the user experience, which will drive more business.

So what do we do?

We organize one day Candidate Labs, intensive prototype testing with end users to check our assumptions at the crucial stages of development.

Let’s talk about how we plan and execute these Candidates Labs.

Engineer coding
Running Candidate Labs helps us understand how coding assessments can be improved.

The Candidate Lab Play-by-Play

Step 1: The Setup Stats with a Technical Interview

We don’t have anything as fancy as a viewing facility in our office. Instead, we connect two conference rooms using Google Hangouts. Then we start the technical interviews–the respondents (in our case, professional developers we label “candidates”) come in one at a time, sit down with a researcher and a developer, and find out what we are working on and how they can help us by solving a Codility task in the meeting. We also tell them what feedback we are fishing for specifically so we can dive in.

Why do we include an internal developer of our own in this conversation? To cultivate engagement and help the candidate feel comfortable. Trust me, it’s an interaction way out of the comfort zone for engineers–it’s something that makes their pulse quicken. But it also helps build a connection with the respondent, which will be important for a later part of the Candidate Lab.

Step 2: The Coding Assessment Exercise

The respondent is then asked to solve a coding task and navigate through a specific area of the interface without us in the room (after all, coding is rarely a social activity). They usually interact with a mock-up, i.e. a very limited and only partially functional development that mimics the final solution. It usually takes a few days to a week to build, but it’s worth it.

As the coder traverses the interface, the researcher and the internal developer move to another conference room where they can observe the candidate’s screen and every action they take. For this, we use screen-sharing or sometimes more sophisticated software like Lookback. Just be sure the respondent consents to their work being monitored during the exercise. The researcher and internal developer are often accompanied by other developers and designers involved in the tested project. They all observe, learn, and discuss the findings in real-time. If the mock-up allows it, the user research crack team can even introduce changes live so that the following candidate works out of a freshly improved prototype.

What we gain from the exercise:

  • As close to a real-life user experience as we can get at this stage of development.
  • A preliminary scaffolding on which we can later build the final solution.
  • A huge time-saver because after all, even if we decided to give up the development after the preliminary research, it’s better to do so after a week of a developers’ work than after a month or more.
Two people meeting at a desk with a laptop and monitor
The Coding Assessment Exercise mimics a real-life user experience.

Step 3: The Feedback & Discussion Follow-up

Once the coding portion of the technical interview is finished, the researcher and the internal developer reconvene with the candidate for feedback and discussion. Again, having the internal developer present for this part is also helpful. The researchers are rarely developers themselves, so it’s easy for them to miss a piece of information that could be useful. The developer also knows all the questions that their team might have about the tested product. If they hear anything useful when talking to the candidate, they can redirect the conversation to collect more data for the background team.

Step 4: Rinse, Repeat, and Reap the Learnings

Each Candidate Lab we run consists of approximately five technical interviews – we try to do them one-after-another so that we gather quick feedback and start development the very next day. We discovered that doing this was more time-efficient than doing technical interview sessions here and there, stopping our work every time and taking pause to frequently discuss results from individual sessions.

We wrap-up every Candidate Lab by analyzing the findings and categorizing them based on their significance. For instance, is it crucial for the feature’s performance? How long would it take to do implement? We’re always aiming for the low-hanging fruits.

One example of this, while testing Codility Angular Tasks, we realized that the ‘Run’ button was confusing Candidates. Some of them believed it initiated running tests and others thought it referred to running and submitting the finished code. Renaming the button to ‘Run Tests’ took an hour of work, but it was a fairly quick win to improve the candidate user experience and overall performance.

We did the same thing with changing task introduction to be more specific in the new interface. It was also a great opportunity to learn that the way we decided to test for Angular skills received extremely positive feedback from front-end engineers. It’s such a boost to morale to hear such great reactions after weeks of work.

Some Candidate Labs reveal crucial errors in the way we think about our own product, while others might surface minor cosmetic improvements. Whatever we learn, however, always translates to a better end-solution and helps us understand how we could do things better next time.

Coffee mug on table
Get insights to improve technical recruiting by running your own Candidate Labs.

Tips & Tricks To Help You Get Started

Here is some practical advice that might help to get you started with the Candidate Labs in your organization:

  • Make sure both the team and the researcher understand what the goals of the Candidate Lab are. What questions do you want to answer? Where are your doubts? What do you need to know once the interviews are over? All this is crucial for obtaining useful data.
  • Remember to recruit respondents that fit your end user profile. Where will you find them? Ideally you’re physically located near your targets. You test a product for mothers? Visit playrooms or shop alleys with toys or children items. Looking for dog lovers? Parks are the place for you. You can always ask friends to look for people that fit your ideal profile in their network. In case of very difficult target groups, you can consider using services like a professional recruiting agency. With time, you can even consider building a Community of your own (which we will talk about in the future blog postings).
  • You don’t necessarily need to do a technical interview on-site. Sometimes on-sites don’t fit the features you are developing currently–some projects are tested better in a remote technical interview. Make your respondents feel like valuable reviewers and co-authors of the solutions. They are not lab rats!
  • Work on your discussion guide beforehand and make sure it’s full of open-ended questions.
  • Try to involve people working on the tested project so that they can directly interact with respondents. It helps to reach consensus on the findings and potential solutions.
  • Feed your co-workers if you book them for longer interview blocks. Nobody thinks well when hungry and the challenging schedule rarely leaves time for coffee or lunch breaks. And since all this is revolving around creating a greater user experience anyways, provide the same courtesy to the respondents too because they often participate straight after work. Remember, snacks can also ease the stress that sometimes accompanies the interview.
  • If your respondents agree, record the interviews for further analysis. It’s helpful to be able to go back and replay the details while implementing specific features inspired by the research.

But most of all, enjoy the rare moments when you get an inside look on how your product is perceived and how it can be made better–they are precious!

Check out other technical recruiting insights from the Codility team!

Agnieszka is involved in Product Research @Codility – bringing to the table 14 years of research experience, she focuses on product performance, new feature development, and data analysis. She uses in-depth knowledge of both qualitative as well as quantitative research methods to engage the team in insight-driven decision making about Codility developments.

Connect on LinkedIn