Is Your Technical Hiring Tool "Valid" for Your Recruitment Process?

Industrial-Organizational (I-O) psychologists rarely get fired up about things. Counting myself among them, I-Os are a fairly staid bunch. For instance, we gladly identify as “scientist-practitioners” and think it’s cool. But many I-Os had something to say about HBO Max and CNN Films’ recent documentary on pre-hire assessments called Persona: The Dark Truth Behind Personality Tests.

Persona makes a compelling, yet, overly stylized and one-sided argument that personality tests hurt people when companies use them for pre-employment screening. The film focuses on one viral personality test: the Myers-Briggs Type Indicator, or MBTI. The MBTI’s popularity makes it seem OK to use in any situation.

But most I-O psychologists who specialize in scientifically valid candidate assessments would agree with Persona’s creators that the MBTI is an ineffective hiring tool. However, most I-Os would disagree with the film that this makes all personality testing or pre-employment screening tests suspect.

Why? Because the MBTI is not valid for hiring.

For the record, the Society for Industrial-Organizational Psychology, or SIOP, released a formal statement countering the documentary, stating that “…the documentary makes several assertions that are inaccurate, potentially undermining good science and practice involving…personality assessments in the workplace.”

While a kerfuffle over personality testing might seem silly, Persona’s claims and SIOP’s response shines a light on why we need to make hiring decisions with tools based on “good science.” And why you need to make sure the tools in your recruitment process are indeed valid for hiring.

What makes a test “valid,” and why does it matter?

Remember SIOP? According to their principles on personnel selection procedures, a test, whether it’s a resume screen, an interview, or a candidate assessment, is “valid” for hiring if it is “job-related.”Developers and users establish a hiring tool’s job-relatedness through an evidence-gathering process called “validation.”

Validation strategies include describing how the tool was developed to represent a job or asking actual job experts to sign off on an assessment’s job relevance. Validation also includes data-driven studies to find relationships between scores on a test and job performance metrics such as supervisor ratings or sales.

With these validation strategies in mind, it’s no wonder that the MBTI isn’t informative or appropriate for hiring. The MBTI wasn’t developed for hiring, its theory doesn’t relate to work, and it doesn’t correlate well with job performance.

But the knocks against this one assessment don’t mean that you should avoid all assessments for hiring. That would be like avoiding hammers because someone used one to make an omelet – it took several eggs and was really messy.

Read more like this: Bringing Industrial & Organizational Psychology to Tech Hiring.

So why does validity matter? Non-validated hiring tools are as effective as flipping a coin. They also have unintended consequences on your employer brand and new hires’ effectiveness on the job. Poor decision-making and negative candidate perceptions can cripple a hiring tool’s return on investment through increased recruiting costs, higher turnover, and lower productivity.

Job-irrelevant hiring tools also open a business to costly and demoralizing legal challenges. In the U.S., the Equal Employment Opportunity Commission evaluates corporate hiring practices. It prosecutes complaints that employers made discriminatory decisions based on job irrelevant factors protected by law, such as gender, race/ethnicity, national origin, and age.

And remember, we’re not just talking about decisions based on pre-employment screening assessments. Candidates and regulatory agencies can challenge any method used to make hiring decisions, including resume screens, reference checks, and interviews. In other words, technically speaking, you should validate every technique you use for hiring.

Are your technical hiring tools valid? How do you know?

So maybe you’re assuming right now that all of your hiring methods are “job-related.”

But, let me challenge you with a few questions:

• Do you know how hiring managers or recruiters set minimum qualifications on job descriptions and if those qualifications are appropriate for the role?
• Do managers conduct their interviews differently, potentially asking job irrelevant or biased questions?
• Does your recruitment process use technical skills tests developed internally several years ago before the job changed significantly?

If you’re answering these questions honestly, you can see why many organizations need to have a critical eye towards decision-making tools in their recruitment process. Every scenario and recruitment process is different.

So for most users who have already implemented hiring tools into a recruitment process, properly validating your hiring procedures means working with your vendors’ I-O psychology team – or hiring independent I-O consultants who specialize in selection.

SIDEBAR: Technical nitpick – I’ve used the phrase “validating a test” as a convenient but technically incorrect shorthand. According to validity’s definition, a test isn’t valid or not valid. Instead, a developer’s or user’s intended score interpretations are valid for a use case. Validating test scores for specific uses and contexts makes validity a never fully settled topic and validation a continual, ongoing process. 

However, prevention is worth a pound of cure and something you can do on your own. So here are a few simple steps you can take to evaluate whether the tools you use are valid for hiring before you procure them:

1. Ask validity-related questions.

If you use hiring technologies such as resume screeners, interview platforms, or pre-employment screening assessments to automate and scale your recruitment process, make sure you choose vendors who have a transparent approach to validation. To help you evaluate their practices, here are two questions to ask:

• “Do you have a technical manual demonstrating that the tool is reliable and valid in similar contexts to ours?”
• “What strategies do you recommend for validating the tool in our context and use case?”

It should be a red flag if your current vendor can’t answer these questions. If that’s the case, you may want to get an outside auditor or source a new vendor that values gathering valid evidence for their clients.

2. Look for validity-related resources and signals when purchasing a tool.

If you have yet to purchase a tool or need to source a new one, make sure to ask for resources that demonstrate that your vendor understands why validity is essential and has a validation track record with previous customers. For instance, ask your vendor if they have a technical manual on file, previously documented validity studies, or I-O psychologists on staff.

Even if you can’t fully evaluate these materials, your vendor should provide them if they are helping you make distinctions or declarations about job candidates you’re using to make a hiring decision.

3. Hire validation experts or opt-in for validation services.

This recommendation is a part of Codility’s approach to validating how we assess technical skills. We believe that our customers should have access to our past documentation, which details how we develop test content with actual job experts and hiring use cases in mind. We also provide validation services to confirm that our assessment(s) are valid for your jobs.

You should expect this from any hiring technology vendor! If you need a validated approach to hiring engineering talent, learn more by visiting our Knowledge Hub for more information.

Using tools and procedures that have established connections to the jobs at hand can help you make sure your recruitment process is effective, fair, and well-regarded by candidates. Not only is it the right thing to do, but it also protects your company and makes sure you get a return on your investment!

Want more informational content hiring trends? Sign up for our newsletter here & never miss a beat.