Don’t build your assessment house on quicksand

April 3, 2024By Paul Boatman, Ph.D.

Assessment takes many forms, but at its most basic level the goal is always the same: use relevant information to categorize results and then make a decision. We use assessment everywhere and are engaged in it daily. For example, we conduct an assessment when deciding what car to buy, who to draft for our fantasy football team, or which Disney character we are most similar to. The assessment of talent is no different. We read time and again about the importance of human capital and selecting the right employees. This means that whatever approach you take to assessing talent, the tool (or tools) you choose must match your needs.

In the context of talent management, types of assessments vary widely from knowledge tests to work samples to cognitive ability tests to personality inventories. Unfortunately, assessments vary as much in quality as they do in type. Because assessments provide the foundation for critical workplace decisions such as determining who to hire and promote and identifying training needs, it is important that they be solid. After all, you wouldn’t build your house on a foundation of quicksand.

But with all of the different vendors and types of assessments, howcan you know which assessments provide a solid foundation? And how can you know which ones are right for you? There are at least 3 important factors to consider when choosing an assessment: 1) Is it reliable? 2) Is it valid? 3) Is it fair?

Reliability

Reliability is a measure of how consistent the assessment is – are you likely to get the same result each time? It is critical to establish because an unreliable assessment can’t be trusted. In the context of talent, the purpose of an assessment is to help measure or predict whether a person is likely to be successful performing a job. If an assessment gives one result at time 1 and a different result at time 2, how can you as a user know which of the results is a true reflection of the person’s likely job performance?

Validity

Validity involves demonstrating that an assessment is measuring what it’s supposed to be measuring. An assessment can be reliable without being valid. For example, your bathroom scale can give you a consistent reading day after day, but if it’s not set correctly, the weight it shows is always off by the same amount. If reliability is established, in this case, validity might be determined by checking your weight on other scales to see if the numbers match, or by weighing multiple people to see if the resulting weights make sense.

An important aspect of validity is job-relatedness. Assessments need to be validated specifically for their intended use. For example, an assessment validated as a medical diagnostic tool should not be used for any purposes other than that, and an assessment validated to measure personality should not be used for hiring customer service workers unless it has been validated specifically for customer service selection.

Beyond asking about the intended use of the assessment, ask about how the assessment was created. Were subject matter experts included in the process? Were statistical analyses used to make sure bad questions were thrown out? Once the assessment was created, was it tested on different groups of people doing the job? Was there a strong relationship between assessment scores and performance on the job? Was the asse

ssment able to accurately identify those who would perform well in the future? In each case, the answer should be yes and there should be documentation for each of these areas discussed.

Finally, any assessment used for making decisions about people should be unbiased. For work-related talent assessments, this generally means that the assessment should not discriminate based on race, ethnicity, gender, or age. Assessments should be carefully written so they do not contain language that would be interpreted or reacted to differently by different groups, and they should be regularly evaluated to ensure different groups do not receive systematically different scores.

A Solid Assessment Foundation

Choosing a solid assessment for talent decisions is critical, and using the wrong assessment can be unhelpful at best and damaging at worst. It can be intimidating to evaluate all of these factors when choosing an assessment, and you might feel unqualified to do so. As a rule of thumb, we recommend working with an assessment provider that utilizes industrial/organizational psychologists in both the design and implementation of the assessment. I/O psychologists specialize in considering and evaluating these factors and will ensure that your assessment house has a solid foundation. Infor Talent Science does exactly this.

Our Point Of View series: