Transport Validity Evidence

Uniform Guideline Requirements

Transportability is a process sanctioned by the Uniform Guidelines, Section 7B, that allows validity evidence gathered in one setting to be borrowed or “transported” to another setting. Transporting validity evidence is a formal process by which criterion-related validity evidence gathered for a particular job in a particular setting is empirically extended to a similar job in a different setting where no empirical validation has been conducted. Therefore, the purpose of a study of this nature is to provide evidence (for a location that lacks such evidence) that the inference drawn about test scores predicting work outcomes or behaviors is valid.

Section 7B of the Guidelines sanctions this formal process, provided the following requirements are met:

Job Similarity

The incumbents in the user’s job and the incumbents in the job or group of jobs on which the validity study was conducted perform substantially the same major work behaviors, as shown by appropriate job analyses both on the job or group of jobs on which the validity study was performed and on the job for which the selection procedure is to be used.

This first requirement stipulates that the extent to which validity evidence can be transported from one job in a particular setting to another job in a separate setting depends largely on establishing that the jobs in question are sufficiently similar. The Uniform Guidelines states that in order to establish similarity, job analyses must be conducted on both jobs, and a means of comparing these results must be established.

Validity Evidence

Evidence from the available studies meeting the standards of Section 14B of this part clearly demonstrates that the selection procedure is valid.

The second requirement simply states that the selection tool used in the original sample (the evidence to be transported) must show “clear” validity evidence.

Fairness Evidence

The studies include a study of test fairness for each race, sex and ethnic group which constitutes a significant factor in borrowing user’s relevant labor market for the job or jobs in question…. Users obtaining selection procedures from publishers should consider, as one factor in the decision to purchase a particular selection procedure, the availability of evidence concerning test fairness.

The final requirement states that the original study must contain an analysis of fairness in regard to legally protected groups. This evidence should be considered “as one factor” in the decision to use a particular selection procedure. It suggests further that a local analysis of fairness would be optimal, if feasible.

Summary

In summary, a transportability study should include methods that are consistent with the best practices outlined by the Uniform Guidelines. At IO Solutions, we have developed procedures aligned with the Guidelines that allow for a scientific and empirical comparison of job similarity based upon job analysis information gathered from the source and transported samples. Due to the nature of this type of study, most transport studies would be conducted with an “off-the-shelf” exam. Through a continued research and development effort to maintain and update our selection of exams’ psychometric properties, the technical information necessary to fulfill the second and third requirements (evidence of validity and analysis of test fairness) of a transportability study is already available.

Validity is discussed scientifically as a body of evidence accumulated such that resulting test scores may be interpreted for a proposed use; practically, this seems not to be the case as courts seem to favor one type of evidence above others. For this reason, to obtain the most legally robust selection tool, a localized criterion-related validity study should be conducted. However, the use of transportability studies to establish validity evidence for those for which more traditional validations strategies are not practical, feasible, or monetarily viable is a vigilant and reasonable approach. This is the most important take-away regarding transportability studies; through conducting such a study, information and evidence are gathered to defend use of a selection tool where no defense would have been established otherwise. The information that is obtained through a transport study is legally sanctioned and provides a reasonable effort to protect the use of a selection procedure for those who would not otherwise be able to establish validity evidence.

References
Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor & Department of Justice. (1978) Uniform guidelines on employee selection procedures. Federal Register, 43(166), 38290-38315

American Educational Research Association, American Psychological Association & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Education Research Association.

Society for Industrial and Organizational Psychology (2003). Principles for the validation and use of personnel selection procedures. Bowling Green, OH: Society for Industrial and Organizational Psychology.

Binning, J. F., & Barrett, G. V. (1989). “Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases.” Journal of Applied Psychology, 74(3), 478-494.