Purpose and Value of Subject-Matter Experts

Selecting SMEs

The selection of the type of SMEs may vary depending on the purpose of their involvement. SMEs may be incumbents or supervisors. In situations where test content or assessment content is being developed or reviewed, it is most appropriate to use subject matter experts who are supervisors. In situations where the job is being discussed for job analysis purposes, it is appropriate to use incumbent SMEs along with supervisor SMEs.

Subject Matter Experts and the Development and Validation of Selection Tools

Subject Matter Experts are used for various purposes during several phases of the development and validation of selection tools. The fundamental building block in starting the development of any selection tool is a thorough job analysis. According to the Equal Employment Opportunity Commission’s (EEOC) “Uniform Guidelines on Employee Selection Procedures,” in order for selection tools to be valid, they must measure knowledge, skills and abilities that are a “necessary prerequisite to performance of critical or important work behavior(s).” In order to ascertain this information, a job analysis must be conducted.

Job Analysis

During the job analysis phase, SMEs have an integral role in providing information about the job via interviews, on-the-job observations, and job analysis questionnaires (JAQs). SMEs are interviewed to gather information on the tasks and responsibilities required for the job, critical knowledge, skill, and ability requirements, along with any other critical incident information that is relevant. Also, subject matter experts may be observed while they are performing their job to obtain first-hand knowledge and information about the job being analyzed. Utilizing the information gathered from the interviews and observations, a JAQ may be developed to distribute to incumbents, supervisors and any other relevant SME group. The JAQ would require a representative sample of SMEs to rate the importance and frequency of tasks and the importance of various knowledge, skill and ability areas required for successful performance in the job. It is evident that the role of SMEs in the job analysis phase is crucial, and their participation allows for a thorough and accurate understanding of the job. Using the information gathered from the interviews, observations and JAQs, valid selection tools can be developed.

Exam Development, Review and Validation

After data from the job analysis have been analyzed, the essential job knowledge areas of the position are known. In order to develop a custom job knowledge exam, sources (e.g., textbooks, internal policy manuals, state laws, local ordinances, etc.) must be selected that contain the essential job knowledge areas of the position. Subject matter experts are used to help in selecting agency-relevant sources that include the essential job knowledge areas for the position. For example, if it is essential for the position to be knowledgeable on state laws, SMEs would assist in selecting the specific state laws that are most relevant for inclusion in the exam. The sources that SMEs assist in selecting would then be communicated to the candidates. The candidates prepare for the exam by studying the sources on this list. The exam items are developed directly from the sources that were selected and communicated to the candidates.

Another phase in which the use of SMEs is important is in the review of various selection tools, such as custom job knowledge tests. Before a custom job knowledge test is administered, it is important to have a group of SMEs review the questions that have been developed. To review the test, a group of SMEs that are above the rank in which the test will be used would be convened. Using a higher ranking group of SMEs would guard against potential biases and other security concerns that may affect perceptions of the test. In the review session, the SMEs would be asked to review each question and determine job-relatedness, review the content covered in the test to determine appropriateness, and review the wording and terminology to determine relevance. Based on the SMEs’ input, questions may be eliminated that are not job-related or may be reworded to be made job-related.

To assist in validation of the custom job knowledge test, SMEs are asked to provide content-validity ratios (CVRs) for each question. Using a provided rating scale, SMEs individually assign a CVR for each question that represents the question’s essentiality to the job in question. To assist in the determination of a cut score for the custom job knowledge test, subject matter experts are asked to individually provide Angoff ratings for each question. The Angoff ratings represent the percentage of minimally qualified incumbents who would be expected to get a particular item correct. The cut score for the exam would be based on the average of all of the Angoff ratings across SMEs for the items selected for inclusion in the final exam. All of the aforementioned input provided by the SMEs results in evidence that is documented and can be used to support the various decisions that need to be made to administer a job-related and valid test.

Development of Assessment Center Exercises

The development of relevant and job-related assessment center exercises is assured when working with SMEs. SMEs are able to provide relevant scenarios for the job in question that can be used to customize various assessment center exercises (e.g., role-plays, written reports, presentations, interviews, etc.). Using situations that are relevant to a particular agency increases the fidelity of the assessment center exercises. As examples, SMEs can assist with the following: discussing policies that are relevant to include as part of an exercise, providing relevant issues that occur within the agency, discussing critical incidents, and suggesting appropriate language to include that will help to ensure that the assessment center exercises will be perceived by the candidates to be realistic. Additionally, SMEs can assist in determining the appropriate performance criteria that would be expected of the candidates. With the help of SMEs, the performance criteria are comprehensive and in line with an agency’s expectations of the performance of their candidates for an exercise. After the assessment center exercises are developed, SMEs can review the exercises for accuracy and completeness and make any last suggestions before they are finalized.

Getting over the Fear of Using Subject Matter Experts

There is no substitute for the insight that can be attained with the use of internal SMEs for the development and review of selection tools.  Unfortunately, some agencies may be hesitant to allow SMEs to review tools before they are administered for fear that secure test content may be leaked to candidates.  Often this fear is exaggerated.  To remedy the fear of test or exercise content from being divulged to candidates, trusted SMEs from supervisory ranks who are not eligible for promotional testing should be used.  Also, the SMEs who are appointed to help in the review and development of the selection tools should be kept at a minimum number.  The SMEs may be asked to sign confidentiality agreements indicating that the content that they are privy to may not be shared.  Any agency that wishes to utilize job-related and valid selection tools should allow SMEs to assist in various phases of their development and review.

References

Section 60-3, Uniform Guidelines on Employee Selection Procedure (1978). 43 FR 38295 (August 25, 1978); 29 CFR Part 1607.