Key Concerns About Fairness in Testing: Applications of Best Practice in the Development of Large-Scale Achievement Tests

Abstract

This chapter discusses the processes that test developers typically follow to help ensure that tests are fair, offers recommendations for how fairness procedures should be conducted, and suggests approaches and considerations that may improve the processes. These processes incorporate item reviews at multiple stages of test development, including early engagement in the articulation of design features, test specifications, and item development. The chapter also includes a discussion of the empirical methods that are used to investigate fairness issues, such as differential item functioning. The authors discusses how procedures could be expanded to incorporate more focused activities, such as expert judgments, focus groups, and cognitive labs to provide more actionable information for test developers. Potential barriers for moving best practices forward include the proprietary nature of fairness practices as defined by different testing organizations, as well as a reluctance to broadly disseminate this information if it might reflect poorly on an assessment. Finally, new aspects of test development, such as automatic item generation, artificial intelligence used for the scoring of open-ended responses, and applications of machine learning, should be more closely studied with respect to fairness issues.  

Keywords: large-scale assessment; test development; development practice