From Pilot to Practice: Implementing AI for Test Form Assembly

I.C.E. Exchange Program Description : In 2024 and early 2025, a collaboration between multiple vendors and a credentialing organization led to the development and validation of an automated test form assembly tool. This tool was designed to accommodate a wide range of blueprint, psychometric, and content parameters, including AI-based semantic similarity for automated identification of enemy items.

To evaluate its effectiveness, we conducted a pilot study in which test forms assembled by psychometricians were deployed alongside forms generated by the automated tool. A third-party psychometrician analyzed the performance of all forms and compared the results. The findings showed that the automatically assembled forms performed at the same level as those created by psychometricians.

Following this validation, the credentialing organization and other stakeholders adopted the tool for full-scale implementation. Additionally, a second pilot is planned for Q2 2025 to validate the tool further with comparative data.

This session will explore the critical role of partnerships between technology vendors, psychometric vendors, and credentialing organizations in piloting and validating AI-driven assessment tools. We will discuss best practices for conducting pilot studies to ensure the reliability and validity of AI applications in test development. Finally, we will examine key challenges in this initiative and others typical of these types of AI initiatives.