A classical test theory and item response theory comparison of the brief electricity and magnetism assessment and the conceptual survey of electricity and magnetism

Abstract

For proper assessment selection understanding the statistical similarities amongst assessments that measure the same, or very similar, topics is imperative. This study seeks to extend the comparative analysis between the brief electricity and magnetism assessment (BEMA) and the conceptual survey of electricity and magnetism (CSEM) presented by Pollock. This is accomplished by using large samples (NBEMA ¼ 5368 and NCSEM ¼ 9905) within classical test theory (CTT) and item response theory (IRT) frameworks. For the IRT comparison, after consideration of the conceptual content addressed in each assessment, it was assumed that each of these assessments are measuring the same student latent ability (θ), specifically a student’s ability to do introductory electricity and magnetism. Via a CTT and IRT analysis it was found that both assessments are essentially equal in overall difficulty. Classical item analysis applied to 7 questions used by both assessments revealed that each assessment functions slightly differently internally. The test information curves found from IRT show that the CSEM has superior information compared to the BEMA in estimating student latent abilities for the entire range of typical latent abilities achieved by students on each assessment, θ ≈ −2 to θ ≈ 3. Information in this case is interpreted as how well a student’s latent ability was estimated by an assessment as a function of latent ability. When the circuits questions are removed from the BEMA the majority of the information is lost in the θ ≈ 0 to θ ≈ 2 range. This means the circuits questions on the BEMA are information heavy for higher ability scores. So, special considerations should be made as to which assessment a study uses depending on the specific questions a researcher is attempting to answer.