- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Chained computerized adaptive testing for the Force Concept Inventory

Jun-ichiro Yasuda*

Michael M. Hull

Naohiro Mae

Kentaro Kojima

  • Center for the Studies of Higher Education, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8601, Japan

  • Kyushu University, Faculty of Arts and Science, 744 Motooka, Nishi-ku, Fukuoka, 819-0395, Japan

  • *Contact author: phys.cat.collaboration@gmail.com
PDF

Phys. Rev. Phys. Educ. Res. 21, 020139 – Published 3 November, 2025

DOI: https://doi.org/10.1103/x2rq-z8lm

Abstract

Although conceptual assessment tests are commonly administered at the beginning and end of a semester, this pre-post approach has inherent limitations. Specifically, education researchers and instructors have limited ability to observe the progression of students’ conceptual understanding throughout the course. Furthermore, instructors are limited in the usefulness of the feedback they can give to the students involved. To address these challenges, we propose an alternative approach that leverages computerized adaptive testing (CAT) and increasing the frequency of CAT-based assessments during the course, while reducing the test length per administration, thus keeping or decreasing the total number of test items administered throughout the course. The feasibility of this idea depends on how far the test length per administration can be reduced without compromising test accuracy or precision. Specifically, the overall test length is desired to be shorter than when the full assessment is administered as a pretest and a subsequent post-test. To achieve this goal, we developed a CAT algorithm called Chain-CAT. This algorithm sequentially links the results of each CAT administration using collateral information. We developed the Chain-CAT algorithm using the items of the Force Concept Inventory (FCI) and analyzed its efficiency through numerical simulations. We found that collateral information significantly improved the test efficiency, and the overall test length could be shorter than that of the pre-post method. Without constraints for item balancing and exposure control, simulation results indicated that the efficiency of Chain-CAT is comparable to that of the pre-post method even when each CAT administration includes only 5 items and the CAT is administered 9 times throughout the semester. However, when these constraints are imposed, we found that the efficiency of the Chain-CAT with a test length of five items is lower than that of the pre-post method. Based on this result and analysis of which items were administered, we suggest expanding the FCI item bank by creating additional items or incorporating items from other research-based assessments that have high discrimination parameters and exhibit sufficient variability in their difficulty parameters.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (70)

  1. D. Hestenes, M. Wells, and G. Swackhamer, Force Concept Inventory, Phys. Teach. 30, 141 (1992).
  2. R. R. Hake, Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses, Am. J. Phys. 66, 64 (1998).
  3. A. Pawl, Using force concept inventory data to develop impactful class activities, Phys. Teach. 58, 94 (2020).
  4. B. R. Wilcox, S. J. Pollock, and D. R. Bolton, Retention of conceptual learning after an interactive introductory mechanics course, Phys. Rev. Phys. Educ. Res. 16, 010140 (2020).
  5. E. Burkholder, C. Walsh, and N. Holmes, Examination of quantitative methods for analyzing data from concept inventories, Phys. Rev. Phys. Educ. Res. 16, 010141 (2020).
  6. Y. Xiao, G. Xu, J. Han, H. Xiao, J. Xiong, and L. Bao, Assessing the longitudinal measurement invariance of the Force Concept Inventory and the conceptual survey of electricity and magnetism, Phys. Rev. Phys. Educ. Res. 16, 020103 (2020).
  7. A. Reinhard, A. Felleson, P. C. Turner, and M. Green, Assessing the impact of metacognitive postreflection exercises on problem-solving skillfulness, Phys. Rev. Phys. Educ. Res. 18, 010109 (2022).
  8. C. Wheatley, J. Wells, D. E. Pritchard, and J. Stewart, Comparing conceptual understanding across institutions with module analysis, Phys. Rev. Phys. Educ. Res. 18, 020132 (2022).
  9. A. Morley, J. M. Nissen, and B. V. Dusen, Measurement invariance across race and gender for the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 19, 020102 (2023).
  10. E. Christman, P. Miller, and J. Stewart, Beyond normalized gain: Improved comparison of physics educational outcomes, Phys. Rev. Phys. Educ. Res. 20, 010123 (2024).
  11. M. Jariwala, J.-S. S. White, B. V. Dusen, and E. W. Close, In-class vs online administration of concept inventories and attitudinal assessments, presented at PER Conf. 2016, Sacramento, CA, 10.1119/perc.2016.pr.039.
  12. J. M. Nissen, M. Jariwala, E. W. Close, and B. V. Dusen, Participation and performance on paper- and computer-based low-stakes assessments, Int. J. STEM Educ. 5, 21 (2018).
  13. B. V. Dusen, M. Shultz, J. M. Nissen, B. R. Wilcox, N. G. Holmes, M. Jariwala, E. W. Close, H. J. Lewandowski, and S. Pollock, Online administration of research-based assessments, Am. J. Phys. 89, 7 (2021).
  14. R. S. Siegler and K. Crowley, The microgenetic method: A direct means for studying cognitive development, Am. Psychol. 46, 606 (1991).
  15. D. Kuhn, Microgenetic study of change: What has it told us?, Psychol. Sci. 6, 133 (1995).
  16. E. C. Sayre and A. F. Heckler, Peaks and decays of student knowledge in an introductory EM course, Phys. Rev. ST Phys. Educ. Res. 5, 013101 (2009).
  17. A. F. Heckler and E. C. Sayre, What happens between pre- and post-tests: Multiple measurements of student understanding during an introductory physics course, Am. J. Phys. 78, 768 (2010).
  18. E. C. Sayre, S. V. Franklin, S. Dymek, J. Clark, and Y. Sun, Learning, retention, and forgetting of Newton’s third law throughout university physics, Phys. Rev. ST Phys. Educ. Res. 8, 010116 (2012).
  19. R. Brock and K. S. Taber, The application of the microgenetic method to studies of learning in science education: Characteristics of published studies, methodological issues and recommendations for future research, Stud. Sci. Educ. 53, 45 (2017).
  20. R. Brock and K. S. Taber, Making claims about learning: A microgenetic multiple case study of temporal patterns of conceptual change in learners’ activation of force conceptions, Int. J. Sci. Educ. 42, 1388 (2020).
  21. R. Brock, K. S. Taber, and D. M. Watts, Assembly required: A microgenetic multiple case study of four students’ assemblages when learning about force, Int. J. Sci. Educ. 46, 1027 (2024).
  22. Elements of Adaptive Testing, edited by W. J. van der Linden and C. A. Glas (Springer, New York, 2010).
  23. D. Magis, D. Yan, and A. A. von Davier, Computerized Adaptive and Multistage Testing with r (Springer, New York, 2017).
  24. International vocabulary of metrology—Basic and general concepts and associated terms (VIM) 3rd edition, Technical Report No. JCGM 200:2012, Joint Committee for Guides in Metrology, 2012.
  25. C. N. Mills and M. Steffen, The GRE computer adaptive test: Operational issues, in Computerized Adaptive Testing: Theory and Practice (Springer, Netherlands, 2000), pp. 75–99.
  26. College Board, Assessment Framework for the Digital SAT Suite, version 3.01 (College Board, New York, 2024).
  27. A. Frey, C. König, and A. Fink, A highly adaptive testing design for PISA, J. Educ. Measure. 62, 415 (2023).
  28. J. W. Morphew, J. P. Mestre, H.-A. Kang, H.-H. Chang, and G. Fabry, Using computer adaptive testing to assess physics proficiency and improve exam performance in an introductory physics course, Phys. Rev. Phys. Educ. Res. 14, 020110 (2018).
  29. M. A. Samsudin, T. S. Chut, M. E. Ismail, and N. J. Ahmad, A calibrated item bank for computerized adaptive testing in measuring science TIMSS performance, Eurasia J. Math. Sci. Technol. Educ. 16, em1863 (2020).
  30. M. D. Linderman, S. A. Suckiel, N. Thompson, D. J. Weiss, J. S. Roberts, and R. C. Green, Development and validation of a comprehensive genomics knowledge scale, Public Health Genomics 24, 291 (2021).
  31. R. Zakwandi, E. Istiyono, and W. S. B. Dwandaru, A two-tier computerized adaptive test to measure student computational thinking skills, Educ. Inf. Technol. 29, 8579 (2024).
  32. J. Morphew, A. Mehrabi, B. V. Dusen, and J. Nissen, AI-based concept inventories: Using cognitive diagnostic computer adaptive testing in lasso for classroom assessment, in Paper Presented at 2024 ASEE Annual Conference and Exposition (2024), 10.18260/1-2--46534.
  33. V. Le, B. V. Dusen, J. M. Nissen, X. Tang, Y. Zhang, H. H. Chang, and J. W. Morphew, Mechanics cognitive diagnostic: Mathematics skills tested in introductory physics courses, presented at PER Conf. 2024, Boston, MA, 10.1119/perc.2024.pr.Le.
  34. W. J. van der Linden, Empirical initialization of the trait estimator in adaptive testing, Appl. Psychol. Meas. 23, 21 (1999).
  35. B. P. Veldkamp, M. Matteucci, and T. J. Eggen, Computerized adaptive testing in computer assisted learning?, Commun. Comput. Inf. Sci. 126, 28 (2011).
  36. Y. Choi and C. McClenen, Development of adaptive formative assessment system using computerized adaptive testing and dynamic Bayesian networks, Appl. Sci. 10, 8196 (2020).
  37. A. C. Yang, B. Flanagan, and H. Ogata, Adaptive formative assessment system based on computerized adaptive testing and the learning memory cycle for personalized learning, Comput. Educ. 3, 100104 (2022).
  38. R Core Team, R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria (2024), https://www.R-project.org/.
  39. R. P. Chalmers, mirt: A multidimensional item response theory package for the r environment, J. Stat. Softw. 48, 1 (2012).
  40. D. Magis and G. Raiche, Random generation of response patterns under computerized adaptive testing with the r package catr, J. Stat. Softw. 48, 1 (2012).
  41. D. Magis and J. R. Barrada, Computerized adaptive testing with r: Recent updates of the package catr, J. Stat. Softw. 76, 1 (2017).
  42. C. DeMars, Item Response Theory (Oxford University Press, New York, 2012).
  43. J. Yasuda, N. Mae, M. M. Hull, and M. Aki Taniguchi, Optimizing the length of computerized adaptive testing for the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 17, 010115 (2021).
  44. J. Yasuda, M. M. Hull, and N. Mae, Improving test security and efficiency of computerized adaptive testing for the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 18, 010112 (2022).
  45. W. M. Yen, Effects of local item dependence on the fit and equating performance of the three-parameter logistic model, Appl. Psychol. Meas. 8, 125 (1984).
  46. A. Maydeu-Olivares, Goodness-of-fit assessment of item response theory models, Meas. Interdiscip. Res. Perspect. 11, 71 (2013).
  47. G. Schwarz, Estimating the dimension of a model, Ann. Stat. 6, 461 (1978).
  48. J. Yasuda and M. M. Hull, Balancing content of computerized adaptive testing for the force concept inventory, presented at PER Conf. 2021, Virtual Conference, 10.1119/perc.2021.pr.Yasuda.
  49. G. G. Kingsbury and A. R. Zara, Procedures for selecting items for computerized adaptive tests, Appl. Meas. Educ. 2, 359 (1989).
  50. N. A. Thompson and D. J. Weiss, A framework for the development of computerized adaptive tests, Pract. Assess. Res. Eval. 16, 1 (2011).
  51. J. S. Bendat and A. G. Piersol, Random Data: Analysis and Measurement Procedures, 4th ed. (Wiley, New York, 2012).
  52. M. H. Dancy and R. Beichner, Impact of animation on assessment of conceptual understanding in physics, Phys. Rev. ST Phys. Educ. Res. 2, 010104 (2006).
  53. S. O. Popp and J. Jane, Can assessment of student conceptions of force be enhanced through linguistic simplification? presented at the American Educational Research Association 2009, San Diego, CA, http://modeling.asu.edu/R&E/OsbornPopp-SFCI-aera09.pdf.
  54. P. Nieminen, A. Savinainen, and J. Viiri, Force Concept Inventory-based multiple-choice test for investigating students’ representational consistency, Phys. Rev. ST Phys. Educ. Res. 6, 020109 (2010).
  55. L. McCullough and D. E. Meltzer, Differences in male/female response patterns on alternative-format versions of the Force Concept Inventory, presented at PER Conf. Proc. 2001, Rochester, NY, 10.1119/perc.2001.pr.013.
  56. T. F. Scott and D. Schumayer, Students’ proficiency scores within multitrait item response theory, Phys. Rev. ST Phys. Educ. Res. 11, 020134 (2015).
  57. J. Stewart, C. Zabriskie, S. DeVore, and G. Stewart, Multidimensional item response theory and the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 14, 010137 (2018).
  58. J. Yang, C. Zabriskie, and J. Stewart, Multidimensional item response theory and the force and motion conceptual evaluation, Phys. Rev. Phys. Educ. Res. 15, 020141 (2019).
  59. P. Eaton, B. Frank, and S. Willoughby, Detecting the influence of item chaining on student responses to the Force Concept Inventory and the force and motion conceptual evaluation, Phys. Rev. Phys. Educ. Res. 16, 020122 (2020).
  60. D. Huffman and P. Heller, What does the force concept inventory actually measure?, Phys. Teach. 33, 138 (1995).
  61. D. Hestenes and I. Halloun, Interpreting the force concept inventory: A response to March 1995 critique by Huffman and Heller, Phys. Teach. 33, 502 (1995).
  62. P. Heller and D. Huffman, Interpreting the force concept inventory: A reply to Hestenes and Halloun, Phys. Teach. 33, 503 (1995).
  63. T. F. Scott, D. Schumayer, and A. R. Gray, Exploratory factor analysis of a Force Concept Inventory data set, Phys. Rev. ST Phys. Educ. Res. 8, 020105 (2012).
  64. M. R. Semak, R. D. Dietz, R. H. Pearson, and C. W. Willis, Examining evolving performance on the Force Concept Inventory using factor analysis, Phys. Rev. Phys. Educ. Res. 13, 010103 (2017).
  65. P. Eaton and S. D. Willoughby, Confirmatory factor analysis applied to the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 14, 010124 (2018).
  66. R. P. Chalmers, Generating adaptive and non-adaptive test interfaces for multidimensional item response theory applications, J. Stat. Softw. 71, 1 (2016).
  67. W. J. van der Linden, Using response times for item selection in adaptive testing, J. Educ. Behav. Stat. 33, 5 (2008).
  68. K. Duss, Formative Assessment and Feedback Tool (Springer Fachmedien, Wiesbaden, 2020).
  69. S. Küchemann, S. Steinert, J. Kuhn, K. Avila, and S. Ruzika, Large language models—valuable tools that require a sensitive integration into teaching and learning physics, Phys. Teach. 62, 400 (2024).
  70. I. Clark, Formative assessment: Assessment is for self-regulated learning, Educ. Psychol. Rev. 24, 205 (2012).

Outline

Information

Phys. Rev. Phys. Educ. Res. 21, 020139– Published 3 November, 2025
  • Received 24 October 2024
  • Accepted 3 October 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation