- Editors' Suggestion
- Open Access
Challenges in using online assessments to measure student understanding in the era of generative AI
Phys. Rev. Phys. Educ. Res. 22, 020161 – Published 19 November, 2026
DOI: https://doi.org/10.1103/xrw4-t58t
Abstract
Many courses now incorporate online methods for assessing student learning. Some instructors assign questions prior to lecture (e.g., prelecture quizzes) to obtain information that helps them tailor their lessons to the current state of student understanding. To assess curriculum efficacy, physics education researchers often use “pretests,” inventories, etc. that provide baseline data that can be compared to results from post-tests. These assessments were initially given on paper but moved online in the early 2000s. At the University of Washington, we found that the impact of this change was minimal; the results of our pretests stayed essentially the same when given on paper or online and were consistent from quarter to quarter and instructor to instructor. However, we have begun to observe systematic changes in pretest performance that are consistent with students using artificial intelligence (AI) tools to answer pretests, even though they are scored only on effort, not correctness. Using two-proportion -tests, mixed-effects logistic regression, and Welch’s tests to analyze response length of explanations, this paper documents the nature and extent of these changes across 39 questions in two introductory algebra-based physics courses. This paper presents results that suggest caution to those using online quizzes to probe student understanding—both before and/or after instruction. We have seen increases in correct responses on pretests as great as 36%, but we have also seen significant decreases on questions that AI tools have tended to answer incorrectly. These findings raise important questions about the validity of online assessments and are intended to raise awareness for anyone using online tests to assess student learning. It remains an open question as to whether, and how, AI can be leveraged to improve instruction.
Physics Subject Headings (PhySH)
Article Text
References (36)
- L. C. McDermott, Oersted medal lecture 2001: “Physics education research—The key to student learning,” Am. J. Phys. 69, 1127 (2001).
- D. Hestenes, M. Wells, and G. Swackhamer, Force concept inventory, Phys. Teach. 30, 141 (1992).
- D. P. Maloney, T. L. O’Kuma, C. J. Hieggelke, and A. Van Heuvelen, Surveying students’ conceptual knowledge of electricity and magnetism, Am. J. Phys. 69, S12 (2001).
- P. R. L. Heron, Effect of lecture instruction on student performance on qualitative questions, Phys. Rev. ST Phys. Educ. Res. 11, 010102 (2015).
- P. S. Shaffer and L. C. McDermott, A research-based approach to improving student understanding of the vector nature of kinematical concepts, Am. J. Phys. 73, 921 (2005).
- K. Kellar and P. Heron, Distinguishing between students’ conceptual understanding and reasoning approaches: An application of dual-process theories, Phys. Rev. Phys. Educ. Res. 21, 010141 (2025).
- H. G. Close and P. R. L. Heron, Research as a guide for improving student learning: An example from momentum conservation, Am. J. Phys. 78, 961 (2010).
- B. A. Lindsey, P. R. L. Heron, and P. S. Shaffer, Student ability to apply the concepts of work and energy to extended systems, Am. J. Phys. 77, 999 (2009).
- B. S. Ambrose, P. R. L. Heron, S. Vokos, and L. C. McDermott, Student understanding of light as an electromagnetic wave: Relating the formalism to physical phenomena, Am. J. Phys. 67, 891 (1999).
- R. E. Scherr, P. S. Shaffer, and S. Vokos, Student understanding of time in special relativity: Simultaneity and reference frames, Am. J. Phys. 69, S24 (2001).
- C. Furió and J. Guisasola, Difficulties in learning the concept of electric field, Sci. Ed. 82, 511 (1998).
- L. Bollen, P. van Kampen, C. Baily, M. Kelly, and M. De Cock, Student difficulties regarding symbolic and graphical representations of vector fields, Phys. Rev. Phys. Educ. Res. 13, 020109 (2017).
- W. M. Christensen and J. R. Thompson, Investigating graphical representations of slope and derivative without a physics context, Phys. Rev. ST Phys. Educ. Res. 8, 023101 (2012).
- Q. X. Ryan, B. R. Wilcox, and S. J. Pollock, Student difficulties with boundary conditions in the context of electromagnetic waves, Phys. Rev. Phys. Educ. Res. 14, 020126 (2018).
- D. P. Smith and P. van Kampen, Teaching electric circuits with multiple batteries: A qualitative approach, Phys. Rev. ST Phys. Educ. Res. 7, 020115 (2011).
- J. Von Korff, B. Archibeque, K. A. Gomez, T. Heckendorf, S. B. McKagan, E. C. Sayre, E. W. Schenk, C. Shepherd, and L. Sorell, Secondary analysis of teaching methods in introductory physics: A 50 k-student study, Am. J. Phys. 84, 969 (2016).
- A. Madsen, S. B. McKagan, and E. C. Sayre, Resource letter RBAI-1: Research-based assessment instruments in physics and astronomy, Am. J. Phys. 85, 245 (2017).
- Mastering digital learning platforms | Pearson, https://www.pearson.com/en-us/higher-education/products-services/mastering.html.
- WebAssign, https://www.webassign.net/index.html.
- PhET interactive simulations, https://phet.colorado.edu/.
- A. Titus and G. Tang, Increasing students’ time on task in calculus and general physics courses through webassign, in 2002 Annual Conference Proceedings (ASEE Conferences, Montreal, Canada, 2002), pp. 7.660.1–7.660.20.
- S. B. McKagan, K. K. Perkins, M. Dubson, C. Malley, S. Reid, R. LeMaster, and C. E. Wieman, Developing and researching PhET simulations for teaching quantum mechanics, Am. J. Phys. 76, 406 (2008).
- C. E. Wieman, W. K. Adams, P. Loeblein, and K. K. Perkins, Teaching physics using PhET simulations, Phys. Teach. 48, 225 (2010).
- K. Lazarova, The role of online homework in low-enrollment college introductory physics courses, J. Coll. Sci. Teach. 44, 17 (2015).
- G. Kortemeyer, Toward AI grading of student problem solutions in introductory physics: A feasibility study, Phys. Rev. Phys. Educ. Res. 19, 020163 (2023).
- R. Mok, F. Akhtar, L. Clare, C. Li, J. Ida, L. Ross, and M. Campanelli, Using large language models for grading in education: An applied test for physics, Phys. Educ. 60, 035006 (2025).
- D. Tong, B. Jin, Y. Tao, H. Ren, A. Y. M. Atiquil Islam, and L. Bao, Exploring the role of human-AI collaboration in solving scientific problems, Phys. Rev. Phys. Educ. Res. 21, 010149 (2025).
- S. Kilde-Westberg, A. Johansson, and J. Enger, Generative AI as a lab partner: A case study, Phys. Rev. Phys. Educ. Res. 21, 020119 (2025).
- G. Kortemeyer and W. Bauer, Cheat sites and artificial intelligence usage in online introductory physics courses: What is the extent and what effect does it have on assessments?, Phys. Rev. Phys. Educ. Res. 20, 010145 (2024).
- P. Tschisgale, H. Maus, F. Kieser, B. Kroehs, S. Petersen, and P. Wulff, Evaluating GPT- and reasoning-based large language models on physics olympiad problems: Surpassing human performance and implications for educational assessment, Phys. Rev. Phys. Educ. Res. 21, 020115 (2025).
- L. C. McDermott and P. S. Shaffer, Tutorials in Introductory Physics (Pearson College Div, Upper Saddle River, New Jersey, 2002).
- G. Kortemeyer, Could an artificial-intelligence agent pass an introductory physics course?, Phys. Rev. Phys. Educ. Res. 19, 010132 (2023).
- ChatGPT, https://chatgpt.com/.
- L. C. McDermott, P. S. Shaffer, and M. D. Somers, Research as a guide for teaching introductory mechanics: An illustration in the context of the Atwood’s machine, Am. J. Phys. 62, 46 (1994).
- Honorlock online proctoring services and software, https://honorlock.com/.
- Prevent cheating in higher Ed with respondus lockdown browser, https://web.respondus.com/he/lockdownbrowser/.