- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Open- versus closed-response methods of measuring students’ reasoning: A case study using the Physics Measurement Questionnaire

Rebeckah K. Fussell1, Jillian S. Epstein1, Gina Passante2, and N. G. Holmes1

PDF

Phys. Rev. Phys. Educ. Res. 22, 020132 – Published 15 September, 2026

DOI: https://doi.org/10.1103/z9n2-tfr9

Abstract

We compare open- and closed-response versions of two questions that measure students’ reasoning on the Physics Measurement Questionnaire, same mean different spread (SMDS), and different mean same spread (DMSS). We conducted an experiment in one class (N=248) to test if asking students for their reasoning on an open-response question changes their answer to the closed-response question. To test whether answering the open-response question alters students’ closed-response answers, we prompted half of the students to explain their reasoning before seeing the closed-response options, while the other half went straight to the closed-response reasoning options. We find no effect of asking students their reasoning in an open-response question on their subsequent closed-response answers. Additionally, we investigated whether we measure students’ reasoning differently on the closed-response versus open-response versions of the question by evaluating the consistency between the students’ open-response and closed-response answers. Although we measure similarities in students’ reasoning across the two methods, the differences indicate that the open-response and closed-response questions provide different information. The results inform the need for critical judgment when deciding between open- and closed-response assessment formats.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (49)

  1. L. K. Nelson, Computational grounded theory: A methodological framework, Sociol. Methods Res. 49, 3 (2020).
  2. T. O. B. Odden, H. Tyseng, J. T. Mjaaland, M. F. Kreutzer, and A. Malthe-Sørenssen, Using text embeddings for deductive qualitative research at scale in physics education, arXiv:2402.18087.
  3. R. K. Fussell, E. M. Stump, and N. G. Holmes, Method to assess the trustworthiness of machine coding at scale, Phys. Rev. Phys. Educ. Res. 20, 010113 (2024).
  4. R. K. Fussell, M. Flynn, A. Damle, M. F. J. Fox, and N. G. Holmes, Comparing large language models for supervised analysis of students’ lab notes, Phys. Rev. Phys. Educ. Res. 21, 010128 (2025).
  5. Z. Chen and T. Wan, Grading explanations of problem-solving process and generating feedback using large language models at human-level accuracy, Phys. Rev. Phys. Educ. Res. 21, 010126 (2025).
  6. J. Wilson, B. Pollard, J. M. Aiken, M. D. Caballero, and H. J. Lewandowski, Classification of open-ended responses to a research-based assessment using natural language processing, Phys. Rev. Phys. Educ. Res. 18, 010141 (2022).
  7. K. F. Stanger-Hall and E. H. Chudler, Multiple-choice exams: An obstacle for higher-level thinking in introductory science classes, CBE–Life Sci. Educ. 11, 294 (2012).
  8. J. T. Laverty, S. M. Underwood, R. L. Matz, L. A. Posey, J. H. Carmel, M. D. Caballero, C. L. Fata-Hartley, D. Ebert-May, S. E. Jardeleza, and M. M. Cooper, Characterizing college science assessments: The three-dimensional learning assessment protocol, PLOS One 11, e0162333 (2016).
  9. Developing Assessments for the Next Generation Science Standards (National Academies Press, Washington, D.C., 2014), p. 18409.
  10. C. Walsh, K. N. Quinn, C. Wieman, and N. G. Holmes, Quantifying critical thinking: Development and validation of the physics lab inventory of critical thinking, Phys. Rev. Phys. Educ. Res. 15, 010135 (2019).
  11. A. B. Heim, D. Esparza, N. G. Holmes, and M. K. Smith, Comparing study features is easy but identifying next steps is hard: Evaluating critical thinking through the biology lab inventory of critical thinking in ecology, Ecol. Evol. 13, e10071 (2023).
  12. A. E. Lawson, The development and validation of a classroom test of formal reasoning, J. Res. Sci. Teach. 15, 11 (1978).
  13. L. Bao, K. Koenig, Y. Xiao, J. Fritchman, S. Zhou, and C. Chen, Theoretical model and quantitative assessment of scientific thinking and reasoning, Phys. Rev. Phys. Educ. Res. 18, 010115 (2022).
  14. G. Geschwind, M. Vignal, M. D. Caballero, and H. J. Lewandowski, Evidence for validity and reliability of a research-based assessment instrument on measurement uncertainty, Phys. Rev. Phys. Educ. Res. 20, 020125 (2024).
  15. B. R. Wilcox and S. J. Pollock, Coupled multiple-response versus free-response conceptual assessment: An example from upper-division physics, Phys. Rev. ST Phys. Educ. Res. 10, 020124 (2014).
  16. Q. Chen, G. Zhu, Q. Liu, J. Han, Z. Fu, and L. Bao, Development of a multiple-choice problem-solving categorization test for assessment of student knowledge structure, Phys. Rev. Phys. Educ. Res. 16, 020120 (2020).
  17. M. Scott, T. Stelzer, and G. Gladding, Evaluating multiple-choice exams in large introductory physics courses, Phys. Rev. ST Phys. Educ. Res. 2, 020102 (2006).
  18. S.-Y. Lin and C. Singh, Can free-response questions be approximated by multiple-choice equivalents?, Am. J. Phys. 81, 624 (2013).
  19. M. M. Wooten, A. M. Cool, E. E. Prather, and K. D. Tanner, Comparison of performance on multiple-choice questions and open-ended questions in an introductory astronomy laboratory, Phys. Rev. ST Phys. Educ. Res. 10, 020103 (2014).
  20. S. Breuer, T. Scherndl, and T. M. Ortner, Effects of response format on achievement and aptitude assessment results: Multi-level random effects meta-analyses, R. Soc. Open Sci. 10, 220456 (2023).
  21. M. C. Rodriguez, Construct equivalence of multiple-choice and constructed-response items: A random effects synthesis of correlations, J. Educ. Meas. 40, 163 (2003).
  22. S. Connor Desai and S. Reimers, Comparing the use of open and closed questions for web-based measures of the continued-influence effect, Behav. Res. Methods 51, 1426 (2019).
  23. V. Baburajan, J. de Abreu e Silva, and F. C. Pereira, Open vs closed-ended questions in attitudinal surveys–comparing, combining, and interpreting using natural language processing, Transp. Res. Part C: Emerging Technol. 137, 103589 (2022).
  24. V. Baburajan, J. de Abreu e Silva, and F. C. Pereira, Do open-ended questions influence the measurement of attitudes? An investigation, Transp. Res. Proc. 76, 608 (2024).
  25. M. R. Federer, R. H. Nehm, J. E. Opfer, and D. Pearl, Using a constructed-response instrument to explore the effects of item position and item features on the assessment of students’ written scientific explanations, Res. Sci. Educ. 45, 527 (2015).
  26. A. F. Heckler and A. M. Bogdan, Reasoning with alternative explanations in physics: The cognitive accessibility rule, Phys. Rev. Phys. Educ. Res. 14, 010120 (2018).
  27. E. M. Stump, M. Dew, G. Passante, and N. G. Holmes, Context affects student thinking about sources of uncertainty in classical and quantum mechanics, Phys. Rev. Phys. Educ. Res. 19, 020157 (2023).
  28. S. Allie, A. Buffler, B. Campbell, and F. Lubben, First-year physics students’ perceptions of the quality of experimental measurements, Int. J. Sci. Educ. 20, 447 (1998).
  29. A. A. diSessa, Knowledge in pieces, in Constructivism in the Computer Age, the Jean Piaget Symposium Series (Lawrence Erlbaum Associates, Inc, Hillsdale, NJ, US, 1988), pp. 49–70.
  30. D. Hammer, Student resources for learning introductory physics, Am. J. Phys. 68, S52 (2000).
  31. A. Buffler, S. Allie, and F. Lubben, The development of first year physics students’ ideas about measurement in terms of point and set paradigms, Int. J. Sci. Educ. 23, 1137 (2001).
  32. F. Lubben, B. Campbell, A. Buffler, and S. Allie, Point and set reasoning in practical science measurement by entering university freshmen, Sci. Educ. 85, 311 (2001).
  33. R. F. Lippmann. Students’ Understanding of Measurement and Uncertainty in the Physics Laboratory: Social Construction, Underlying Concepts, and Quantitative Analysis, Ph. D. thesis, University of Maryland, 2003.
  34. D. L. Deardorff, Introductory Physics Students’ Treatment of Measurement Uncertainty, Ph. D. thesis, North Carolina State University, 2001.
  35. M. Vignal, G. Geschwind, B. Pollard, R. Henderson, M. D. Caballero, and H. J. Lewandowski, Survey of physics reasoning on uncertainty concepts in experiments: An assessment of measurement uncertainty for introductory physics labs, Phys. Rev. Phys. Educ. Res. 19, 020139 (2023).
  36. R. Fussell, J. Epstein, N. Holmes, and G. Passante, Open versus closed-response methods of assessing students’ reasoning (2025). https://doi.org/10.5281/zenodo.17871686.
  37. J. R. Landis and G. G. Koch, The measurement of observer agreement for categorical data, Biometrics 33, 159 (1977).
  38. E. B. Wilson, Probable inference, the law of succession, and statistical inference, J. Am. Stat. Assoc. 22, 209 (1927).
  39. J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. (Routledge, New York, 2013).
  40. J. Cohen, The earth is round (p<.05)., Am. Psychol. 49, 997 (1994).
  41. G. Cumming, The new statistics: Why and how, Psychol. Sci. 25, 7 (2013).
  42. B. A. Nosek, C. R. Ebersole, A. C. DeHaven, and D. T. Mellor, The preregistration revolution, Proc. Natl. Acad. Sci. U.S.A. 115, 2600 (2018).
  43. D. Tannen, Framing in Discourse (Oxford University Press on Demand, Oxford, UK, 1993).
  44. K. Shar, R. S. Russ, and J. T. Laverty, Student epistemological framing on paper-based assessments, Phys. Rev. Phys. Educ. Res. 16, 020101 (2020).
  45. J. K. Hubbard, M. A. Potts, B. A. Couch, and K. E. Perez, How question types reveal student thinking: An experimental comparison of multiple-true-false and free-response formats, CBE–Life Sci. Educ. 16, ar26 (2017).
  46. P. Tschisgale, P. Wulff, and M. Kubsch, Integrating artificial intelligence-based methods into qualitative research in physics education research: A case for computational grounded theory, Phys. Rev. Phys. Educ. Res. 19, 020123 (2023).
  47. J. T. Mjaaland, M. F. Kreutzer, H. Tyseng, R. K. Fussell, G. Passante, N. G. Holmes, A. Malthe-Sørenssen, and T. O. B. Odden, Scalable and consistent few-shot classification of survey responses using text embeddings, arXiv:2508.19836.
  48. E. M. Stump, M. Hughes, G. Passante, and N. G. Holmes, Comparing introductory and beyond-introductory students’ reasoning about uncertainty, Phys. Rev. Phys. Educ. Res. 19, 020147 (2023).
  49. A. Schang, M. Dew, E. M. Stump, N. G. Holmes, and G. Passante, New perspectives on student reasoning about measurement uncertainty: More or better data, Phys. Rev. Phys. Educ. Res. 19, 020105 (2023).

Outline

Information

Phys. Rev. Phys. Educ. Res. 22, 020132– Published 15 September, 2026
  • Received 9 December 2025
  • Accepted 17 July 2026
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation