- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Combining physics education and machine learning research to measure evidence of students’ mechanistic sensemaking

Kaitlin Gili1,*, Kyle Heuton1, Astha Shah1, David Hammer2,†, and Michael C. Hughes1,†

  • 1Department of Computer Science, Tufts University, Medford, Massachusetts, USA
  • 2Department of Education and Department of Physics and Astronomy, Tufts University, Medford, Massachusetts, USA

  • *Contact author: kgili01@tufts.edu
  • †These authors contributed equally to this work.
PDF

Phys. Rev. Phys. Educ. Res. 21, 020161 – Published 22 December, 2025

DOI: https://doi.org/10.1103/v6gw-lv4c

Abstract

Advances in machine learning (ML) offer new possibilities for science education research. We report on early progress in the design of an ML-based tool to analyze students’ mechanistic sensemaking, working from a coding scheme that is aligned with previous work in physics education research (PER) and that is amenable to recently developed ML classification strategies using language encoders. We describe pilot tests of the tool, in three versions with different language encoders, to analyze sensemaking evident in college students’ written responses to brief conceptual questions. The results show, first, that the tool’s measurements of sensemaking can achieve useful agreement with a human coder, and, second, that encoder design choices entail a tradeoff between accuracy and computational expense. We discuss the promise and limitations of this approach, providing examples of how this measurement scheme may serve PER in the future. We conclude with reflections on the use of ML to support PER research, offering cautious optimism for strategies of codesign between PER and ML.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (60)

  1. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv:1810.04805.
  2. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, LLaMa: Open and efficient foundation language models, arXiv:2302.13971.
  3. P. BehnamGhader, V. Adlakha, M. Mosbach, D. Bahdanau, N. Chapados, and S. Reddy, LLM2Vec: Large language models are secretly powerful text encoders, arXiv:2404.05961.
  4. J. Haab, N. Deutschmann, and M. R. Martínez, Is attention interpretation? A quantitative assessment on sets, arXiv:2207.13018.
  5. N. Mylonas, I. Mollas, and G. Tsoumakas, An attention matrix for every decision: Faithfulness-based arbitration among multiple attention-based interpretations of transformers in text classification, arXiv:2209.10876.
  6. Applying Machine Learning in Science Education Research: When, How, and Why?, edited by P. Wulff, M. Kubsch, and C. Krist, 1st ed., Springer Texts in Education Vol. 1 (Springer, Cham, 2025), pp. XIII + 369, 10.1007/978-3-031-74227-9.
  7. C. Krist, M. Kubsch, and P. Wulff, Introduction, in Applying Machine Learning in Science Education Research: When, How, and Why? (Springer Nature Switzerland, Cham, 2025), pp. 1–12, 10.1007/978-3-031-74227-9_1.
  8. R. K. Fussell, M. Flynn, A. Damle, M. F. J. Fox, and N. G. Holmes, Comparing large language models for supervised analysis of students’ lab notes, Phys. Rev. Phys. Educ. Res. 21, 010128 (2025).
  9. R. K. Fussell, E. M. Stump, and N. G. Holmes, Method to assess the trustworthiness of machine coding at scale, Phys. Rev. Phys. Educ. Res. 20, 010113 (2024).
  10. T. O. B. Odden and R. S. Russ, Defining sensemaking: Bringing clarity to a fragmented theoretical construct, Sci. Educ. 103, 187 (2019).
  11. J. Radoff, L. Z. Jaber, and D. Hammer, “It’s scary but it’s also exciting”: Evidence of meta-affective learning in science, Cognit. Instr. 37, 73 (2019).
  12. T. O. B. Odden and R. S. Russ, Vexing questions that sustain sensemaking, Int. J. Sci. Educ. 41, 1 (2019).
  13. F. F. Zhao and A. Schuchardt, Development of the sci-math sensemaking framework: Categorizing sensemaking of mathematical equations in science, Int. J. STEM Educ. 8, 10 (2021).
  14. A. M. L. Phillips, J. Watkins, and D. Hammer, Beyond “asking questions”: Problematizing as a disciplinary activity., J. Res. Sci. Teach. 55, 982 (2018).
  15. A. Sirnoorkar, J. T. Laverty, and P. D. O. Bergeron, Sensemaking and scientific modeling: Intertwined processes analyzed in the context of physics problem solving, arXiv:2207.03939.
  16. T. O. B. Odden, How conceptual blends support sensemaking: A case study from introductory physics, Sci. Educ. 105, 989 (2021).
  17. B. A. Danielak, A. Gupta, and A. Elby, Marginalized identities of sense-makers: Reframing engineering student retention, J. Eng. Educ. 103, 8 (2014).
  18. K. H. Hunter, J.-M. G. Rodriguez, and N. M. Becker, Making sense of sensemaking: Using the sensemaking epistemic game to investigate student discourse during a collaborative gas law activity, Chem. Educ. Res. Pract. 22, 328 (2021).
  19. T. O. B. Odden and R. S. Russ, Sensemaking epistemic game: A model of student sensemaking processes in introductory physics, Phys. Rev. Phys. Educ. Res. 14, 020122 (2018).
  20. M. J. Ford, A dialogic account of sense-making in scientific argumentation and reasoning, Cognit. Instr. 30, 207 (2012).
  21. S. Kapon, Unpacking sensemaking, Sci. Educ. 101, 165 (2017).
  22. R. S. Russ, R. E. Scherr, D. Hammer, and J. Mikeska, Recognizing mechanistic reasoning in student scientific inquiry: A framework for discourse analysis developed from philosophy of science, Sci. Educ. 92, 499 (2008).
  23. G. G. M. Selen and T. Stelzer, Classical Mechanics (FlipItPhysics for University Physics, Macmillan Learning, New York, NY, 2016).
  24. M.-L. M. Ko and M. J. Luna, The glue that makes it “hang together”: A framework for identifying how metadiscourse facilitates uncertainty navigation during knowledge building discussions, J. Res. Sci. Teach. 61, 457(2024).
  25. tufts-ml/AuSeM, GitHub repository, https://github.com/tufts-ml/AuSeM (2025).
  26. A. M. L. Phillips, J. Watkins, and D. Hammer, Problematizing as a scientific endeavor, Phys. Rev. Phys. Educ. Res. 13, 020107 (2017).
  27. P. Machamer, L. Darden, and C. F. Craver, Thinking about mechanisms, Philos. Sci. 67, 1 (2000).
  28. F. M. Watts, J. A. Schmidt-McCormack, C. A. Wilhelm, A. Karlin, A. Sattar, B. C. Thompson, A. R. Gere, and G. V. Shultz, What students write about when students write about mechanisms: Analysis of features present in students’ written descriptions of an organic reaction mechanism, Chem. Educ. Res. Pract. 21, 1148 (2020).
  29. F. M. Watts, A. J. Dood, and G. V. Shultz, Developing machine learning models for automated analysis of organic chemistry students’ written descriptions of organic reaction mechanisms, in Student Reasoning in Organic Chemistry (The Royal Society of Chemistry, London, United Kingdom, 2022).
  30. J. M. Aiken, R. De Bin, H. J. Lewandowski, and M. D. Caballero, Framework for evaluating statistical models in physics education research, Phys. Rev. Phys. Educ. Res. 17, 020104 (2021).
  31. T. D. Ullmann, Automated analysis of reflection in writing: Validating machine learning approaches, Int. J. Artif. Intell. Educ. 29, 217 (2019).
  32. C. M. Nakamura, S. K. Murphy, M. G. Christel, S. M. Stevens, and D. A. Zollman, Automated analysis of short responses in an interactive synthetic tutoring system for introductory physics, Phys. Rev. Phys. Educ. Res. 12, 010122 (2016).
  33. J. Wilson, B. Pollard, J. M. Aiken, M. D. Caballero, and H. J. Lewandowski, Classification of open-ended responses to a research-based assessment using natural language processing, Phys. Rev. Phys. Educ. Res. 18, 010141 (2022).
  34. P. Tschisgale, P. Wulff, and M. Kubsch, Integrating artificial intelligence-based methods into qualitative research in physics education research: A case for computational grounded theory, Phys. Rev. Phys. Educ. Res. 19, 020123 (2023).
  35. T. O. B. Odden, A. Marin, and M. D. Caballero, Thematic analysis of 18 years of physics education research conference proceedings using natural language processing, Phys. Rev. Phys. Educ. Res. 16, 010142 (2020).
  36. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, Mistral 7B, arXiv:2310.06825.
  37. Y. Long, H. Luo, and Y. Zhang, Evaluating large language models in analysing classroom dialogue, npj Sci. Learn. 9, 60 (2024).
  38. Q. Long, Y. Wu, W. Wang, and S. J. Pan, Does in-context learning really learn? Rethinking how large language models respond and solve tasks via in-context learning, in First Conference on Language Modeling (2024), https://openreview.net/forum?id=i2oJjC0ESQ.
  39. S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, Rethinking the role of demonstrations: what makes in-context learning work?, in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, edited by Y. Goldberg, Z. Kozareva, and Y. Zhang (Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022), pp. 11048–11064, https://aclanthology.org/2022.emnlp-main.759/.
  40. Z. Jiang, F. F. Xu, J. Araki, and G. Neubig, How can we know what language models know?, Trans. Assoc. Comput. Linguist. 8, 423 (2020).
  41. J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen, What makes good in-context examples for GPT-3?, in Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures Online, E. Agirre, M. Apidianaki, and I. Vulić (Association for Computational Linguistics, Dublin, Ireland, 2022), pp. 100–114, https://aclanthology.org/2022.deelio-1.10/.
  42. P. P. Liang, C. Wu, L.-P. Morency, and R. Salakhutdinov, Towards understanding and mitigating social biases in language models, in Proceedings of the 38th International Conference on Machine Learning (PMLR, 2021), pp. 6565–6576, https://proceedings.mlr.press/v139/liang21a.html.
  43. W. A. Qader, M. M. Ameen, and B. I. Ahmed, An overview of bag of words; importance, implementation, applications, and challenges, in Proceedings of the 2019 International Engineering Conference (IEC), Erbil, Iraq, 2019 (2019), pp. 200–204, 10.1109/IEC47844.2019.8950616.
  44. K. P. Murphy, Probabilistic Machine Learning: An Introduction (MIT Press, Cambridge, MA, 2022), https://probml.github.io/pml-book/book1.html.
  45. R. E. Turner, An introduction to transformers, arXiv:2304.10557.
  46. C. Sun, X. Qiu, Y. Xu, and X. Huang, How to fine-tune BERT for text classification?, arXiv:1905.05583.
  47. E. Gibaja and S. Ventura, Multi-label learning: A review of the state of the art and ongoing research, WIREs Data Min. Knowl. Discov. 4, 411 (2014).
  48. D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, in Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, edited by Y. Bengio and Y. LeCun (2015), arXiv:1412.6980.
  49. C. Chelba and A. Acero, Adaptation of maximum entropy capitalizer: Little data can help a lot, Comput. Speech Lang. 20, 382 (2006).
  50. L. Xuhong, Y. Grandvalet, and F. Davoine, Explicit inductive bias for transfer learning with convolutional networks, in International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research (PMLR), Stockholm, Sweden, 2018).
  51. D. Masters and C. Luschi, Revisiting small batch training for deep neural networks, arXiv:1804.07612.
  52. B. Efron, Bootstrap methods: Another look at the jackknife, in Breakthroughs in Statistics: Methodology and Distribution (Springer New York, New York, NY, 1992), pp. 569–593, 10.1007/978-1-4612-4380-9_41.
  53. S. Raschka, Model evaluation, model selection, and algorithm selection in machine learning, arXiv:1811.12808.
  54. J. E. van Engelen and H. H. Hoos, A survey on semi-supervised learning, Mach. Learn. 109, 373 (2020).
  55. E. Kuo, M. M. Hull, A. Elby, and A. Gupta, Assessing mathematical sensemaking in physics through calculation-concept crossover, Phys. Rev. Phys. Educ. Res. 16, 020109 (2020).
  56. J. Cohen, A coefficient of agreement for nominal scales, Educ. Psychol. Meas. 20, 37 (1960).
  57. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, On the dangers of stochastic parrots: Can language models be too big?, in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT (Association for Computing Machinery, New York, NY, USA, 2021), pp. 610–623, 10.1145/3442188.3445922.
  58. W. X. Zhao et al., A survey of large language models, arXiv:2303.18223.
  59. T. B. Brown et al., Language models are few-shot learners, arXiv:2005.14165.
  60. M. B. McDermott, H. Zhang, L. H. Hansen, G. Angelotti, and J. Gallifant, A closer look at AUROC and AUPRC under class imbalance, in Proceedings of the 38th Annual Conference on Neural Information Processing Systems (2024), https://openreview.net/forum?id=S3HvA808gk.

Outline

Information

Phys. Rev. Phys. Educ. Res. 21, 020161– Published 22 December, 2025
  • Received 13 April 2025
  • Accepted 4 December 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation