- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Can large language models correctly interpret equations with errors?

Lachlan McGinness* and Peter Baumgartner

  • *Contact author: Lachlan.McGinness@anu.edu.au
PDF

Phys. Rev. Phys. Educ. Res. 21, 020155 – Published 15 December, 2025

DOI: https://doi.org/10.1103/v8f8-s11v

Abstract

This paper explores the potential of large language models to accurately extract and translate equations from typed student responses into a standard format. This is a useful task as standardized equations can be graded reliably using a computer algebra system or a satisfiability modulo theories solver. Therefore physics instructors interested in automated grading would not need to rely on the mathematical reasoning capabilities of language models. We used two novel frameworks to improve the translations. The first is consensus where a pair of models verify the correctness of the translations. The second is a neurosymbolic LLM-modulo approach were models receive feedback from an automated reasoning tool. We performed experiments using responses to the Australian Physics Olympiad exam. We report on results, finding that no open-source model was able to translate the student responses at the desired level of accuracy. Future work could involve breaking the task into smaller components before parsing to improve performance or generalizing the experiments to translate hand-written responses.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (57)

  1. T. H. Trinh, Y. Wu, Q. V. Le, H. He, and T. Luong, Solving olympiad geometry without human demonstrations, Nature (London) 625, 476 (2024).
  2. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, Training verifiers to solve math word problems, arXiv:2110.14168.
  3. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, Measuring mathematical problem solving with the MATH dataset, arXiv:2103.03874.
  4. J. Ahn, R. Verma, R. Lou, D. Liu, R. Zhang, and W. Yin, Large language models for mathematical reasoning: Progresses and challenges, in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, edited by N. Falk, S. Papi, and M. Zhang (Association for Computational Linguistics, St. Julian’s, Malta, 2024), pp. 225–237.
  5. H. Zhang, J. Da, D. Lee, V. Robinson, C. Wu, W. Song, T. Zhao, P. V. Raja, C. Zhuang, D. Z. Slack, Q. Lyu, S. M. Hendryx, R. Kaplan, M. Lunati, and S. Yue, A careful examination of large language model performance on grade school arithmetic, in Advances in Neural Information Processing Systems 37 (NeurIPS 2023) (Neural Information Processing Systems Foundation, Vancouver, Canada, 2023).
  6. H. Barbosa, Challenges in SMT proof production and checking for arithmetic reasoning, in CEUR Workshop Proceedings (CEUR-WS.org, Tromsø, Norway, 2023),Vol. 3455.
  7. D. Guidotti, L. Pandolfo, and L. Pulina, Leveraging satisfiability modulo theory solvers for verification of neural networks in predictive maintenance applications, Information 14, 397 (2023).
  8. D. Winterer and Z. Su, Validating SMT solvers for correctness and performance via grammar-based enumeration, Proc. ACM Program. Lang. 8, 2378 (2024).
  9. C. Barrett, P. Fontaine, and A. Stump, The SMT-LIB standard, Technical Report, University of California, Berkeley, 2024.
  10. H. Lachnitt, M. Fleury, L. Aniva, A. Reynolds, H. Barbosa, A. Nötzli, C. Barrett, and C. Tinelli, IsaRare: Automatic verification of SMT rewrites in Isabelle/HOL, in Tools and Algorithms for the Construction and Analysis of Systems, edited by B. Finkbeiner and L. Kovács, Lecture Notes in Computer Science Vol. 14570 (Springer Nature Switzerland, Cham, 2024), pp. 311–330.
  11. S. Kambhampati, K. Valmeekam, L. Guan, M. Verma, K. Stechly, S. Bhambri, L. P. Saldyt, and A. B. Murthy, Position: LLMs can’t plan, but can help planning in LLM-modulo frameworks, in ICML24: Proceedings of the 41st International Conference on Machine Learning (PMLR, Cambridge, MA, 2024), pp. 22895–22907.
  12. S. Burrows, I. Gurevych, and B. Stein, The eras and trends of automatic short answer grading, Int. J. Artif. Intell. Educ. 25, 60 (2015).
  13. U. Hasanah, A. E. Permanasari, S. S. Kusumawardani, and F. S. Pribadi, A review of an information extraction technique approach for automatic short answer grading, in Proceedings of the 1st International Conference on Information Technology, Information Systems and Electrical Engineering, ICITISEE, Yogyakarta, Indonesia (IEEE, New York, 2016), pp. 192–196.
  14. T. A. Mat, A. Lajis, and H. Nasir, Text data preparation in RapidMiner for short free text answer in assisted assessment, in Proceedings of the IEEE 5th International Conference on Smart Instrumentation, Measurement and Application, ICSIMA, Songkla, Thailand (IEEE, New York, 2018), pp. 1–4.
  15. H.-C. Wang, C.-Y. Chang, and T.-Y. Li, Assessing creative problem-solving with automated text grading, Comput. Educ. 51, 1450 (2008).
  16. R. Ziai, N. Ott, and D. Meurers, Short answer assessment: Establishing links between research strands, in Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, Montréal, Canada (Association for Computational Linguistics, Stroudsburg, PA, 2012), pp. 190–200.
  17. L. Mcginness and P. Baumgartner, CON-FOLD explainable machine learning with confidence, Theory Pract. Logic Program. 24, 663 (2024).
  18. C. Grévisse, LLM-based automatic short answer grading in undergraduate medical education, BMC Med. Educ. 24, 1060 (2024).
  19. S. Tobler, Smart grading: A generative AI-based tool for knowledge-grounded answer evaluation in educational assessments, MethodsX 12, 102531 (2024).
  20. Z. Yan, R. Zhang, and F. Jia, Exploring the potential of large language models as a grading tool for conceptual short-answer questions in introductory physics, in Proceedings of the 9th International Conference on Distance Education and Learning, Guangzhou, China (ACM, New York, NY, 2024), pp. 308–314.
  21. G. Kortemeyer, Performance of the pre-trained large language model GPT-4 on automated short answer grading, Discover Artif. Intell. 4, 47 (2024).
  22. S. Bonthu, S. R. Sree, and M. Krishna Prasad, Framework for automation of short answer grading based on domain-specific pre-training, Eng. Appl. Artif. Intell. 137, 109163 (2024).
  23. T. Menezes, L. Egherman, and N. Garg, AI-grading standup updates to improve project-based learning outcomes, in Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, Milan Italy (ACM, New York, NY, 2024), pp. 17–23.
  24. C. Senanayake and D. Asanka, Rubric based automated short answer scoring using large language models (LLMs), in Proceedings of the International Research Conference on Smart Computing and Systems Engineering (SCSE), Colombo, Sri Lanka (IEEE, New York, 2024), pp. 1–6.
  25. A. Filighera, S. Ochs, T. Steuer, and T. Tregel, Cheating automatic short answer grading with the adversarial usage of adjectives and adverbs, Int. J. Artif. Intell. Educ. 34, 616 (2024).
  26. R. Mok, F. Akhtar, L. Clare, C. Li, J. Ida, L. Ross, and M. Campanelli, Using AI large language models for grading in education: A hands-on test for physics, arXiv:2411.13685.
  27. Z. Chen and T. Wan, Achieving human level partial credit grading of written responses to physics conceptual question using GPT-3.5 with only prompt engineering, presented at PER Conf. 2024, Boston, MA, 10.1119/perc.2024.pr.Chen.
  28. Z. Chen and T. Wan, Grading explanations of problem-solving process and generating feedback using large language models at human-level accuracy, Phys. Rev. Phys. Educ. Res. 21, 010126 (2025).
  29. G. Kortemeyer and J. Nohl, Assessing confidence in AI-assisted grading of physics exams through psychometrics: An exploratory study, Phys. Rev. Phys. Educ. Res. 21, 010136 (2025).
  30. Department of Industry Science and Resources, Australian Government, Australia’s Artificial Intelligence Ethics Principles (Australian Government, Department of Industry, Science and Resources, 2024), https://www.industry.gov.au/publications/australias-artificial-intelligence-ethics-principles/australias-ai-ethics-principles.
  31. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, On the dangers of stochastic parrots: Can language models be too big?, in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event, Canada (ACM, New York, NY, 2021), pp. 610–623.
  32. J. Morgan and M. Chiang, Ollama Documentation (Ollama, 2025), https://ollama.com.
  33. L. De Moura and N. Bjørner, Satisfiability modulo theories: Introduction and applications, Commun. ACM 54, 69 (2011).
  34. L. De Moura and N. Bjørner, Z3: An efficient SMT solver, in Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’08/ETAPS’08 (Springer-Verlag, Berlin, Heidelberg, 2008), pp. 337–340.
  35. D. Beyer, C. Barrett, P. Fontaine, A. Niemetz, M. Preiner, and S. Hans-Jorg, SMT-COMP 2025, in 20th International Satisfiability Modulo Theories Competition (SMT-COMP 2025) (SMT Steering Committee, 2025), https://smt-comp.github.io/2025/.
  36. J. H. Davenport, Y. Siret, and E. Tournier, Computer Algebra (2nd ed.): Systems and Algorithms for Algebraic Computation (Academic Press Professional, Inc., New York, 1993).
  37. A. Meurer et al., sympy: Symbolic computing in Python, PeerJ Comput. Sci. 3, e103 (2017).
  38. G. Kortemeyer, Toward AI grading of student problem solutions in introductory physics: A feasibility study, Phys. Rev. Phys. Educ. Res. 19, 020163 (2023).
  39. G. Kortemeyer, J. Nöhl, and D. Onishchuk, Grading assistance for a handwritten thermodynamics exam using artificial intelligence: An exploratory study, Phys. Rev. Phys. Educ. Res. 20, 020144 (2024).
  40. P. Baumgartner and L. McGinness, The AlphaPhysics term rewriting system for marking algebraic expressions in physics exams, arXiv:2507.18337.
  41. L. McGinness, P. Baumgartner, E. Onyango, and Z. Lema, Highlighting case studies in LLM literature review of interdisciplinary system science, in AI 2024: Advances in Artificial Intelligence (Springer, Cham, 2024), pp. 29–43.
  42. L. McGinness and P. Baumgartner, Large language models imitate logical reasoning, but at what cost?, arXiv:2509.12645.
  43. S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J.-R. Wen, and C. Li, Large language diffusion models, arXiv:2502.09992.
  44. J. Geiping, S. McLeish, N. Jain, A. Bhatele, and T. Goldstein, Scaling up test-time compute with latent reasoning: A recurrent depth approach, arXiv:2502.05171.
  45. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, Scaling laws for neural language models, arXiv:2001.08361.
  46. NVIDIA, NVIDIA A100 tensor core GPU datasheet, Technical Report, NVIDIA Corporation, 2021.
  47. NVIDIA, NVIDIA A100 Tensor Core GPU: Unprecedented Acceleration at Every Scale, NVIDIA Corporation, 2022.
  48. NVIDIA, NVIDIA H100 Tensor Core GPU Datasheet, NVIDIA Corporation, 2024.
  49. Uptime Institute, Uptime institute global data center survey results 2025, Technical Report, Uptime Institute, New York, 2025, https://uptimeinstitute.com/resources/research-and-reports/uptime-institute-global-data-center-survey-results-2025.
  50. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter, arXiv:1910.01108.
  51. Meta AI, Llama 3.2 Model Card (Meta AI, 2025), https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/.
  52. D. Guo and Z. Gao, DeepSeek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning, arXiv:2501.12948.
  53. Mistral AI, Mathstral Model Announcement (Mistral AI, 2025), https://mistral.ai/news/mathstral.
  54. M. I. Abdin and Y. Zhang, Phi4 technical report, Technical Report, Microsoft Research, 2024.
  55. Meta AI, Llama 3.3 Model Card (Meta AI, 2025), https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_3/.
  56. A. Grattafiori, The Llama 3 herd of models, arXiv:2407.21783.
  57. X. Bi, Y. Zou et al., DeepSeek LLM: Scaling open-source language models with longtermism, arXiv:2401.02954.

Outline

Information

Phys. Rev. Phys. Educ. Res. 21, 020155– Published 15 December, 2025
  • Received 18 May 2025
  • Accepted 7 November 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation