- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Development of a multiple-choice test to assess critical thinking in climate change

Magdalena Micoloi1,*, Lana Ivanjek2, Mieke De Cock3, Thomas Schubatzky4, Sarah Wildbichler4, Rainer Wackermann5, and Gesche Pospiech1

  • 1TU Dresden, Faculty of Physics, Physics Education Research Group, Haeckelstraße 3, 01069 Dresden, Germany
  • 2JKU Linz, School of Education, Altenberger Straße 69, 4040 Linz, Austria
  • 3KU Leuven, Department of Physics and Astronomy and LESEC, Celestijnenlaan 200c—Box 2406, 3001 Leuven, Belgium
  • 4University of Innsbruck, Physics Education Research Group, Fürstenweg 176, 6020 Innsbruck, Austria
  • 5Ruhr University Bochum, Physics Education Research Group, Universitätsstraße 150, 44801 Bochum, Germany

  • *Contact author: magdalena.micoloi@tu-dresden.de
PDF

Phys. Rev. Phys. Educ. Res. 21, 020154 – Published 15 December, 2025

DOI: https://doi.org/10.1103/jnbq-1qyk

Abstract

Recognizing the complex environmental challenges that humanity faces, there is a growing need to develop critical thinking skills specifically oriented toward understanding and addressing climate change. To facilitate the assessment of these skills, a critical thinking in climate change (CTCC) test was developed. In the first step, 16 semistructured interviews were conducted with high school and university students. Drawing on insights from these interviews alongside previous research, a first version of the CTCC test comprising 41 items was administered to a sample of N=289 high school and university students from Austria, Belgium, and Germany. The test development process and subsequent evaluation were performed using Rasch modeling to determine test functioning. Through an iterative process that included three rounds of testing involving a total number of N=809 students and analyses, the test items were revised. The final version of the CTCC test consists of 24 items. Results from Rasch analysis suggest that the test has high item reliability (0.96) and person reliability (0.80), making it a useful tool for assessing critical thinking skills among high school graduates and first-semester university students.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (99)

  1. Intergovernmental Panel on Climate Change (IPCC, Climate change 2022—Impacts, in Adaptation and Vulnerability: Working Group II Contribution to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change, 1st ed. (Cambridge University Press, Cambridge, England, 2023), 10.1017/9781009325844.
  2. K. M. d’I. Treen, H. T. P. Williams, and S. J. O’Neill, Online misinformation about climate change, WIREs Clim. Change 11, e665 (2020).
  3. D. Ding, E. W. Maibach, X. Zhao, C. Roser-Renouf, and A. Leiserowitz, Support for climate policy and societal action are linked to perceptions about scientific agreement, Nat. Clim. Change 1, 462 (2011).
  4. J. A. Harvey et al., Internet blogs, polar bears, and climate-change denial by proxy, BioScience 68, 281 (2018).
  5. H. T. P. Williams, J. R. McMurray, T. Kurz, and F. Hugo Lambert, Network analysis reveals open forums and echo chambers in social media discussions of climate change, Global Environ. Change Rep. 32, 126 (2015).
  6. T. Dingler, B. Tag, and A. Vargo, Technologies to Support Critical Thinking in an Age of Misinformation (Dagstuhl Seminar 22172) (Leibniz-Zentrum für Informatik, Schloss Dagstuhl, 2022), 10.4230/DAGREP.12.4.72.
  7. J. Cook, P. Ellerton, and D. Kinkead, Deconstructing climate misinformation to identify reasoning errors, Environ. Res. Lett. 13, 024018 (2018).
  8. USGCRP, Climate Literacy: The Essential Principles of Climate Science (United States Global Change Research Program, Washington, DC, 2009).
  9. P. A. Facione, Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction (The Delphi Report), American Psychological Association (California Academic Press, Millbrae, CA, 1990).
  10. Council of the European Union, Council Recommendation on Learning for the Green Transition and Sustainable Development, Official Journal of the European Union (Council of the European Union, Brussels, 2022).
  11. Education, Audiovisual, and Culture Executive Agency, Citizenship Education at School in Europe (Publications Office, Luxembourg, 2017), 10.2797/612559.
  12. R. F. Yuretich, Encouraging critical thinking: Measuring skills in large introductory science classes, J. Coll. Sci. Teach. 33, 40 (2003), https://www.jstor.org/stable/26491263.
  13. Global Education Monitoring Report Team, Climate change communication and education country profiles: Approaches to greening education around the world, GEM Report No. 0000387867, UNESCO, 2023, 10.54676/XBVG6945.
  14. H. L. Bouygues, Teaching critical thinking in K-12: When there’s a will but not always a way, 2022, https://reboot-foundation.org/wp-content/uploads/2022/07/Reboot-White-Paper_NAEP-5.pdf [accessed March 31, 2025].
  15. K. L. Flores, G. S. Matkin, M. E. Burbach, C. E. Quinn, and H. Harding, Deficient critical thinking skills among college graduates: Implications for leadership, Educ. Philos. Theory 44, 212 (2012).
  16. L. Jarrett and G. Takacs, Secondary students’ ideas about scientific concepts underlying climate change, Environ. Educ. Res. 26, 400 (2020).
  17. D. Bhattacharya, K. Carroll Steward, and C. T. Forbes, Empirical research on K-16 climate education: A systematic review of the literature, J. Geosci. Educ. 69, 223 (2021).
  18. L. C. McDermott and P. S. Shaffer, Research as a guide for curriculum development: An example from introductory electricity. Part I: Investigation of student understanding, Am. J. Phys. 60, 994 (1992).
  19. D. F. Halpern, Thought and Knowledge: An Introduction to Critical Thinking (Psychology Press, New York, 2014).
  20. R. H. Ennis, Critical thinking and the curriculum, in Thinking Skills Instruction: Concepts and Techniques. Building Students’ Thinking Skills Series (National Education Association, Washington, DC, 1987).
  21. R. H. Ennis, Critical thinking assessment, Theory Pract. 32, 179 (1993).
  22. J. E. McPeck, Critical Thinking and Education, Issues and Ideas in Education Series (Robertson, Oxford 1981).
  23. D. F. Halpern, Teaching critical thinking for transfer across domains: Dispositions, skills, structure training, and metacognitive monitoring, Am. Psychol. 53, 449 (1998).
  24. D. Moseley et al., Frameworks for Thinking: A Handbook for Teaching and Learning (Cambridge University Press, Cambridge, England, 2005).
  25. R. D. Renaud and H. G. Murray, A comparison of a subject-specific and a general measure of critical thinking, Thinking Skill Creativity 3, 85 (2008).
  26. The Palgrave handbook of critical thinking in higher education, in Palgrave Handbooks, edited by M. Davies and R. Barnett (Palgrave Macmillan, New York, NY, 2015).
  27. E. R. Lai, Critical Thinking: A Literature Review (Pearson, 2011), https://www.researchgate.net/publication/297782058_Critical_thinking_A_literature_review.
  28. P. Hager and M. Kaye, Critical thinking in teacher education: A process-oriented research agenda, Aust. J. Teach. Educ. 17, 26 (1992).
  29. C. P. Dwyer, An evaluative review of barriers to critical thinking in educational and real-world settings, J. Intell. 11, 105 (2023).
  30. M. Lipman, Thinking in Education, 2nd ed. Reprinted (Cambridge University Press, Cambridge, England, 2003), 10.1017/CBO9780511840272.
  31. D. F. Halpern, Halpern Critical Thinking Assessment Items and Scoring Procedures (Schuhfried GmbH, Mödling, 2016).
  32. American Psychological Association, The assessment cyberguide for learning goals and outcomes, 2009, https://www.apa.org/ed/governance/bea/assessment-cyberguide-v2.pdf [accessed February 8, 2024].
  33. F. Sadidi, Development and Evaluation of a Teaching Unit in Particle Physics to Promote Students’ Critical Thinking (TU Dresden, Dresden, 2022), https://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-849186.
  34. D. T. Tiruneh, M. De Cock, A. G. Weldeslassie, J. Elen, and R. Janssen, Measuring critical thinking in physics: Development and validation of a critical thinking test in electricity and magnetism, Int. J. Sci. Math. Educ. 15, 663 (2017).
  35. J. Sermeus, M. De Cock, and J. Elen, Critical thinking in electricity and magnetism: Assessing and stimulating secondary school students, Int. J. Sci. Educ. 43, 1 (2021).
  36. P. C. Abrami, R. M. Bernard, E. Borokhovski, D. I. Waddington, C. A. Wade, and T. Persson, Strategies for teaching students to think critically: A meta-analysis, Rev. Educ. Res. 85, 275 (2015).
  37. R. H. Ennis, Critical thinking and subject specificity: Clarification and needed research, Educ. Res. 18, 4 (1989).
  38. R. H. Ennis, J. Millman, and T. Tomko, Cornell Critical Thinking Tests level X & level Z—Manual (Midwest Publications, Pacific Grove, CA, 1985).
  39. R. H. Ennis and E. Weir, The Ennis-Wier Critical Thinking Essay Test: An Instrument for Testing and Teaching (Pacific Grove, CA, 1985).
  40. G. Watson and E. Glaser, Watson-Glaser Critical Thinking Appraisal (Pearson Assessment, London, England, 2002).
  41. L. Niu, L. S. Behar-Horenstein, and C. W. Garvan, Do instructional interventions influence college students’ critical thinking skills? A meta-analysis, Educ. Res. Rev. 9, 114 (2013).
  42. D. T. Tiruneh, A. Verburgh, and J. Elen, Effectiveness of critical thinking instruction in higher education: A systematic review of intervention studies, High. Educ. Stud. 4, 1 (2014).
  43. J. H. McMillan, Enhancing college students’ critical thinking: A review of studies, Res. High. Educ. 26, 3 (1987).
  44. M. M.-Y. Yeung, J. W.-M. Yuen, J. M.-T. Chen, and K. K.-L. Lam, The efficacy of team-based learning in developing the generic capability of problem-solving ability and critical thinking skills in nursing education: A systematic review, Nurse Educ. Today 122, 105704 (2023).
  45. B. Harjo, B. Kartowagiran, and A. Mahmudi, Development of critical thinking skill instruments on mathematical learning high school, Int. J. Instr. 12, 149 (2019).
  46. I. J. Quitadamo, C. L. Faiola, J. E. Johnson, and M. J. Kurtz, Community-based inquiry improves critical thinking in general education biology, CBE Life Sci. Educ. 7, 327 (2008).
  47. C. Walsh, K. N. Quinn, C. Wieman, and N. G. Holmes, Quantifying critical thinking: Development and validation of the physics lab inventory of critical thinking, Phys. Rev. Phys. Educ. Res. 15, 010135 (2019).
  48. A. B. Heim, G. Lawrence, R. Agarwal, M. K. Smith, and N. G. Holmes, Perceptions of interdisciplinary critical thinking among biology and physics undergraduates, Phys. Rev. Phys. Educ. Res. 21, 010138 (2025).
  49. D. T. Tiruneh, X. Gu, M. De Cock, and J. Elen, Systematic design of domain-specific instruction on near and far transfer of critical thinking skills, Int. J. Educ. Res. 87, 1 (2018).
  50. J. C. Bonk and S. G. Smith, Alternative instructional strategies for creative and critical thinking in the accounting curriculum, J. Acc. Educ. 16, 261 (1998).
  51. S. Liu, G. Roehrig, D. Bhattacharya, and K. Varma, In-service teachers’ attitudes, knowledge and classroom teaching of global climate change, Sci. Educ. 24, 12 (2015), https://files.eric.ed.gov/fulltext/EJ1069990.pdf.
  52. T. Schubatzky et al., Entwicklung des concept-inventory CCCI-422 zu den naturwissenschaftlichen Grundlagen des Klimawandels, Z. Didakt. Naturwiss. 29, 10 (2023).
  53. T. Schubatzky, R. Wackermann, C. Haagen-Schützenhöfer, and C. Wöhlke, How well do German a-level students understand the scientific underpinnings of climate change?, Sustainability 16, 7264 (2024).
  54. S. Wildbichler, C. Haagen-Schützenhöfer, and T. Schubatzky, Students’ ideas about the scientific underpinnings of climate change: A systematic review of the literature, Stud. Sci. Educ. 61, 117 (2025).
  55. A. M. Bodzin, D. Anastasio, D. Sahagian, T. Peffer, C. Dempsey, and R. Steelman, Investigating climate change understandings of urban middle-level students, J. Geosci. Educ. 62, 417 (2014).
  56. A. N. Versprille and M. H. Towns, General chemistry students’ understanding of climate change and the chemistry related to climate change, J. Chem. Educ. 92, 603 (2015).
  57. A. Versprille, A. Zabih, T. A. Holme, L. McKenzie, P. Mahaffy, B. Martin, and M. Towns, Assessing student knowledge of chemistry and climate science concepts associated with climate change: Resources to inform teaching and learning, J. Chem. Educ. 94, 407 (2017).
  58. V. Dawson and K. Carson, Australian secondary school students’ understanding of climate change, Teach. Sci. 59, 9 (2013).
  59. D. P. Shepardson, D. Niyogi, A. Roychoudhury, and A. Hirsch, Conceptualizing climate change in the context of a climate system: Implications for climate and environmental education, Environ. Educ. Res. 18, 323 (2012).
  60. E. Boyes and M. Stanisstreet, The “greenhouse effect”: Children’s perceptions of causes, consequences and cures, Int. J. Sci. Educ. 15, 531 (1993).
  61. J. L. McCuin, K. Hayhoe, and D. Hayhoe, Comparing the effects of traditional vs. misconceptions-based instruction on student understanding of the greenhouse effect, J. Geosci. Educ. 62, 445 (2014).
  62. D. Pruneau, H. Gravel, W. Bourque, and J. Langis, Experimentation with a socio-constructivist process for climate change education, Environ. Educ. Res. 9, 429 (2003).
  63. D. P. Shepardson, D. Niyogi, S. Choi, and U. Charusombat, Students’ conceptions about the greenhouse effect, global warming, and climate change, Clim. Change 104, 481 (2011).
  64. D. Lombardi, G. M. Sinatra, and E. M. Nussbaum, Plausibility reappraisals and shifts in middle school students’ climate change conceptions, Learn. Instr. 27, 50 (2013).
  65. R. Bellamy, Public perceptions of climate tipping points, Publ. Understand. Sci. 32, 1033 (2023).
  66. M. S. McCaffrey and S. M. Buhr, Clarifying climate confusion: Addressing systemic holes, cognitive gaps, and misconceptions through climate literacy, Phys. Geogr. 29, 512 (2008).
  67. L. Zangori, A. Peel, A. Kinslow, P. Friedrichsen, and T. D. Sadler, Student development of model-based reasoning about carbon cycling and climate change in a socio-scientific issues unit, J. Res. Sci. Teach. 54, 1249 (2017).
  68. K. I. Danielson and K. D. Tanner, Investigating undergraduate science students’ conceptions and misconceptions of ocean acidification, CBE Life Sci. Educ. 14, ar29 (2015).
  69. V. Papadimitriou, Prospective primary teachers’ understanding of climate change, greenhouse effect, and ozone layer depletion, J. Sci. Educ. Technol. 13, 299 (2004).
  70. D. Lombardi and G. M. Sinatra, College students’ perceptions about the plausibility of human-induced climate change, Res. Sci. Educ. 42, 201 (2012).
  71. D. Read, A. Bostrom, M. G. Morgan, B. Fischhoff, and T. Smuts, What do people know about global climate change? 2. Survey studies of educated lay people, Risk Anal. 14, 971 (1994).
  72. R. M. V. Gowda, J. C. Fox, and R. D. Magelky, Students’ understanding of climate change: Insights for scientists and educators, Bull. Am. Meteorol. Soc. 78, 2232 (1997).
  73. Schülervorstellungen und Physikunterricht: Ein Lehrbuch für Studium, Referendariat und Unterrichtspraxis, edited by H. Schecker, T. Wilhelm, M. Hopf, and R. Duit (Springer, Berlin, Heidelberg, 2018), 10.1007/978-3-662-57270-2.
  74. S. Wildbichler et al., Tutorials in climate change, Phys. Teach. 63, 414 (2025).
  75. X. Liu and W. J. Boone, Applications of Rasch Measurement in Science Education (JAM Press, Maple Grove, MN, 2006).
  76. X. Liu, Using and Developing Measurement Instruments in Science Education: A Rasch Modeling Approach (Information Age Publishing, Charlotte, NC, 2010).
  77. W. J. Boone, J. R. Staver, and M. S. Yale, Rasch Analysis in the Human Sciences (Springer, Dordrecht, 2014), 10.1007/978-94-007-6857-4.
  78. T. G. Bond and C. M. Fox, Applying the Rasch Model: Fundamental Measurement in the Human Sciences, 3rd ed. (Routledge, Taylor & Francis Group, New York, London, 2015).
  79. M. Planinic, W. J. Boone, A. Susac, and L. Ivanjek, Rasch analysis in physics education research: Why measurement matters, Phys. Rev. Phys. Educ. Res. 15, 020111 (2019).
  80. L. Ivanjek, L. Morris, T. Schubatzky, M. Hopf, Jan-Philipp Burde, C. Haagen-Schützenhöfer, L. Dopatka, V. Spatz, and T. Wilhelm, Development of a two-tier instrument on simple electric circuits, Phys. Rev. Phys. Educ. Res. 17, 020123 (2021).
  81. K. Matejak Cvenic, M. Planinic, A. Susac, L. Ivanjek, K. Jelicic, and M. Hopf, Development and validation of the conceptual survey on wave optics, Phys. Rev. Phys. Educ. Res. 18, 010103 (2022).
  82. V. Mešić et al., Measuring students’ conceptual understanding of wave optics: A Rasch modeling approach, Phys. Rev. Phys. Educ. Res. 15, 010115 (2019).
  83. I. Neumann, K. Neumann, and R. Nehm, Evaluating instrument quality in science education: Rasch-based analyses of a nature of science test, Int. J. Sci. Educ. 33, 1373 (2011).
  84. R. Berger, C. Kulgemeyer, and P. Lensing, Ein multiple-choice-test zum konzeptuellen verständnis der kraftwirkung auf ladungsträger in statischen elektrischen und magnetischen Feldern, Z. Didakt. Naturwiss. 25, 197 (2019).
  85. G. Rasch, Probabilistic Models for Some Intelligence and Attainment Tests (Danmarks Paedagogiske Institut, Copenhagen, 1960).
  86. A. E. Kazdin, Research Design in Clinical Psychology, 5th ed. (Cambridge University Press, Cambridge, England, 2021), 10.1017/9781108993647.
  87. K. A. Ericsson and H. A. Simon, How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking, Mind Cult. Act. 5, 178 (1998).
  88. A. Reinhart et al., Think-aloud interviews: A tool for exploring student statistical reasoning, J. Stat. Data Sci. Educ. 30, 100 (2022).
  89. U. Kuckartz and S. Rädiker, Qualitative Inhaltsanalyse. Methoden, Praxis, Umsetzung mit Software und künstlicher Intelligenz, 6th ed. (Juventa Verlag, Weinheim, 2024).
  90. J. M. Linacre, A user’s guide to Winsteps ministeps Rasch-model computer programs, 2025, https://www.winsteps.com/a/Winsteps-Manual.pdf [accessed April 11, 2025].
  91. See Supplemental Material at http://link.aps.org/supplemental/10.1103/jnbq-1qyk for the English translation of the CTCC test (with the correct answers marked) and also the removed items throughout the development process—including a table listing their Infit and Outfit MNSQ values, correlations, and reasons for removal.
  92. M. Henle, On the relation between logic and thinking, Psychol. Rev. 69, 366 (1962).
  93. M. Schwichow, S. Croker, C. Zimmerman, T. Höffler, and H. Härtig, Teaching the control-of-variables strategy: A meta-analysis, Dev. Policy Rev. 39, 37 (2016).
  94. J. M. Linacre, Critical eigenvalue sizes (variances) in standardized residual principal components analysis (PCA), https://www.rasch.org/rmt/rmt191h.htm.
  95. S. Mohamad Hasim, R. Rosli, and L. Halim, A quantitative case study of secondary school students’ level of statistical thinking, Eurasia J. Math. Sci. Technol. Educ. 20, em2421 (2024).
  96. M. Marshman and P. K. Dunn, Improving statistical thinking, Math. Educ. Res. J. 36, 1 (2024).
  97. L. Ivanjek, A. Susac, M. Planinic, A. Andrasevic, and Z. Milin-Sipus, Student reasoning about graphs in different contexts, Phys. Rev. Phys. Educ. Res. 12, 010106 (2016).
  98. N. Ramadhani, K. Salija, and S. Baa, The correlation between students’ critical thinking skills and their reading comprehension at undergrate study program English Department Faculty of Languages and Literature Universitas Negeri Makassar, Klasikal J. Educ. Lang. Teach. Sci. 5, 25 (2023).
  99. K. Y. L. Ku, Assessing students’ critical thinking performance: Urging for measurements using multi-response format, Thinking Skills Creating 4, 70 (2009).

Outline

Information

Phys. Rev. Phys. Educ. Res. 21, 020154– Published 15 December, 2025
  • Received 14 May 2025
  • Accepted 12 November 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation