[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] Physics education researchers (PER) often analyze student data with single-level regression models (e.g., linear and logistic regression). However, education datasets can have hierarchical structures, such as students nested within courses, that single-level models fail to account for. The improper use of single-level models to analyze hierarchical datasets can lead to biased findings. Hierarchical models (also known as multilevel models) account for this hierarchical nested structure in the data. In this publication, we outline the theoretical differences between how single-level and multilevel models handle hierarchical datasets. We then present analysis of a dataset from 112 introductory physics courses using both multiple linear regression and hierarchical linear modeling to illustrate the potential impact of using an inappropriate analytical method on PER findings and implications. Research can leverage multi-institutional datasets to improve the field’s understanding of how to support student success in physics. There is no post hoc fix, however, if researchers use inappropriate single-level models to analyze multilevel datasets. To continue developing reliable and generalizable knowledge, PER should use hierarchical models when analyzing hierarchical datasets. The Supplemental Material includes a sample dataset, R code to model the building and analysis presented in the paper, and an HTML output from the R code.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] While many studies have examined the structure, validity, and reliability of the Force Concept Inventory, far less research has been performed on other conceptual instruments in widespread use in physics education research. This study performs a confirmatory analysis of the Conceptual Survey of Electricity and Magnetism (CSEM) guided by a theoretical model of expert understanding of electricity and magnetism. Multidimensional Item Response Theory (MIRT) with the discrimination matrix constrained to the theoretical model was used to investigate two large datasets ($N_1=2014$ and $N_2=2657$) from two research universities in the United States. The optimal model identified by MIRT was similar, but not identical, for the two datasets and had very good model fit with comparative fit indices of 0.975 and 0.984, respectively. The most parsimonious optimal model required 23 independent principles of electricity and magnetism and was significantly better fitting than a more general model dividing the CSEM into 6 general topics. The optimal models for the two samples were quite similar, sharing 22 of a possible 26 conceptual principles. Most of the overall item difficulties and discriminations were significantly different between the two samples; however, the rank order of the overall difficulty and discrimination were generally similar. There was much more similarity between the discrimination by item of the individual principles. Five items had a difficulty ranking that was substantially different between the two samples, indicating that while generally similar, relative difficulty does depend on the student population and instructional environment.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] Physics education researchers (PER) commonly use complete-case analysis to address missing data. For complete-case analysis, researchers discard all data from any student who is missing any data. Despite its frequent use, no PER article we reviewed that used complete-case analysis provided evidence that the data met the assumption of missing completely at random necessary to ensure accurate results. Not meeting this assumption raises the possibility that prior studies have reported biased results with inflated gains that may obscure differences across courses. To test this possibility, we compared the accuracy of complete-case analysis and multiple imputation (MI) using simulated data. We simulated the data based on prior studies such that students who earned higher grades participated at higher rates, which made the data missing at random. PER studies seldom use MI, but MI uses all available data, has less stringent assumptions, and is more accurate and more statistically powerful than complete-case analysis. Results indicated that complete-case analysis introduced more bias than MI and this bias was large enough to obscure differences between student populations or between courses. We recommend that the PER community adopt the use of MI for handling missing data to improve the accuracy in research studies.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] The application of social network analysis (SNA) has recently grown prevalent in science, technology, engineering, and mathematics education research. Research on classroom networks has led to greater understandings of student persistence in physics majors, changes in their career-related beliefs (e.g., physics interest), and their academic success. In this paper, we aim to provide a practitioner’s guide to carrying out research using SNA, including how to develop data collection instruments, setup protocols for gathering data, as well as identify network methodologies relevant to a wide range of research questions beyond what one might find in a typical primer. We illustrate these techniques using student anxiety data from active-learning physics classrooms. We explore the relationship between students’ physics anxiety and the social networks they participate in throughout the course of a semester. We find that students’ with greater numbers of outgoing interactions are more likely to experience decrease in anxiety even while we control for pre-anxiety, gender, and final course grade. We also explore the evolution of student networks and find that the second half of the semester is a critical period for participating in interactions associated with decreased physics anxiety. Our study further supports the benefits of dynamic group formation strategies that give students an opportunity to interact with as many peers as possible throughout a semester. To complement our guide to SNA in education research, we also provide a set of tools for other researchers to use this approach in their work—the SNA toolbox—that can be accessed on GitHub.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] A central aim of physics education research is to understand the processes of learning and use that understanding to inform instruction. To this end, researchers often conduct studies to measure the effect of classroom interventions on student outcomes. Many of these intervention studies have provided an empirical foundation of reformed teaching techniques, such as active engagement. However, many times there is not sufficient evidence to conclude that the intervention had the intended effect, and these null results often end up in the proverbial file drawer. In this paper, we argue that null results can make significant contributions to physics education research, even if the results are not statistically significant. First, we review social science and biomedical research that documents widespread publication bias against null results, exploring why it occurs and how it can hurt the field. We then present three cases from physics education research to highlight how studies that yield null results can contribute to our understanding of teaching and learning. Finally, we distill from these studies some general principles for learning from null results, proposing that we should evaluate them not on whether they reject the null hypothesis but according to their potential for generating new understanding.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] How data are collected and how they are analyzed is typically described in the literature, but how the data are encoded is often not described in detail. In this paper, we discuss how data typically gathered in PER are encoded and how the choice of encoding plays a role in data analysis. We describe the kinds of data that are found when using short answer, multiple choice, Likert-scale, ranking task, and free response questions in terms of nominal, ordinal, interval, and ratio data. We discuss the mathematical operations that are available for each kind of data and how this affects ways that similarity and difference between student responses can be determined, a topic we discuss in terms of measures of distances and correlation. Finally, we use several papers from the literature to discuss ways in which data have been encoded and analyzed, with examples of normalized gain, factor analysis, model analysis, cluster analysis, and the investigation of epistemological agreement. We highlight both strengths and weaknesses of the data encoding approaches used in these studies. Our goal is not a comprehensive review, but one that is illustrative and can help researchers understand their own and each other’s work more deeply.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] While other fields such as statistics and education have examined various issues with quantitative work, few studies in physics education research (PER) have done so. We conducted a two-phase study to identify and to understand the extent of these issues in quantitative PER. During phase 1, we conducted a focus group of three experts in this area, followed by six interviews. Subsequent interviews refined our plan. Both the focus group and interviews revealed issues regarding the lack of details in sample descriptions, lack of institutional or course contextual information, lack of reporting on limitation, and overgeneralization or overstatement of conclusions. During phase 2, we examined 72 manuscripts that used four conceptual or attitudinal assessments (Force Concept Inventory, Conceptual Survey of Electricity and Magnetism, Brief Electricity and Magnetism Assessment, and Colorado Learning Attitudes about Science Survey). Manuscripts were coded on whether they featured various sample descriptions, institutional or course context information, limitations, and whether they overgeneralized conclusions. We also analyzed the data to see if reporting has changed from the earlier periods to more recent times. We found that not much has changed regarding sample descriptions and institutional or course context information, but reporting and overgeneralizing conclusions have improved over time. We offer some questions for researchers, reviewers, and readers in PER to consider when conducting or using quantitative work.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] The notion of quantitative physics education research (PER) so far has been mostly limited to the mere use of statistical methods or use of computational tools for analyzing numerical data. Little attention, in fact, has been given to the underpinnings of this research paradigm. To fill the gap, this theoretical paper addresses key and yet often tacit (or even misunderstood) principles of three commonly used quantitative genres in PER, which I, respectively, refer to as measurement (quantification of individual constructs), controlled exploration of relations (quantification of relationships between multiple constructs through controlled experiments), and data mining (quantification of new information from large datasets). For each genre, I elucidate the paradigmatic basis by focusing on its ontological assumptions (theories about the reality under quantitative investigation), epistemological commitments (views about the knowledge gained through quantitative investigation), and methodological implications for empirical investigations. Although framed in the context of physics education research, the discussions herein are applicable to other discipline-based or general education research employing quantitative methods.

In the present theoretical study, we introduce the entropy concept into Chi’s ontological shift theory. Chi distinguishes between two categories of process phenomena, direct and emergent, and claims that incorrectly considering emergent processes as direct ones is one of the sources of students’ robust scientific misconceptions. The present study aims to address the needs of high school and undergraduate physics, chemistry, and engineering students being already familiar with the basics of mechanics and the kinetic molecular theory. Acknowledging the contribution of the ontological shift category theory to improving the learning of science, the present paper aims at taking this theory one step further. We argue that more information about scientific phenomena could be gained if we view direct and emergent phenomena as edges of the same scale level of emergency. We show that entropy can be used to evaluate the level of emergency of physical processes. We believe that interpreting scientific phenomena in terms of level of emergency and entropy might promote students’ understanding about the underlying mechanisms explaining these phenomena, as well as about the concept of entropy itself. We provide two pedagogical examples of teaching heat and sound using the level of emergency scale and the entropy concept. We demonstrate analytically in these terms: (a) the development of the heat flow rate equation; and (b) the adiabatic nature of the sound propagation process.

Proper understanding of and learning from physics phenomena and experiments requires—among other competencies—flexible and coherent use of multiple representations (MRs). These can include everything from the “enactive” or “operational” manipulation of the experimental devices and materials to the most abstract level of a mathematical formulation of the phenomenon investigated in a given experiment. An essential prerequisite for effective work with MRs is the ability to achieve coherence between different representations. However, research indicates that the level of representational coherence ability of learners across various age groups is low. In order to improve this state of affairs, an intervention study about the use of MRs related to physics experiments was carried out (content area geometrical optics). Specific learning tasks (representational activity tasks, RATs) were designed which explicitly require various types of coherent connections, such as comparing, completing, and correcting representations. In a quasiexperimental repeated measurement study ($N=302$) using a multilevel analysis for measuring changes, a comparison of a treatment group learning with RATs vs a control group learning with conventional tasks was carried out (with identical content, lesson plans, and duration of the intervention in both groups; moreover, each of the four schools had corresponding classes of both groups. They were taught by the same teacher). Results showed a highly significant and practically relevant effect on students’ representational coherence ability ($p<0.001$; $d=0.69$). The positive effect of RATs could still be found six weeks after the end of the intervention ($p<0.001$; $d=0.43$). Several covariates (gender, pre-instructional knowledge in physics, mathematics, three facets of intelligence) were analyzed, with no or small influence on these effects. Finally, some limitations and implications of the study for classroom practice and further research are discussed.

Cultivating institutional transformation has been of recent interest in education research. This theoretical paper presents six principles for supporting sustained change efforts at the department level. Considering change efforts at the level of “principles” is valuable because principles are grounded in theoretical and empirical knowledge, but are abstract enough to be adapted to many contexts. For each principle we argue for its value, drawing on previous literature in higher education, organizational change, discipline-based education research, and design thinking. We then give illustrative examples of how each principle was embodied within the Departmental Action Team (DAT) project. The DAT project facilitates the implementation of effective changes within university science, technology, engineering, and mathematics departments. We conclude with a discussion of how these principles can be applicable across a variety of institutional transformation efforts.

Models of physical systems are used to explain and predict experimental results and observations. The Modeling Framework for Experimental Physics describes the process by which physicists revise their models to account for the newly acquired observations, or change their apparatus to better represent their models when they encounter discrepancies between actual and expected behavior of a system. While modeling is a nationally recognized learning outcome for undergraduate physics lab courses, no assessments of students’ model-based reasoning exist for upper-division labs. As part of a larger effort to create two assessments of students’ modeling abilities, we used the Modeling Framework to develop and code think-aloud problem-solving activities centered on investigating an inverting amplifier circuit. This study is the second phase of a multiphase assessment instrument development process. Here, we focus on characterizing the range of modeling pathways students employ while interpreting the output signal of a circuit functioning far outside its recommended operation range. We end by discussing four outcomes of this work: (i) Students engaged in all modeling subtasks, and they spent the most time making measurements, making comparisons, and enacting revisions; (ii) each subtask occurred in close temporal proximity to all other subtasks; (iii) sometimes, students propose causes that do not follow from observed discrepancies; (iv) similarly, students often rely on their experiential knowledge and enact revisions that do not follow from articulated proposed causes.

Quantum mechanics is a notoriously counterintuitive subject within physics and has been the subject of a number of studies at the undergraduate level, and a few pioneering studies at the graduate level. The sketching of wave functions in a confining well is in one sense one of the most basic activities in quantum mechanics. But in another sense, it may be viewed as a rather advanced skill, as it requires the coherent inclusion of a number of details of the wave function, such as wavelength, probability amplitude, and boundary conditions, among others. Although sketching a wave function is not a common activity at the graduate level, a great deal of graduate work is concerned with the aforementioned details, especially boundary conditions. Whether it is seen as a basic skill, or as linked to higher-level understanding, sketching a wave function is an ability that physics graduate students should have. Here, we report on graduate students’ ability to sketch wave functions in an asymmetric potential well. We find that the frequency of many errors is not significantly reduced from pretest to posttest, meaning that many errors persist through to the end of graduate quantum mechanics instruction. We find that only 5% of graduate students tested can sketch the 2nd excited state wave function without errors. We include quantitative and diagrammatic descriptions of student errors covering a broader range of misunderstandings than has previously been identified, and we include interview information that speaks to the persistence of some of the errors.

Mathematical reasoning with algebraic and geometric representations is essential for success in upper-division and graduate-level physics courses. Complex algebra requires student to fluently move between algebraic and geometric representations. By designing a task for middle-division physics students to translate a geometric representation to several algebraic representations, we identified students’ facilities and difficulties with complex number algebra. Students were successful when they isolated approaches, which was characterized by students’ use of circle trigonometry, triangle trigonometry, the Pythagorean theorem, and the square of the norm approaches. When students were not successful, they inappropriately mixed approaches to the problem, including failing to distinguish between triangle and circle trigonometry.

We present the design, statistical analysis, and validation of a questionnaire to assess students’ knowledge about basic aspects of quantum mechanics (QM). The QM evaluation (QME) is a true-false and multiple-choice mixed questionnaire that features 10 two-tier items spanning three relevant themes in quantum mechanics: wave behavior of matter, measurement, and atoms and electrons behavior. Its validity was assessed through a pilot administration to students and interviews with course instructors. We checked its internal consistency using both classic test theory and Rasch analysis to account for the different difficulty of each tier and for different scoring methods of the items. The questionnaire was administered to about 450 undergraduate physics students and high school physics teachers. Data show that it is a reliable instrument and all items have a good discriminatory power. Since the test does not require an advanced mathematical knowledge, it ideally lends itself to probe students’ knowledge about quantum mechanics in a variety of university courses, from the introductory ones to those more formal and mathematically oriented.

A considerable body of research shows that individuals who demonstrate a wide variety of metacognitive skills perform better on scientific problem solving. We developed a way of implementing homework in an undergraduate physics course to promote metacognitive skills. This novel homework design includes both an individual and a team phase and was implemented in a team-based learning environment. We developed a coding table to conduct both qualitative and quantitative analyses of students’ expressions of metacognitive thinking. Results indicate that students’ expressions of metacognitive thinking improved during the term. Data also show that team discussion helps improve metacognitive skills.

Introductory physics lab instruction is undergoing a transformation, with increasing emphasis on developing experimentation and critical thinking skills. These changes present a need for standardized assessment instruments to determine the degree to which students develop these skills through instructional labs. In this article, we present the development and validation of the physics lab inventory of critical thinking (PLIC). We define critical thinking as the ability to use data and evidence to decide what to trust and what to do. The PLIC is a 10-question, closed-response assessment that probes student critical thinking skills in the context of physics experimentation. Using interviews and data from 5584 students at 29 institutions, we demonstrate, through qualitative and quantitative means, the validity and reliability of the instrument at measuring student critical thinking skills. This establishes a valuable new assessment instrument for instructional labs.

In this paper, we present a case study of a pair of students as they use nondisciplinary communicative practices to mechanistically reason about binary star dynamics. To do so, we first review and bring together the theoretical perspectives of social semiotics and embodied cognition, therein developing a new methodological approach for analyzing student interactions during the learning of physics (particularly for those interactions involving students’ bodies). Through the use of our new approach, we are able to show how students combine a diverse range of meaning-making resources into complex, enacted analogies, thus forming explanatory models that are grounded in embodied intuition. We reflect on how meaning-making resources—even when not physically persistent—can act as coordinating hubs for other resources as well as how we might further nuance the academic conversation around the role of the body in physics learning.

This theoretical paper introduces a new way to view and characterize learning astronomy. It describes a framework, based on results from empirical data, analyzed through standard qualitative research methodology, in which a theoretical model for a vital competency of learning astronomy is proposed: reading the sky, a broad description under with various skills and competencies are included. This model takes into account not only disciplinary knowledge but also disciplinary discernment and extrapolating three dimensionality. Together, these constitute the foundation for the competency referred to as reading the sky. In this paper, these competencies are described and discussed and merged to form a new framework vital for learning astronomy to better match the challenges students face when entering the discipline of astronomy.