- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Psychometric evaluation of the culture around systemic change survey: A tool for assessing facets of departmental culture in physics

Diana Sachmpazidi* and Mike Verostek*

Chandra Turpen and Jayna Petrella

  • *These authors contributed equally to this work.
PDF

Phys. Rev. Phys. Educ. Res. 22, 010107 – Published 23 January, 2026

DOI: https://doi.org/10.1103/mt34-y14m

Abstract

Physics programs are continually evolving to better support student learning and meet the diverse needs of their students. Achieving many of these goals requires not only structural adjustments but also fundamental shifts in departmental culture. Recognizing this, disciplinary organizations in physics have placed systemic change and equity at the center of reform efforts, identifying them as essential pillars of meaningful and sustainable change. Yet, tools for assessing departmental culture around educational change remain limited. In this study, we introduce the Culture around Systemic Change Survey (CSCS), designed to assess departmental culture around pursuing any program-level change (undergraduate and/or graduate) aimed at improving student experience or outcomes (e.g., curriculum revisions, instructional practices, and efforts to strengthen departmental climate). This new instrument measures faculty and staff perceptions of their department’s “current” and “ideal” states with respect to their typical approaches to leading change efforts. Using responses from the “current” scale only (N=111 participants across 33 departments), we conducted a psychometric evaluation of the CSCS. Exploratory factor analysis supported a five-factor structure, including open-mindedness (OM), student involvement (SI), collective interpretation of evidence (CIE), sustainability (S), and disruption of systemic injustices (DSI). As survey development is an iterative process, future work will focus on refinement and confirmatory analysis. This work sets the foundation for conducting population studies that assess the state of progress of the physics community along an equitable and systemic culture to pursuing educational change.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (98)

  1. J. Vespa, D. M. Armstrong, and L. Medina, Demographic turning points for the United States: Population projections for 2020 to 2060 (Report No. P25-1144). U.S. Census Bureau (2020), https://www.census.gov/library/publications/2020/demo/p25‐1144.html.
  2. A. Kezar and P. D. Eckel, The effect of institutional culture on change strategies in higher education: Universal principles or culturally responsive concepts?, J. Higher Educ. 73, 435 (2002).
  3. American Physical Society, APS-IDEA: The Inclusion, Diversity, and Equity Alliance, American Physical Society, https://www.aps.org/initiatives/inclusion/idea.
  4. American Physical Society & American Association of Physics Teachers, Departmental Action Leadership Institute (DALI), EP3 Guide, https://ep3guide.org/dali/.
  5. American Institute of Physics Foundation, TEAM-UP Together, American Institute of Physics, https://www.aip.org/foundation/team-up-together.
  6. K. Foote, X. Neumeyer, C. Henderson, M. Dancy, and R. Beichner, Diffusion of research-based instructional strategies: The case of SCALE-UP, Int. J. STEM Educ. 1, 10 (2014).
  7. T. Hodapp and K. Woodle, A bridge between undergraduate and doctoral degrees, Phys. Today 70, No. 2, 50 (2017).
  8. E. Schein, Organizational Culture and Leadership (Jossey-Bass, San Francisco, 2004).
  9. G. M. Quan, J. C. Corbo, N. D. Finkelstein, A. Pawlak, K. Falkenberg, C. Geanious, C. Ngai, C. Smith, S. Wise, M. E. Pilgrim, and D. L. Reinholz, Designing for institutional transformation: Six principles for department-level interventions, Phys. Rev. Phys. Educ. Res. 15, 010141 (2019).
  10. C. L. Fry, Achieving systemic change: A sourcebook for advancing and funding undergraduate STEM education, Association of American Colleges and Universities, 2014.
  11. A. Kezar, How Colleges Change: Understanding, Learning, and Enacting Change (Routledge, New York, 2014).
  12. K. E. Weick and R. E. Quinn, Organizational change and development, Annu. Rev. Psychol. 50, 361 (1999).
  13. P. D. Eckel and A. J. Kezar, Taking the Reins: Institutional Transformation in Higher Education (Greenwood Publishing Group, Westport, CT, 2003).
  14. D. Sachmpazidi, C. Turpen, J. Petrella, R. P. Dalka, and F. N. Abdurrahman, Recognizing dominant cultures around assessment and educational change in physics programs, Phys. Rev. Phys. Educ. Res. 20, 010132 (2024).
  15. American Association for the Advancement of Science, Vision and Change in Undergraduate Biology Education: A Call to Action, American Association for the Advancement of Science, Washington, DC, 2011.
  16. J. Kania and M. Kramer, Collective impact, Stanford Soc. Innov. Rev. 9, 36 (2011).
  17. C. Henderson, A. Beach, and N. Finkelstein, Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature, J. Res. Sci. Teach. 48, 952 (2011).
  18. A. Kezar, Understanding sensemaking/sensegiving in transformational change processes from the bottom up, Higher Educ. 65, 761 (2013).
  19. M. T. Hora, J. Bouwma-Gearhart, and H. J. Park, Data driven decision-making in the era of accountability: Fostering faculty data cultures for learning, Rev. Higher Educ. 40, 391 (2017).
  20. A. Datnow, Collaboration and contrived collegiality: Revisiting Hargreaves in the age of accountability, J. Educ. Change 12, 147 (2011).
  21. E. M. Bensimon, Closing the achievement gap in higher education: An organizational learning perspective, New Dir. Higher Educ. 2005, 99 (2005).
  22. E. Bertschinger, Systemic change: TEAM-UP and beyond, presented at PER Conf. 2020, Virtual Conference, 10.1119/perc.2020.pr.Bertschinger.
  23. D. T. Holt, A. A. Armenakis, H. S. Feild, and S. G. Harris, Readiness for organizational change: The systematic development of a scale, J. Appl. Behav. Sci. 43, 232 (2007).
  24. A. A. Armenakis, J. B. Bernerth, J. P. Pitts, and H. J. Walker, Organizational change recipients’ beliefs scale: Development of an assessment instrument, J. Appl. Behav. Sci. 43, 481 (2007).
  25. M. Beer and N. Nohria, Cracking the code of change, Harv. Bus. Rev. 78, 133 (2000), http://www.ceewl.ca/12599-PDF-ENG.PDF#page=89.
  26. C. Ngai, M. E. Pilgrim, D. L. Reinholz, J. C. Corbo, and G. M. Quan, Developing the DELTA: Capturing cultural changes in undergraduate departments, CBE Life Sci. Educ. 19, ar15 (2020).
  27. M. Cooley, Information design, in Human-Centered Design (MIT Press, Cambridge, MA, 2000), pp. 59–81, https://mitpress.mit.edu/9780262100694/information-design/?utm_source=chatgpt.com.
  28. D. Sachmpazidi and C. Turpen, Measuring what matters: A human-centered design approach to survey development around departmental culture in physics, presented at PER Conf. 2025, Washington, DC, 10.1119/perc.2025.pr.Sachmpazidi.
  29. S. Chasteen, J. C. Corbo, R. Dalka, and C. Turpen, Results from the 2020 EP3 Survey to Physics Department Chairs: External Report (American Physical Society, College Park, MD, 2020).
  30. University of Maryland, UMD survey instrumentation user interface, available at https://umdsurvey.umd.edu/homepage/ui (2025) [accessed July 16, 2025].
  31. J. Blair, R. F. Czaja, and E. A. Blair, Designing Surveys: A Guide to Decisions and Procedures, 3rd ed. (Sage Publications, Thousand Oaks, CA, 2013).
  32. C. T. Beck and R. K. Gable, Ensuring content validity: An illustration of the process, J. Nurs. Meas. 9, 201 (2001).
  33. A. W. Meade and S. B. Craig, Identifying careless responses in survey data, Psychol. Methods 17, 437 (2012).
  34. D. T. Berry, M. W. Wetter, R. A. Baer, L. Larsen, C. Clark, and K. Monroe, MMPI-2 random responding indices: Validation using a self-report methodology, Psychol. Assess. 4, 340 (1992).
  35. C. K. Brower, Too long and too boring: The effects of survey length and interest on careless responding, Master’s thesis, Wright State University, 2018.
  36. K. Bhaskaran and L. Smeeth, What is the difference between missing completely at random and missing at random?, Int. J. Epidemiol. 43, 1336 (2014).
  37. R. J. Little, A test of missing completely at random for multivariate data with missing values, J. Am. Stat. Assoc. 83, 1198 (1988).
  38. D. B. Rubin, Inference and missing data, Biometrika 63, 581 (1976).
  39. B. Muthén, D. Kaplan, and M. Hollis, On structural equation modeling with data that are not missing completely at random, Psychometrika 52, 431 (1987).
  40. J. Nissen, R. Donatello, and B. Van Dusen, Missing data and bias in physics education research: A case for using multiple imputation, Phys. Rev. Phys. Educ. Res. 15, 020106 (2019).
  41. R. L. Brown, Efficacy of the indirect approach for estimating structural equation models with missing data: A comparison of five methods, Struct. Equation Model. 1, 287 (1994).
  42. G. L. Mazza, C. K. Enders, and L. S. Ruehlman, Addressing item-level missing data: A comparison of proration and full information maximum likelihood estimation, Multivariate Behav. Res. 50, 504 (2015).
  43. J. W. Graham and D. L. Coffman, Structural equation modeling with missing data, in Handbook of Structural Equation Modeling (The Guilford Press, New York, 2015), pp. 277–294.
  44. S. van Buuren, Flexible Imputation of Missing Data, 2nd ed. (CRC Press, Boca Raton, FL, 2018).
  45. C. K. Enders and D. L. Bandalos, The relative performance of full information maximum likelihood estimation for missing data in structural equation models, Struct. Equation Model. 8, 430 (2001).
  46. V. Nassiri, A. Lovik, G. Molenberghs, and G. Verbeke, On using multiple imputation for exploratory factor analysis of incomplete data, Behav. Res. Methods 50, 501 (2018).
  47. W. Revelle, psych: Procedures for Psychological, Psychometric, and Personality Research, r Package Version 2.4.6 (Northwestern University, Evanston, Illinois, 2024).
  48. R Core Team, r: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, Austria, 2024).
  49. J. R. Van Ginkel and P. M. Kroonenberg, Using generalized procrustes analysis for multiple imputation in principal component analysis, J. Classif. 31, 242 (2014).
  50. T. C. Bates, M. C. Neale, and H. H. Maes, umx: A library for structural equation and twin modelling in r, Twin Res. Hum. Genet. 22, 27 (2019).
  51. R. B. Cattell, The scree test for the number of factors, Multivar. Behav. Res. 1, 245 (1966).
  52. J. L. Horn, A rationale and test for the number of factors in factor analysis, Psychometrika 30, 179 (1965).
  53. H. F. Kaiser, The application of electronic computers to factor analysis, Educ. Psychol. Meas. 20, 141 (1960).
  54. G. R. Norman and D. L. Streiner, Biostatistics: The Bare Essentials (People’s Medical Publishing House, Shelton, CT, 2014).
  55. J. Hair, W. Black, B. Babin, and R. Anderson, Multivariate Data Analysis (Pearson, New York, 2010).
  56. American Psychological Association, National Council on Measurement in Education, and Joint Committee on Standards for Educational and Psychological Testing (U.S., Standards for Educational and Psychological Testing (American Educational Research Association, Washington, DC, 2014).
  57. E. Knekta, C. Runyon, and S. Eddy, One size doesn’t fit all: Using factor analysis to gather validity evidence when using surveys in your research, CBE Life Sci. Educ. 18, rm1 (2019).
  58. L. J. Cronbach, Coefficient alpha and the internal structure of tests, Psychometrika 16, 297 (1951).
  59. G. J. Geldhof, K. J. Preacher, and M. J. Zyphur, Reliability estimation in a multilevel confirmatory factor analysis framework, Psychol. Methods 19, 72 (2014).
  60. D. McNeish, Thanks coefficient alpha, we’ll take it from here, Psychol. Methods 23, 412 (2018).
  61. W. Revelle and D. M. Condon, Reliability from α to ω: A tutorial, Psychol. Assess. 31, 1395 (2019).
  62. R. P. McDonald, Test Theory: A Unified Treatment (Lawrence Erlbaum Associates, Hillsdale, NJ, 1999), see Chaps. 6, 9, and 10 for discussion of theory and application of coefficient omega.
  63. R. L. Gorsuch, Factor Analysis (Lawrence Erlbaum Associates, Inc., Hillsdale, NJ, 1983), see Chap. 11 on Higher Order Factors for discussion related to reliability and validity.
  64. A. Rodriguez, S. P. Reise, and M. G. Haviland, Evaluating bifactor models: Calculating and interpreting statistical indices, Psychol. Methods 21, 137 (2016), see the Appendix for practical guidance on calculating omega in r.
  65. D. B. Flora, Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using r to obtain better reliability estimates, Adv. Methods Pract. Psychol. Sci. 3, 484 (2020).
  66. D. T. Campbell and D. W. Fiske, Convergent and discriminant validation by the multitrait-multimethod matrix, Psychol. Bull. 56, 81 (1959).
  67. R. P. Bagozzi, Y. Yi, and L. W. Phillips, Assessing construct validity in organizational research, Adm. Sci. Q. 36, 421 (1991).
  68. Y. Rosseel, lavaan: An r package for structural equation modeling, J. Stat. Softw. 48, 1 (2012).
  69. C. Fornell and D. F. Larcker, Evaluating structural equation models with unobservable variables and measurement error, J. Market. Res. 18, 39 (1981).
  70. G. W. Cheung, H. D. Cooper-Thomas, R. S. Lau, and L. C. Wang, Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations, Asia Pac. J. Manage. 41, 745 (2024).
  71. H. F. Kaiser and J. Rice, Little Jiffy, Mark IV, Educ. Psychol. Meas. 34, 111 (1974).
  72. M. S. Bartlett, Properties of sufficiency and statistical tests, Proc. R. Soc. A 160, 268 (1937).
  73. R. B. Kline, Principles and Practice of Structural Equation Modeling (The Guilford Press, New York, 2016).
  74. D. L. Bandalos and S. J. Finney, Factor analysis: Exploratory and confirmatory, in The Reviewer’s Guide to Quantitative Methods in the Social Sciences (Routledge, New York, 2018), pp. 98–122.
  75. L. R. Fabrigar and D. T. Wegener, Exploratory Factor Analysis (Oxford University Press, New York, 2012).
  76. P. J. Curran, S. G. West, and J. F. Finch, The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis, Psychol. Methods 1, 16 (1996).
  77. A. E. Hendrickson and P. O. White, Promax: A quick method for rotation to oblique simple structure, Br. J. Stat. Psychol. 17, 65 (1964).
  78. R. C. MacCallum, K. F. Widaman, S. Zhang, and S. Hong, Sample size in factor analysis, Psychol. Methods 4, 84 (1999).
  79. M. Verostek, D. Sachmpazidi, J. Petrella, and C. Turpen, Assessing the culture around systemic change in physics programs: A pilot study from 33 programs in the United States, presented at PER Conf. 2025, Washington, DC, 10.1119/perc.2025.pr.Verostek.
  80. C. O. Fritz, P. E. Morris, and J. J. Richler, Effect size estimates: Current use, calculations, and interpretation, J. Exp. Psychol. 141, 2 (2012).
  81. J. Cohen, Statistical Power Analysis for the Behavioral Sciences (Routledge, New York, 1988).
  82. J. Cohen, A power primer, Psychol. Bull. 112, 155 (1992).
  83. S. P. Reise, W. E. Bonifay, and M. G. Haviland, Scoring and modeling psychological measures in the presence of multidimensionality, J. Pers. Assess. 95, 129 (2013).
  84. D. M. Dueber and M. D. Toland, A bifactor approach to subscore assessment, Psychol. Methods, 28, 222 (2023).
  85. D. D. Suhr, Exploratory or confirmatory factor analysis?, in Proceedings of the 31st Annual SAS Users Group International Conference, Software Intelligence Corporation, Spring Valley, CA (SAS Institute, Inc., Cary, NC, 2006).
  86. A. G. Yong, S. Pearce, A beginner’s guide to factor analysis: Focusing on exploratory factor analysis, Tutor. Quant. Methods Psychol. 9, 79 (2013).
  87. R. A. Kass and H. E. A. Tinsley, Factor analysis, J. Leis. Res. 11, 120 (1979).
  88. W. A. Arrindell and J. Van der Ende, An empirical test of the utility of the observations-to-variables ratio in factor and components analysis, Appl. Psychol. Meas. 9, 165 (1985).
  89. E. Guadagnoli and W. F. Velicer, Relation of sample size to the stability of component patterns, Psychol. Bull. 103, 265 (1988).
  90. D. R. Johnson and J. C. Creech, Ordinal measures in multiple indicator models: A simulation study of categorization error, Am. Sociol. Rev. 48, 398 (1983).
  91. G. Norman, Likert scales, levels of measurement and the “laws” of statistics, Adv. Health Sci. Educ. 15, 625 (2010).
  92. I. T. Jolliffe, Principal Component Analysis (Springer, New York, 2002).
  93. S. P. Reise, N. G. Waller, and A. L. Comrey, Factor analysis and scale revision, Psychol. Assess. 12, 287 (2000).
  94. K. V. Mardia, Measures of multivariate skewness and kurtosis with applications, Biometrika 57, 519 (1970).
  95. J. Schmid and J. M. Leiman, The development of hierarchical factor solutions, Psychometrika 22, 53 (1957).
  96. A. Rodriguez, S. P. Reise, and M. G. Haviland, Applying bifactor statistical indices in the evaluation of psychological measures, J. Pers. Assess. 98, 223 (2016).
  97. R. E. Zinbarg, I. Yovel, W. Revelle, and R. P. McDonald, Estimating generalizability to a latent variable common to all of a scale’s indicators: A comparison of estimators for ωh, Appl. Psychol. Meas. 30, 121 (2006).
  98. S. P. Reise, The rediscovery of bifactor measurement models, Multivar. Behav. Res. 47, 667 (2012).

Outline

Information

Phys. Rev. Phys. Educ. Res. 22, 010107– Published 23 January, 2026
  • Received 25 September 2025
  • Accepted 22 December 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation