- Physical Review Journals
All JournalsPhysics Magazine

Physical Review E

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Bias-inducing geometries: An exactly solvable data model with fairness implications

Stefano Sarao Mannelli*

Federica Gerace†

Negar Rostamzadeh‡

Luca Saglietti§

  • Google Responsible AI, Montreal, Canada

  • *Contact author: s.saraomannelli@chalmers.se
  • †Contact author: federica.gerace@unibo.it
  • ‡Contact author: nrostamzadeh@google.com
  • §Contact author: luca.saglietti@unibocconi.it
PDF

Phys. Rev. E 112, 025304 – Published 7 August, 2025

DOI: https://doi.org/10.1103/nlfl-35t6

Abstract

Machine learning (ML) may be oblivious to human bias but it is not immune to its perpetuation. Marginalization and iniquitous group representation are often traceable in the very data used for training and may be reflected or even enhanced by the learning models. In the present work, we aim to clarify the role played by data geometry in the emergence of ML bias. We introduce an exactly solvable high-dimensional model of data imbalance, where parametric control over the many bias-inducing factors allows for an extensive exploration of the bias inheritance mechanism. Through the tools of statistical physics, we analytically characterize the typical properties of learning models trained in this synthetic framework and obtain exact predictions for the observables that are commonly employed for fairness assessment. Simplifying the nature of the problem to its minimal components, we can retrace and unpack typical unfairness behavior observed on real-world datasets. Finally, we focus on the effectiveness of bias mitigation strategies, first by considering a loss-reweighing scheme that allows for an implicit minimization of different unfairness metrics and a quantification of the incompatibilities between existing fairness criteria. Then, we propose a mitigation strategy based on a matched inference setting that entails the introduction of coupled learning models. Our theoretical analysis of this approach shows that the coupled strategy can strike superior fairness-accuracy trade-offs.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (63)

  1. J. Buolamwini and T. Gebru, Gender shades: Intersectional accuracy disparities in commercial gender classification, in Proceedings of the Conference on Fairness, Accountability, and Transparency (PMLR, 2018), pp. 77–91.
  2. L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh et al., Ethical and social risks of harm from language models, arXiv:2112.04359.
  3. R. Benjamin, Race after Technology: Abolitionist Tools for the New Jim Code (Polity, Cambridge, UK, 2019).
  4. S. U. Noble, Algorithms of Oppression (New York University Press, New York, NY, 2018).
  5. V. Eubanks, Automating Inequality: How High-tech Tools Profile, Police, and Punish the Poor (St. Martin's Press, New York, NY, 2018).
  6. M. Broussard, Artificial Unintelligence: How Computers Misunderstand the World (MIT Press, Cambridge, MA, 2018).
  7. H. Suresh and J. V. Guttag, A framework for understanding sources of harm throughout the machine learning life cycle, in Proceedings of the ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO'21), Virtual Event, October 5–9, 17:1 (ACM, New York, NY, 2021).
  8. C. C. Perez, Invisible Women: Data Bias in a World Designed for Men (Abrams, New York, NY, 2019).
  9. S. Kotsiantis, D. Kanellopoulos, P. Pintelas et al., Handling imbalanced datasets A review, GESTS Int. Trans. Comput. Sci. Eng. 30, 25 (2006).
  10. L. Wang, M. Han, X. Li, N. Zhang, and H. Cheng, Review of classification methods on unbalanced data sets, IEEE Access 9, 64606 (2021).
  11. X.-w. Chen and M. Wasikowski, FAST: A ROC-based feature selection metric for small samples and imbalanced data classification problems, in Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM, New York, NY, 2008), pp. 124–132.
  12. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, SMOTE: synthetic minority over-sampling technique, J. Artif. Intell. Res. 16, 321 (2002).
  13. X.-Y. Liu, J. Wu, and Z.-H. Zhou, Exploratory undersampling for class-imbalance learning, IEEE Trans. Syst. Man Cybern. Part B Cybern. 39, 539 (2008).
  14. Z. Wang, K. Qinami, I. C. Karakozis, K. Genova, P. Nair, K. Hata, and O. Russakovsky, Towards fairness in visual recognition: Effective strategies for bias mitigation, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, NJ, 2020), pp. 8919–8928.
  15. B. Y. Idrissi, M. Arjovsky, M. Pezeshki, and D. Lopez-Paz, Simple data balancing achieves competitive worst-group-accuracy, in Proceedings of the Conference on Causal Learning and Reasoning (PMLR, 2022), pp. 336–351.
  16. R. N. Hughes, Sex does matter: Comments on the prevalence of male-only investigations of drug effects on rodent behaviour, Behav. Pharmacol. 18, 583 (2007).
  17. R. N. Hughes, Sex still matters: Has the prevalence of male-only studies of drug effects on rodent behaviour changed during the past decade? Behav. Pharmacol. 30, 95 (2019).
  18. L. Zdeborová and F. Krzakala, Statistical physics of inference: Thresholds and algorithms, Adv. Phys. 65, 453 (2016).
  19. Z. Liu, P. Luo, X. Wang, and X. Tang, Deep learning face attributes in the wild, in Proceedings of International Conference on Computer Vision (ICCV) (IEEE Computer Society, Piscataway, NJ, USA, 2015).
  20. F. Mignacco, F. Krzakala, Y. Lu, P. Urbani, and L. Zdeborova, The role of regularization in classification of high-dimensional noisy gaussian mixture, in Proceedings of the International Conference on Machine Learning (PMLR, 2020), pp. 6874–6883.
  21. L. Saglietti and L. Zdeborová, Solvable model for inheriting the regularization through knowledge distillation, in Proceedings of the Conference on Mathematical and Scientific Machine Learning (PMLR, 2022), pp. 809–846.
  22. M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian, Certifying and removing disparate impact, in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Association for Computing Machinery (ACM), New York, NY, USA, 2015), pp. 259–268.
  23. U. E. E. O. Commission et al., Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection procedures (U.S. Equal Employment Opportunity Commission, Washington, DC, 1979).
  24. P. Charbonneau, E. Marinari, M. Mézard, G. Parisi, F. Ricci-Tersenghi, G. Sicuro, and F. Zamponi, Spin Glass Theory and Far Beyond (World Scientific, Singapore, 2023).
  25. M. Mézard, G. Parisi, and M. A. Virasoro, Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications (World Scientific, Singapore, 1987), Vol. 9.
  26. A. Engel and C. Van den Broeck, Statistical Mechanics of Learning (Cambridge University Press, Cambridge, UK, 2001).
  27. C. Thrampoulidis, S. Oymak, and B. Hassibi, Regularized linear regression: A precise analysis of the estimation error, in Proceedings of the Conference on Learning Theory (PMLR, 2015), pp. 1683–1709.
  28. B. Loureiro, C. Gerbelot, H. Cui, S. Goldt, F. Krzakala, M. Mezard, and L. Zdeborová, Learning curves of generic features maps for realistic datasets with a teacher-student model, Adv. Neural Info. Process. Syst. 34, 18137 (2021).
  29. F. Gerace, L. Saglietti, S. S. Mannelli, A. Saxe, and L. Zdeborová, Probing transfer learning with a model of synthetic correlated datasets, Mach. Learn.: Sci. Technol. 3, 015030 (2022).
  30. T. Speicher, H. Heidari, N. Grgic-Hlaca, K. P. Gummadi, A. Singla, A. Weller, and M. B. Zafar, A unified approach to quantifying algorithmic unfairness: Measuring individual & group unfairness via inequality indices, in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Association for Computing Machinery (ACM), New York, NY, USA, 2018), pp. 2239–2248.
  31. A. Castelnovo, R. Crupi, G. Greco, D. Regoli, I. G. Penco, and A. C. Cosentini, A clarification of the nuances in the fairness metrics landscape, Sci. Rep. 12, 4209 (2022).
  32. A. Agarwal, A. Beygelzimer, M. Dudík, J. Langford, and H. Wallach, A reductions approach to fair classification, in Proceedings of the International Conference on Machine Learning (PMLR, 2018), pp. 60–69.
  33. A. Agarwal, M. Dudík, and Z. S. Wu, Fair regression: Quantitative definitions and reduction-based algorithms, in Proceedings of the International Conference on Machine Learning (PMLR, 2019), pp. 120–129.
  34. L. E. Celis, L. Huang, V. Keswani, and N. K. Vishnoi, Classification with fairness constraints: A meta-algorithm with provable guarantees, in Proceedings of the Conference on Fairness, Accountability, and Transparency (2019), pp. 319–328.
  35. J. Kleinberg, S. Mullainathan, and M. Raghavan, Inherent trade-offs in the fair determination of risk scores, in Proceedings of the 8th Innovations in Theoretical Computer Science Conference, ITCS 2017, Berkeley, CA, USA (Schloss Dagstuhl Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 2017), pp. 3315–3323.
  36. S. Corbett-Davies and S. Goel, The measure and mismeasure of fairness: A critical review of fair machine learning, J. Mach. Learn. Res. 24, 3121 (2023).
  37. S. Barocas, M. Hardt, and A. Narayanan, Fairness and Machine Learning (2019).
  38. F. Kamiran and T. Calders, Data preprocessing techniques for classification without discrimination, Knowl. Inf. Syst 33, 1 (2012).
  39. D. Plecko and N. Meinshausen, Fair data adaptation with quantile preservation, J. Mach. Learn. Res. 21, 1 (2020).
  40. K. Lum and J. Johndrow, A statistical framework for fair predictive algorithms, arXiv:1610.08077.
  41. A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, Progressive neural networks, arXiv:1606.04671.
  42. C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, Fairness through awareness, in Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (Association for Computing Machinery (ACM), New York, NY, USA, 2012), pp. 214–226.
  43. S. Corbett-Davies, E. Pierson, A. Feller, S. Goel, and A. Huq, Algorithmic decision making and the cost of fairness, in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Association for Computing Machinery (ACM), New York, NY, USA, 2017), pp. 797–806.
  44. M. Hardt, E. Price, and N. Srebro, Equality of opportunity in supervised learning, Adv. Neural Info. Process. Syst. 29, 3315 (2016).
  45. M. B. Zafar, I. Valera, M. G. Rogriguez, and K. P. Gummadi, Fairness constraints: Mechanisms for fair classification, in Proceedings of the Conference on  Artificial Intelligence and Statistics (PMLR, 2017), pp. 962–970.
  46. A. Chouldechova, Fair prediction with disparate impact: A study of bias in recidivism prediction instruments, Big Data 5, 153 (2017).
  47. F. Zenke, B. Poole, and S. Ganguli, Continual learning through synaptic intelligence, in Proceedings of the International Conference on Machine Learning (PMLR, 2017), pp. 3987–3995.
  48. L. Saglietti, S. S. Mannelli, and A. Saxe, An analytical theory of curriculum learning in teacher-student networks, J. Stat. Mech: Theory Exp. (2022) 114014.
  49. T. Calders and S. Verwer, Three naive Bayes approaches for discrimination-free classification, Data Min. Knowl. Discovery 21, 277 (2010).
  50. R. H. Myers and R. H. Myers, Classical and Modern Regression with Applications (Duxbury Press, Belmont, CA, 1990), Vol. 2.
  51. S. Dutta, D. Wei, H. Yueksel, P.-Y. Chen, S. Liu, and K. Varshney, Is there a trade-off between fairness and accuracy? A perspective using mismatched hypothesis testing, in Proceedings of the  International Conference on Machine Learning (PMLR, 2020), pp. 2803–2813.
  52. The illustrated checkpoints are used only to show the similarity of behavior in synthetic data and realistic data (CelebA), and not used or recommended to use in any face recognition systems or scenarios.
  53. S. Goldt, M. Mézard, F. Krzakala, and L. Zdeborová, Modeling the influence of data structure on learning in neural networks: The hidden manifold model, Phys. Rev. X 10, 041044 (2020).
  54. U. Adomaityte, G. Sicuro, and P. Vivo, Classification of heavy-tailed features in high dimensions a superstatistical approach, Adv. Neural Info. Process. Syst. 36, 43880 (2024).
  55. E. Cornacchia, F. Mignacco, R. Veiga, C. Gerbelot, B. Loureiro, and L. Zdeborová, Learning curves for the multi-class teacher–student perceptron, Mach. Learn.: Sci. Technol. 4, 015019 (2023).
  56. S. Sagawa, A. Raghunathan, P. W. Koh, and P. Liang, An investigation of why overparameterization exacerbates spurious correlations, in Proceedings of the International Conference on Machine Learning (PMLR, 2020), pp. 8346–8356.
  57. S. J. Bell and L. Sagun, Simplicity bias leads to amplified performance disparities, in Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery (ACM), New York, NY, USA, 2023), pp. 355–369.
  58. L. A. Blewett, J. A. Rivera Drew, R. Griffin, N. Del Ponte, and P. Convey, IPUMS Health Surveys: Medical Expenditure Panel Survey, version 2.1 (IPUMS, Minneapolis, MN, 2021).
  59. F. Chollet, Xception: Deep learning with depthwise separable convolutions, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, Piscataway, NJ, USA, 2017), pp. 1251–1258.
  60. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, NJ, 2009), pp. 248–255.
  61. To be mindful on the Ethical Considerations of using the CelebA datast, we do not use protected attributes like binary genders and age.
  62. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg et al., TensorFlow: Large-scale Machine Learning on Heterogeneous Systems (tensorflow.org, 2015).
  63. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel et al., Scikit-learn machine learning in Python, J. Mach. Learn. Res. 12, 2825 (2011).

Outline

Information

Phys. Rev. E 112, 025304– Published 7 August, 2025
  • Received 26 March 2024
  • Revised 8 January 2025
  • Accepted 8 July 2025
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI. Funded by Bibsam.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation