rutaBAGA: A Visualization Approach for Bias Awareness in University Admissions
Article Sidebar
Main Article Content
Abstract
University admissions is a complex decision making process where cognitive and implicit biases, may impact the way reviewers individually and collectively make decisions. Education-based methods like training courses often have limited impact due to the subconscious nature of these biases. This paper introduces a visualization system, rutaBAGA, that promotes heightened awareness of implicit biases through real-time system interactions. The system enables reviewers to scrutinize their own processes to ensure fair and consistent review procedures. We present the results of a controlled study that shows (i) implicit racial bias correlates to observable differences in university application review behaviors and decisions and (ii) our system can affect individuals' review processes. Additionally, we present a case study where rutaBAGA was used in the 2022-2023 Ph.D. admissions cycle in the Computer Science department at a private university demonstrating rutaBAGA's potential to iteratively transform university application review processes to ensure adherence to fair procedural goals.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
References
Slate. Retrieved November 30, 2023 from https://slate.org/
Chadia Abras, Diane Maloney-Krichmar, and Jenny Preece. 2004. User-Centered Design. Bainbridge, W. Encyclopedia of Human-Computer Interaction. Thousand Oaks: Sage Publications 37, 4 (2004), 445–456.
Civil Rights Act. 1964. Civil Rights Act of 1964. Title VII, Equal Employment Opportunities (1964).
Marianne Bertrand and Sendhil Mullainathan. 2004. Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. American economic review 94, 4 (2004), 991–1013. https://doi.org/10.1257/0002828042002561
Katerina Bezrukova, Karen A. Jehn, and Chester S. Spell. 2012. Reviewing Diversity Training: Where We Have Been and Where We Should Go. Academy of Management Learning amp; Education 11, 2 (June 2012), 207–227. https://doi.org/10.5465/amle.2008.0090
David Borland, Jonathan Zhang, Smiti Kaul, and David Gotz. 2020. Selection-Bias-Corrected Visualization via Dynamic Reweighting. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 1481–1491. https://doi.org/10.1109/tvcg.2020.3030455
John Brooke. 2013. SUS: A Retrospective. Journal of usability studies 8, 2 (2013), 29–40.
Hilary Burns and Neena Hagen. 2024. ‘It is heartbreaking’: Black first-year enrollment plunges at top colleges, post-affirmative action. The Boston Globe (25 September 2024). https://www.bostonglobe.com/2024/09/25/metro/affirmative-action-colleges-black-enrollment/
Quinn Capers IV, Daniel Clinchot, Leon McDougle, and Anthony G Greenwald. 2017. Implicit Racial Bias in Medical School Admissions. Academic Medicine 92, 3 (2017), 365–369. https://doi.org/10.1097/acm.0000000000001388
Mengyu Chen, Yijun Liu, and Emily Wall. 2024. Unmasking Dunning-Kruger Effect in Visual Reasoning & Judgment. IEEE Transactions on Visualization and Computer Graphics (2024). https://doi.org/10.1109/TVCG.2024.3456326
Mengyu Chen and Emily Wall. 2022. Perception of Skill in Visual Problem Solving: An Analysis of Interactive Behaviors, Personality Traits, and the Dunning-Kruger Effect. (2022).
Isaac Cho, Ryan Wesslen, Alireza Karduni, Sashank Santhanam, Samira Shaikh, and Wenwen Dou. 2017. The Anchoring Effect in Decision-Making with Visual Analytics. In 2017 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 116–126. https://doi.org/10.1109/vast.2017. 8585665
Arthur L Coleman and Jamie Lewis Keith. 2018. Understanding Holistic Review in Higher Education Admissions. New York: College Board (2018).
Evanthia Dimara, Gilles Bailly, Anastasia Bezerianos, and Steven Franconeri. 2018. Mitigating the Attraction Effect with Visualizations. IEEE transactions on visualization and computer graphics 25, 1 (2018), 850–860. https://doi.org/10.1109/tvcg.2018.2865233
Evanthia Dimara, Anastasia Bezerianos, and Pierre Dragicevic. 2016. The Attraction Effect in Information Visualization. IEEE transactions on visualization and computer graphics 23, 1 (2016), 471–480. https://doi.org/10.1109/tvcg.2016.2598594
Evanthia Dimara, Steven Franconeri, Catherine Plaisant, Anastasia Bezerianos, and Pierre Dragicevic. 2018. A Task-Based Taxonomy of Cognitive Biases for Information Visualization. IEEE transactions on visualization and computer graphics 26, 2 (2018), 1413–1432. https://doi.org/10.1109/tvcg. 2018.2872577
Frank Dobbin and Alexandra Kalev. 2018. Why Doesn’t Diversity Training Work? The Challenge for Industry and Academia. Anthropology Now 10,2 (May 2018), 48–55. https://doi.org/10.1080/19428200.2018.1493182
Mi Feng, Evan Peck, and Lane Harrison. 2018. Patterns and Pace: Quantifying Diverse Exploration Behavior with Visualizations on the Web. IEEEtransactions on visualization and computer graphics 25, 1 (2018), 501–511. https://doi.org/10.1109/tvcg.2018.2865117
Robert Fullinwider. 2018. Affirmative Action. In The Stanford Encyclopedia of Philosophy (Summer 2018 ed.), Edward N. Zalta (Ed.). MetaphysicsResearch Lab, Stanford University.
Adrian Furnham. 1986. Response Bias, Social Desirability and Dissimulation. Personality and individual differences 7, 3 (1986), 385–400. https://doi.org/10.1016/0191-8869(86)90014-0
David Gotz, Shun Sun, and Nan Cao. 2016. Adaptive Contextualization: Combating Bias During High-Dimensional Visualization and Data Selection.In Proceedings of the 21st International Conference on Intelligent User Interfaces. 85–95. https://doi.org/10.1145/2856767.2856779
Anthony G Greenwald and Mahzarin R Banaji. 1995. Implicit Social Cognition: Attitudes, Self-Esteem, and Stereotypes. Psychological review 102, 1(1995), 4. https://doi.org/10.1037//0033-295x.102.1.4
Anthony G Greenwald and Linda Hamilton Krieger. 2006. Implicit Bias: Scientific Foundations. California law review 94, 4 (2006), 945–967.https://doi.org/10.2307/20439056
Anthony G Greenwald, Debbie E McGhee, and Jordan LK Schwartz. 1998. Measuring Individual Differences in Implicit Cognition: The ImplicitAssociation Test. Journal of personality and social psychology 74, 6 (1998), 1464. https://doi.org/10.1037/0022-3514.74.6.1464
Sunwoo Ha, Shayan Monadjemi, Roman Garnett, and Alvitta Ottley. 2022. A Unified Comparison of User Modeling Techniques for Predicting DataInteraction and Detecting Exploration Bias. IEEE Transactions on Visualization and Computer Graphics (2022). https://doi.org/10.1109/tvcg.2022.3209476
Anemona Hartocollis. 2024. Harvard’s Black Student Enrollment Dips After Affirmative Action Ends. The New York Times (11 September 2024).https://www.nytimes.com/2024/09/11/us/harvard-affirmative-action-diversity-admissions.html
Eric Horvitz. 1999. Principles of Mixed-Initiative User Interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems.159–166. https://doi.org/10.1145/302979.303030
Project Implicit. 2023. Implicit Association Test. Retrieved September 13, 2023 from https://implicit.harvard.edu/implicit/selectatest.html
Sarah M. Jackson, Amy L. Hillard, and Tamera R. Schneider. 2014. Using Implicit Bias Training to Improve Attitudes toward Women in STEM.Social Psychology of Education 17, 3 (May 2014), 419–438. https://doi.org/10.1007/s11218-014-9259-5
Po-Ming Law and Rahul C Basole. 2018. Designing Breadth-Oriented Data Exploration for Mitigating Cognitive Biases. In Cognitive Biases inVisualizations. Springer, 149–159. https://doi.org/10.1007/978-3-319-95831-6_11
Sukwon Lee, Sung-Hee Kim, and Bum Chul Kwon. 2016. VLAT: Development of a Visualization Literacy Assessment Test. IEEE transactions onvisualization and computer graphics 23, 1 (2016), 551–560. https://doi.org/10.1109/tvcg.2016.2598920
Qianyu Liu, Haoran Jiang, Zihao Pan, Qiushi Han, Zhenhui Peng, and Quan Li. 2024. BiasEye: A Bias-Aware Real-time Interactive MaterialScreening System for Impartial Candidate Assessment. In Proceedings of the 29th International Conference on Intelligent User Interfaces. 325–343.https://doi.org/10.1145/3640543.3645166
Ronald A Metoyer, Tee Chuanromanee, Gina M Girgis, Qiyu Zhi, and Eleanor C Kinyon. 2020. Supporting Storytelling With Evidence inHolistic Review Processes: A Participatory Design Approach. Proceedings of the ACM on Human-Computer Interaction 4, CSCW1 (2020), 1–24.https://doi.org/10.1145/3392870
Matthew B Miles, A Michael Huberman, and Johnny Saldaña. 2018. Qualitative Data Analysis: A Methods Sourcebook. Sage publications.
Corinne A Moss-Racusin, John F Dovidio, Victoria L Brescoll, Mark J Graham, and Jo Handelsman. 2012. Science Faculty’s Subtle Gender BiasesFavor Male Students. Proceedings of the national academy of sciences 109, 41 (2012), 16474–16479. https://doi.org/10.1073/pnas.1211286109
Atilla Alpay Nalcaci, Dilara Girgin, Semih Balki, Fatih Talay, Hasan Alp Boz, and Selim Balcisoy. 2019. Detection of Confirmation and DistinctionBiases in Visual Analytics Systems. In TrustVis@ EuroVis. 13–17. https://doi.org/10.2312/trvis.20191185
Arpit Narechania, Adam Coscia, Emily Wall, and Alex Endert. 2021. Lumos: Increasing Awareness of Analytic Behavior during Visual Data Analysis.IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 1009–1018. https://doi.org/10.1109/tvcg.2021.3114827
Chris North, Remco Chang, Alex Endert, Wenwen Dou, Richard May, Bill Pike, and Glenn Fink. 2011. Analytic Provenance: Process+interaction+insight. In CHI’11 Extended Abstracts on Human Factors in Computing Systems. 33–36. https://doi.org/10.1145/1979742.1979570
Supreme Court of the United States. 2023. Students for Fair Admissions v. Harvard. Retrieved September 13, 2023 from https://www.supremecourt.gov/opinions/22pdf/20-1199_hgdj.pdf
Brandy Pieper and Masha Krsmanovic. 2022. Faculty perceptions on (implicit) bias during the graduate admission review process. Studies inGraduate and Postdoctoral Education 14, 2 (Nov. 2022), 117–133. https://doi.org/10.1108/sgpe-05-2022-0040
Julie Posselt, Theresa E. Hernandez, Cynthia D. Villarreal, Aireale J. Rodgers, and Lauren N. Irwin. 2020. Evaluation and Decision Making in HigherEducation: Toward Equitable Repertoires of Faculty Practice. Springer International Publishing, 453–515. https://doi.org/10.1007/978-3-030-31365-4_8
Daniel L Reinholz, Samantha Ridgway, Poorna Talkad Sukumar, and Niral Shah. 2023. Visualizing Inequity: How STEM Educators Interpret DataVisualizations to Make Judgments about Racial Inequity. SN Social Sciences 3, 5 (2023), 76. https://doi.org/10.1007/s43545-023-00664-0
Daniel L Reinholz and Niral Shah. 2018. Equity Analytics: A Methodological Approach for Quantifying Participation Patterns in MathematicsClassroom Discourse. Journal for Research in Mathematics Education 49, 2 (2018), 140–177. https://doi.org/10.5951/jresematheduc.49.2.0140
Daniel L Reinholz, Amelia Stone-Johnstone, and Niral Shah. 2020. Walking the Walk: Using Classroom Analytics to Support Instructors to AddressImplicit Bias in Teaching. International Journal for Academic Development 25, 3 (2020), 259–272. https://doi.org/10.1080/1360144x.2019.1692211
Phoebe Sengers, Kirsten Boehner, Shay David, and Joseph’Jofish’ Kaye. 2005. Reflective Design. In Proceedings of the 4th decennial conference onCritical computing: between sense and sensibility. 49–58. https://doi.org/10.1145/1094562.1094569
Cheryl Staats, Kelly Capatosto, Lena Tenney, and Sarah Mamo. 2017. State of the Science: Implicit Bias Review 2017. Kirwan Institute for the Studyof Race and Ethnicity (2017).
Poorna Talkad Sukumar and Ronald Metoyer. 2018. A Visualization Approach to Addressing Reviewer Bias in Holistic College Admissions. InCognitive Biases in Visualizations. Springer, 161–175. https://doi.org/10.1007/978-3-319-95831-6_12
Poorna Talkad Sukumar, Ronald Metoyer, and Shuai He. 2018. Making a Pecan Pie: Understanding and Supporting The Holistic Review Process inAdmissions. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1–22. https://doi.org/10.1145/3274438
Richard H Thaler and Cass R Sunstein. 2009. Nudge: Improving Decisions about Health, Wealth, and Happiness. Penguin.
Andre Calero Valdez, Martina Ziefle, and Michael Sedlmair. 2017. Priming and Anchoring Effects in Visualization. IEEE transactions on visualizationand computer graphics 24, 1 (2017), 584–594. https://doi.org/10.1109/tvcg.2017.2744138
Emily Wall, Leslie Blaha, Celeste Paul, and Alex Endert. 2019. A Formative Study of Interactive Bias Metrics in Visual Analytics Using AnchoringBias. In IFIP Conference on Human-Computer Interaction. Springer, 555–575. https://doi.org/10.1007/978-3-030-29384-0_34
Emily Wall, Leslie M Blaha, Lyndsey Franklin, and Alex Endert. 2017. Warning, Bias May Occur: A Proposed Approach to Detecting Cognitive Biasin Interactive Visual Analytics. In 2017 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 104–115. https://doi.org/10.1109/vast.2017.8585669
Emily Wall, Leslie M Blaha, Celeste Lyn Paul, Kristin Cook, and Alex Endert. 2018. Four Perspectives on Human Bias in Visual Analytics. InCognitive biases in visualizations. Springer, 29–42. https://doi.org/10.1007/978-3-319-95831-6_3
Emily Wall, Arpit Narechania, Adam Coscia, Jamal Paden, and Alex Endert. 2021. Left, Right, and Gender: Exploring Interaction Traces to MitigateHuman Biases. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 966–975. https://doi.org/10.1109/tvcg.2021.3114862
Emily Wall, John Stasko, and Alex Endert. 2019. Toward a Design Space for Mitigating Cognitive Bias in Vis. In 2019 IEEE Visualization Conference(VIS). IEEE, 111–115. https://doi.org/10.1109/visual.2019.8933611
Monica L Wang, Alexis Gomes, Marielis Rosa, Phillipe Copeland, and Victor Jose Santana. 2023. A Systematic Review of Diversity, Equity,and Inclusion and Antiracism Training Studies: Findings and Future Directions. Translational Behavioral Medicine 14, 3 (Oct. 2023), 156–171.https://doi.org/10.1093/tbm/ibad061
Xiaomei Wang, Ann M. Bisantz, Matthew L. Bolton, Lora Cavuoto, and Varun Chandola. 2020. Cognitive Work Analysis and Visualization Designfor the Graduate Admission Decision Making Process. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 64, 1 (Dec. 2020),815–819. https://doi.org/10.1177/1071181320641189
Ryan Wesslen, Sashank Santhanam, Alireza Karduni, Isaac Cho, Samira Shaikh, and Wenwen Dou. 2019. Investigating Effects of Visual Anchors onDecision-Making about Misinformation. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 161–171. https://doi.org/10.1111/cgf.13679
Sang Eun Woo, James M. LeBreton, Melissa G. Keith, and Louis Tay. 2023. Bias, Fairness, and Validity in Graduate-School Admissions: A PsychometricPerspective. Perspectives on Psychological Science 18, 1 (2023), 3–31. https://doi.org/10.1177/17456916211055374