Comparing Retrospective and Pre-Post Survey Measurement of First-Year African Engineering Students’ Skill and Perception Changes through a PBL Course
DOI:
https://doi.org/10.54337/irspbl-11047Keywords:
Retrospective survey, First-year, Self-assessment bias, Dunning-Kruger effect, Memory/recall biasAbstract
For many first-year engineering students at Ashesi University in Ghana, the project-based learning (PBL) approach they experience in Introduction to Engineering differs from the rote-based pedagogies they have previously experienced. The authors’ previous efforts to understand this pedagogy's impact on the students have relied on pre-post survey tools. For some responses, higher-than-expected scores on the pre-survey have been noticed. This study aimed to understand whether a different methodology, a retrospective pre-post survey, would yield similar results and what reasons may underlie any differences.
A mixed-method approach was deployed with students in the 2024 cohort of the course. They completed two surveys – one at the beginning (pre-survey) and one at the end of the course (retrospective pre-post survey). Five skill/perception categories were deployed with Likert-scale questions. T. tests revealed no statistically significant difference in the traditional pre-post analysis, while in the retrospective analysis, all five categories had statistically significant increases with small to medium effect sizes. Interviews with select respondents revealed self-assessment bias, as they tended to overestimate their skills at the beginning of the course. These results suggest that the retrospective approach may be better suited for studies on this target group.
References
Ampomah, C. O. A., & Beem, H. (2024). Full Paper: Continued exploration of the relevance of self-efficacy, self-determination, and agency in describing the first-year African engineering student’s experience. 15th Annual First-Year Engineering Experience Conference (FYEE), Boston, Massachusetts. https://doi.org/10.18260/1-2--48592
Baer, R. A., Ballenger, J., Berry, D. T. R., & Wetter, M. W. (1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68(1), 139-151. https://doi:10.1207/s15327752jpa6801_11
Beem, H. R. (2021a). Exploring the role of project-based learning in building self-efficacy in first-year African engineering students. In 2021, ASEE Virtual Annual Conference Content Access.
Beem, H.R. (2021b). The role of machine shop sessions in building confidence amongst first- year engineering students in Ghana. IEEE Global Engineering Education Conference, EDUCON, 2021-April(April), 1469–1474. https://doi.org/10.1109/EDUCON46332.2021.9454012
Beem, H. (2022). Exploring the role of the physical environment in building self-efficacy in first-year African engineering students. ASEE Annual Conference and Exposition, Conference Proceedings.
Beem, H., Ampomah, C. O. A., Takyi, J. P. K., & Adomdza, G. (2023a). A preliminary exploration of the relevance of self-efficacy, self-determination, and agency in describing the first-year African engineering students’ experience. In 2023 ASEE Annual Conference & Exposition.
Beem, H. R., Ampomah, C., Takyi, J., & Adomdza, G. K. (2023b). Development of an Online Project-Based Learning Design Course for African First-Year Students and Its Impact on Self- Efficacy Levels. IEEE Transactions on Education, 66(5), 410–420. https://doi.org/10.1109/TE.2023.3312968
Berry, D. T. R., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4(3), 340-345. https://doi:10.1037/1040-3590.4.3.340
Christensen, T. C., Wood, J. V., & Barrett, L. F. (2003). Remembering everyday experience through the prism of self-esteem. Personality and Social Psychology Bulletin, 29(1), 51–62. https://doi.org/10.1177/0146 167202238371
Cumming, J., Woodcock, C., Cooley, S. J., Holland, M. J. G., & Burns, V. E. (2015). Development and validation of the groupwork skills questionnaire (GSQ) for higher education. Assessment and Evaluation in Higher Education, 40(7), 988–1001. https://doi.org/10.1080/02602938.2014.957642
Davis, G. C., Baral, R., Strayer, T., & Serrano, E. L. (2018). Using pre-and post-survey instruments in interventions: Determining the random response benchmark and its implications for measuring effectiveness. Public Health Nutrition, 21(6), 1043-1047.
Drennan J & Hyde A (2008) Controlling response shift bias: The use of the retrospective pre- test design in the evaluation of a master's programme. Assessment and Evaluation in Higher Education,33 (6):699-709
Dunning, D. (2011). The Dunning–Kruger effect: On being ignorant of one's own ignorance. In Advances in experimental social psychology (Vol. 44, pp. 247-296). Academic Press.
Evans, C. M. (2022). What About Pre-and Post-Motivation Survey Scores? How Might They Support My Conclusions? Reading Motivation: A Guide to Understanding and Supporting Children's Willingness to Read, 91.
Ferguson, A., Gilkey, B., Kern, D., & Jasmine, D. (2012). An education initiative concerning lead in an Arkansas community: results from pre-and post-surveys. Journal of Community Health, 37, 599-609.
Forsyth, D. R.(2008). Self-Serving Bias. International Encyclopedia of the Social Sciences. Edited by William A. Darity. 2nd ed. Vol. 7. Detroit: Macmillan Reference USA.
Gok, T. (2012). Development of Problem Solving Confidence Questionnaire: Study of validation and reliability. Am. J. Phys. Educ, 6(1). http://www.lajpe.org
Grimes, A., Medway, D., Foos, A., & Goatman, A. (2015). Impact bias in student evaluations of higher education. Studies in Higher Education, 42(6), 945–962. https://doi.org/10.1080/03075079.2015.1071345
Hill, L. G., & Betz, D. L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26(4), 501–517.
Howard, G.S. (1980). Response-shift bias: A problem in evaluating interventions with pre/post self-reports. Evaluation Review, 4, 93-106.
Howard, G. S., Ralph, K. F., Gulanick, N. A., Maxwell, S. E., & Gerber, S. K. (1979). Internal validity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3(1), 1–23.
Hufford, J. R. (2010). What are they learning? Pre-and post-assessment surveys for LIBR 1100, Introduction to library research. College & Research Libraries, 71(2), 139-158.
Johnson, D. D. P., & Fowler, J. H. (2011). The evolution of overconfidence. Nature, 477(7364), 317–320. https://doi.org/10.1038/nature10384
Lamb, T. (2005). The retrospective pretest: An imperfect but useful tool. The evaluation exchange, 11(2), 18.
Kowalski, M. J. (2023). Measuring changes with traditional and retrospective pre-posttest self-report surveys for a brief intervention program. Evaluation and Program Planning, 99, 102323.
Lay, J. C., Gerstorf, D., Scott, S. B., Pauly, T., & Hoppmann, C. A. (2017). Neuroticism and extraversion magnify discrepancies between retrospective and concurrent affect reports. Journal of Personality, 85(6), 817–829. https://doi.org/10.1111/jopy.12290
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437-455.
Mill, A., Realo, A., & Allik, J. (2015). Retrospective ratings of emotions: the effects of age, daily tiredness, and personality. Frontiers in Psychology, 6, 2020. https://doi.org/10.3389/fpsyg.2015.02020
Moore, D., & Tananis, C. A. (2009). Measuring change in a short-term educational program using a retrospective pretest design. American Journal of Evaluation, 30(2), 189-202.
Nimon, K. (2014). Explaining differences between retrospective and traditional pretest self-assessments: competing theories and empirical evidence. International Journal of Research & Method in Education, 37(3), 256–269. https://doi.org/10.1080/ 1743727X.2013.820644
Parkinson, B., Briner, R. B., Reynolds, S., & Totterdell, P. (1995). Time frames for mood: Relations between momentary and generalized ratings of affect. Personality and Social Psychology Bulletin, 21(4), 331–339. https://doi.org/10.1177/0146167295214003
Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). Academic Press. https://doi.org/10.1016/B978-0-12-590241- 0.50006-X
Porter, S. R., Whitcomb, M. E., & Weitzer, W. H. (2004). Multiple surveys of students and survey fatigue. New Directions for Institutional Research, 121, 63–73.
Robinson, O. C. (2014). Sampling in interview-based qualitative research: A theoretical and practical guide. Qualitative research in psychology, 11(1), 25-41.
Saldaña, J. (2009). The coding manual for qualitative researchers. Sage Publications Ltd
Schwartz, C. E., & Sprangers, M. A. (2010). Guidelines for improving the stringency of response shift research using the thentest. Quality of Life Research, 19, 455-464.
Wetzel, E., Böhnke, J. R., & Brown, A. (2016). Response biases. In F. T. L. Leong, D. Bartram, F. M. Cheung, K. F. Geisinger, & D. Iliescu (Eds.), The ITC international handbook of testing and assessment (pp. 349–363). Oxford University Press. https://doi.org/10.1093/med:psych/9780199356942.003.0024
Wirtz, D., Kruger, J., Napa Scollon, C., & Diener, E. (2003). What to do on spring break? The role of predicted, on-line, and remembered experience in future choice. Psychological Science, 14(5), 520–524. https://doi.org/10.1111/1467-9280.03455
Zurbriggen, C. L. A., Jendryczko, D., & Nussbeck, F. W. (2021). Rosy or blue? Change in recall bias of students’ affective experiences during early adolescence. Emotion, 21(8), 1637– 1649. https://doi.org/10.1037/emo0001031
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.
