Keywords
Delphi method
Facilitation
Problem-based learning
Problem design
How to Cite
Abstract
Problem based learning (PBL), though recognized as beneficial to student development, faces implementation challenges in engineering education. We conducted a Delphi study with PBL and (aerospace) engineering domain experts to understand challenges around learning outcomes, problem design, facilitation, and assessment. The context for the study was an introductory aerospace engineering course, transitioning from a traditional lecture to a PBL format. We found consensus among both expert groups as it relates to ideas about learning outcomes. However, with respect to problem design and facilitation, we observed a slower and more contentious convergence, with some ideas failing to reach consensus. From this, four salient issues emerged as potential barriers to PBL implementation that supports students in important aspects of their professional development: problem framing, making relevant connections to society, a deficit view of students, and discomfort with facilitation. Implications related to these issues are discussed, with consideration given to complications stemming from the isolated, individual-course implementations of PBL commonly found in practice.
References
Adler, M., & Ziglio, E. (1996). Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health. Jessica Kingsley Publishers.
Beagon, Ú., Niall, D., & Ní Fhloinn, E. (2019). Problem-based learning: Student perceptions of its value in developing professional skills for engineering practice. European Journal of Engineering Education, 44(6), 850–865. https://doi.org/10.1080/03043797.2018.1536114
Belton, I., MacDonald, A., Wright, G., & Hamlin, I. (2019). Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process. Technological Forecasting and Social Change, 147, 72–82. https://doi.org/10.1016/j.techfore.2019.07.002
Blair, K., Miller, D. W., Darmofal, D., Young, C. P. W., & Brodeur, D. (2002). Problem Based Learning in Aerospace Engineering Education. 2002 Annual Conference, https://doi.org/10.18260/1-2--10974
Boelt, A. M., Kolmos, A., & Holgaard, J. E. (2022). Literature review of students’ perceptions of generic competence development in problem-based learning in engineering education. European Journal of Engineering Education, 47(6), 1399–1420. https://doi.org/10.1080/03043797.2022.2074819
Borrego, M., Cutler, S., Prince, M., Henderson, C., & Froyd, J. E. (2013). Fidelity of Implementation of Research-Based Instructional Strategies (RBIS) in Engineering Science Courses. Journal of Engineering Education, 102(3), 394–425. https://doi.org/10.1002/jee.20020
Capobianco, B. M., & Yu, J. H. (2014). Using the Construct of Care to Frame Engineering as a Caring Profession Toward Promoting Young Girls’ Participation. Journal of Women and Minorities in Science and Engineering, 20(1). https://doi.org/10.1615/JWomenMinorScienEng.2014006834
Chen, J., Kolmos, A., & Du, X. (2021). Forms of implementation and challenges of PBL in engineering education: A review of literature. European Journal of Engineering Education, 46(1), 90–115. https://doi.org/10.1080/03043797.2020.1718615
Clayton, M. (1997). Delphi: A Technique to Harness Expert Opinion for Critical Decision-Making Tasks in Education. Educational Psychology, 17, 373–386. https://doi.org/10.1080/0144341970170401
Creswell, J. W., & Poth, C. N. (2016). Qualitative Inquiry and Research Design: Choosing Among Five Approaches. SAGE Publications.
Cukier, K., Mayer-Schönberger, V., & Véricourt, F. de. (2022). Framers: Human Advantage in an Age of Technology and Turmoil. Penguin.
Davis, L. P., & Museus, S. D. (2019). What Is Deficit Thinking? An Analysis of Conceptualizations of Deficit Thinking and Implications for Scholarly Research. NCID Currents, 1(1). https://doi.org/10.3998/currents.17387731.0001.110
Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program planning: A guide to nominal group and delphi processes. Scott Foresman Company Glenview.
Direito, I., Pereira, A., & Duarte, A. M. de O. (2012). Engineering Undergraduates’ Perceptions of Soft Skills: Relations with Self-Efficacy and Learning Styles. Procedia - Social and Behavioral Sciences, 55, 843–851. https://doi.org/10.1016/j.sbspro.2012.09.571
Donohoe, H. M., & Needham, R. D. (2009). Moving best practice forward: Delphi characteristics, advantages, potential problems, and solutions. International Journal of Tourism Research, 11(5), 415–437. https://doi.org/10.1002/jtr.709
Dorst, K. (2019). Co-evolution and emergence in design. Design Studies, 65, 60–77. https://doi.org/10.1016/j.destud.2019.10.005
Du, X., Lundberg, A., Ayari, M. A., Naji, K. K., & Hawari, A. (2022). Examining engineering students’ perceptions of learner agency enactment in problem- and project-based learning using Q methodology. Journal of Engineering Education, 111(1), 111–136. https://doi.org/10.1002/jee.20430
Dym, C. L., Agogino, A. M., Eris, O., Frey, D. D., & Leifer, L. J. (2005). Engineering Design Thinking, Teaching, and Learning. Journal of Engineering Education, 94(1), 103–120. https://doi.org/10.1002/j.2168-9830.2005.tb00832.x
Ertmer, P. A., & Glazewski, K. D. (2019). Scaffolding in PBL Environments. Structuring and Problematizing Relevant Task Features. In M. Moallem, W. Hung, & N. Dabbagh (Eds.), The Wiley Handbook of Problem-Based Learning (pp. 321–342). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781119173243.ch14
Felder, R. M., Brent, R., & Prince, M. J. (2011). Engineering Instructional Development: Programs, Best Practices, and Recommendations. Journal of Engineering Education, 100(1), 89–122. https://doi.org/10.1002/j.2168-9830.2011.tb00005.x
Fink-Hafner, D., Dagen, T., Doušak, M., Novak, M., & Hafner-Fink, M. (2019). Delphi method: Strengths and weaknesses. Advances in Methodology and Statistics, 16(2). https://doi.org/10.51936/fcfm6982
Galand, B., Frenay, M., & Raucent, B. (2012). Effectiveness of Problem-Based Learning In Engineering Education: A Comparative Study on Three Levels of Knowledge Structure. International Journal of Engineering Education, 28(4), 939–947. http://www.ijee.ie.gate.lib.buffalo.edu/articles/Vol28-4/21_ijee2596ns.pdf
Garcia-Barriocanal, E., Nchez-Alonso, S. S., & Rodriguez-Garcia, D. (2011). Devising Instruction from Empirical Findings on Student Errors: A Case in Usability Engineering Education. International Journal of Engineering Education, 27(1), 1–7. http://www.ijee.ie.gate.lib.buffalo.edu/articles/Vol27-1/05_ijee2396ns.pdf
Gordon, T. J. (1994). The delphi method. Futures Research Methodology, 2(3), 1–30.
Green, R. A. (2014). The Delphi Technique in Educational Research. SAGE Open, 4(2), 2158244014529773. https://doi.org/10.1177/2158244014529773
Henry, H., Tawfik, A., Jonassen, D., Winholtz, R., & Khanna, S. (2012). “I Know This is Supposed to be More Like the Real World, But . . .”: Student Perceptions of a PBL Implementation in an Undergraduate Materials Science Course. Interdisciplinary Journal of Problem-Based Learning, 6(1). https://doi.org/10.7771/1541-5015.1312
Hmelo-Silver, C. E. (2004). Problem-Based Learning: What and How Do Students Learn? Educational Psychology Review, 16(3), 235–266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3
Hmelo-Silver, C. E. (2012). International Perspectives on Problem-based Learning: Contexts, Cultures, Challenges, and Adaptations. Interdisciplinary Journal of Problem-Based Learning, 6(1). https://doi.org/10.7771/1541-5015.1310
Hmelo-Silver, C. E., Bridges, S. M., & McKeown, J. M. (2019). Facilitating Problem-Based Learning. In M. Moallem, W. Hung, & N. Dabbagh (Eds.), The Wiley Handbook of Problem-Based Learning (pp. 297–319). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781119173243.ch13
Holgaard, J. E., Guerra, A., Kolmos, A., & Petersen, L. S. (2017). Getting a hold on the problem in a problem-based learning environment. The International Journal of Engineering Education, 33(3), 1070–1085. https://www-ijee-ie.gate.lib.buffalo.edu/latestissues/Vol33-3/14_ijee3445ns.pdf
Hung, W. (2006). The 3C3R Model: A Conceptual Framework for Designing Problems in PBL. Interdisciplinary Journal of Problem-Based Learning, 1(1). https://doi.org/10.7771/1541-5015.1006
Hung, W. (2016). All PBL Starts Here: The Problem. Interdisciplinary Journal of Problem-Based Learning, 10(2). https://doi.org/10.7771/1541-5015.1604
Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85. https://doi.org/10.1007/BF02300500
Jonassen, D. H. (2010). Learning to solve problems: A handbook for designing problem-solving learning environments. Routledge. https://doi.org/10.4324/9780203847527
Jonassen, D. H. (2014). Engineers as Problem Solvers. In A. Johri, & B. M. Olds (Eds.), Cambridge Handbook of Engineering Education Research (pp. 103–118). Cambridge University Press. https://doi.org/10.1017/CBO9781139013451.009
Kolmos, A., Holgaard, J. E., & Clausen, N. R. (2021). Progression of student self-assessed learning outcomes in systemic PBL. European Journal of Engineering Education, 46(1), 67–89. https://doi.org/10.1080/03043797.2020.1789070
Kolmos, A., & de Graaff, E. (2014). Problem-Based and Project-Based Learning in Engineering Education. In A. Johri & B. M. Olds (Eds.), Cambridge Handbook of Engineering Education Research (pp. 141–160). Cambridge University Press. https://doi.org/10.1017/CBO9781139013451.012
Lavi, R., & Bertel, L. B. (2024). The System Architecture-Function-Outcome Framework for Fostering and Assessing Systems Thinking in First-Year STEM Education and Its Potential Applications in Case-Based Learning. Education Sciences, 14(7), 720. https://doi.org/10.3390/educsci14070720
Long III, L. L., & Mejia, J. A. (2016). Conversations about Diversity: Institutional Barriers for Underrepresented Engineering Students. Journal of Engineering Education, 105(2), 211–218. https://doi.org/10.1002/jee.20114
Lord, S. M., & Chen, J. C. (2014). Curriculum design in the middle years. In A. Johri, & B. M. Olds (Eds.), Cambridge Handbook of Engineering Education Research (pp. 181–200). Cambridge University Press. https://doi.org/10.1017/CBO9781139013451.014
Mabley, S., Ventura-Medina, E., & Anderson, A. (2020). ‘I’m lost’ – a qualitative analysis of student teams’ strategies during their first experience in problem-based learning. European Journal of Engineering Education, 45(3), 329–348. https://doi.org/10.1080/03043797.2019.1646709
Magana, A. J. (2017). Modeling and Simulation in Engineering Education: A Learning Progression. Journal of Professional Issues in Engineering Education and Practice, 143(4), 04017008. https://doi.org/10.1061/(ASCE)EI.1943-5541.0000338
McCracken, M., & Waters, R. (1999). Why? When an otherwise successful intervention fails. Proceedings of the 4th Annual SIGCSE/SIGCUE ITiCSE Conference on Innovation and Technology in Computer Science Education, 9–12. https://doi.org/10.1145/305786.305828
McGee, E., & Bentley, L. (2017). The Equity Ethic: Black and Latinx College Students Reengineering Their STEM Careers toward Justice. American Journal of Education, 124(1), 1–36. https://doi.org/10.1086/693954
Mejia, J. A., Revelo, R. A., Villanueva, I., & Mejia, J. (2018). Critical Theoretical Frameworks in Engineering Education: An Anti-Deficit and Liberative Approach. Education Sciences, 8(4), Article 4. https://doi.org/10.3390/educsci8040158
Mills, J. E., & Treagust, D. F. (2003). Engineering education—Is problem-based or project-based learning the answer. Australasian Journal of Engineering Education, 3(2), 2–16. http://www.aaee.com.au/journal/2003/mills_treagust03.pdf
Minichiello, A. (2018). From Deficit Thinking to Counter Storying: A Narrative Inquiry of Nontraditional Student Experience within Undergraduate Engineering Education. International Journal of Education in Mathematics, Science and Technology, 6(3), 266–284. https://doi.org/10.18404/ijemst.428188
Moesby, E. (2005). Curriculum Development for Project-Oriented and Problem-Based Learning (POPBL) with Emphasis on Personal Skills and Abilities. Global Journal of Engineering Education, 9(2), 21-28. https://wiete.com.au/journals/GJEE/Publish/vol9no2/Moesby.pdf
Mora, C., Anorbe, B., Gonzalez Marrero, A., & Martin-Gutierrez, J. (2017). Motivational Factors to Consider when Introducing Problem-Based Learning in Engineering Education Courses. International Journal of Engineering Education, 33(3), 1000–1017.
Olewnik, A., Horn, A., Schrewe, L., & Ferguson, S. (2023a). A Systematic Review of PBL Literature: In Search of Implementation Guidelines for Engineering Situated Problem Design, Facilitation, and Assessment. International Journal of Engineering Education, 39(6), 1450–1463.
Olewnik, A., Schrewe, L., & Ferguson, S. M. (2023b). Understanding Expert Perceptions of PBL Integration in Introductory Aerospace Engineering Courses: Thematic Analysis of Focus Groups with PBL and Aerospace Engineering Instructors. Proceedings of the ASEE Annual Conference & Exposition. https://peer.asee.org/44536
Pasandín, A. M., & Pérez, I. P. (2021). Developing theory from practice: A case study in civil engineering airport design problem-based learning. Computer Applications in Engineering Education, 29(5), 1112–1131. https://doi.org/10.1002/cae.22364
Passow, H. J., & Passow, C. H. (2017). What Competencies Should Undergraduate Engineering Programs Emphasize? A Systematic Review. Journal of Engineering Education, 106(3), 475–526. https://doi.org/10.1002/jee.20171
Perrenet, J. C., Bouhuijs, P. A. J., & Smits, J. G. M. M. (2000). The Suitability of Problem-Based Learning for Engineering Education; Theory and Practice. Teaching in Higher Education, 5(3), 345–358. https://doi.org/10.1080/713699144
Prince, M. J., & Felder, R. M. (2006). Inductive teaching and learning methods: Definitions, comparisons, and research bases. Journal of Engineering Education, 95(2), 123–138. https://doi.org/10.1002/j.2168-9830.2006.tb00884.x
Riis, J. O., Achenbach, M., Israelsen, P., Kyvsgaard Hansen, P., Johansen, J., & Deuse, J. (2017). Dealing with complex and ill-structured problems: Results of a Plan-Do-Check-Act experiment in a business engineering semester. European Journal of Engineering Education, 42(4), 396–412. https://doi.org/10.1080/03043797.2016.1189881
Roy, J. (2019). Engineering by the Numbers. https://www.asee.org/documents/papers-and-publications/publications/college-profiles/2018-Engineering-by-Numbers-Engineering-Statistics-UPDATED-15-July-2019.pdf
Schmidt, R. C. (1997). Managing Delphi Surveys Using Nonparametric Statistical Techniques. Decision Sciences, 28(3), 763–774. https://doi.org/10.1111/j.1540-5915.1997.tb01330.x
Servant-Miklos, V. F. C., & Kolmos, A. (2022). Student conceptions of problem and project based learning in engineering education: A phenomenographic investigation. Journal of Engineering Education, 111(4), 792–812. https://doi.org/10.1002/jee.20478
Servant-Miklos, V. F. C., Norman, G. R., & Schmidt, H. G. (2019). A Short Intellectual History of Problem-Based Learning. In M. Moallem, W. Hung, & N. Dabbagh (Eds.), The Wiley Handbook of Problem-Based Learning (pp. 3–24). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781119173243.ch1
Skulmoski, G. J., Hartman, F. T., & Krahn, J. (2007). The Delphi Method for Graduate Research. Journal of Information Technology Education: Research, 6(1), 1–21. https://doi.org/10.28945/199
Streveler, R., Olds, B., Miller, R., & Nelson, M. (2003). Using A Delphi Study to Identify The Most Difficult Concepts For Students To Master In Thermal And Transport Sciences. ASEE Annual Conference & Exposition, 8–1235. https://peer.asee.org/12592
Strobel, J., & van Barneveld, A. (2009). When is PBL More Effective? A Meta-synthesis of Meta-analyses Comparing PBL to Conventional Classrooms. Interdisciplinary Journal of Problem-Based Learning, 3(1). https://doi.org/10.7771/1541-5015.1046
Svihla, V., & Reeve, R. (2016). Facilitating Problem Framing in Project-Based Learning. Interdisciplinary Journal of Problem-Based Learning, 10(2). https://doi.org/10.7771/1541-5015.1603
Terenzini, P. T., Cabrera, A. F., Colbeck, C. L., Parente, J. M., & Bjorklund, S. A. (2001). Collaborative Learning vs. Lecture/Discussion: Students’ Reported Learning Gains. Journal of Engineering Education, 90(1), 123–130. https://doi.org/10.1002/j.2168-9830.2001.tb00579.x
Tharayil, S., Borrego, M., Prince, M., Nguyen, K. A., Shekhar, P., Finelli, C. J., & Waters, C. (2018). Strategies to mitigate student resistance to active learning. International Journal of STEM Education, 5(1), 7. https://doi.org/10.1186/s40594-018-0102-y
Tik, C. C. (2014). Problems Implementing Problem-Based Learning by a Private Malaysian University. Journal of Problem Based Learning in Higher Education, 2(1), Article 1. https://doi.org/10.5278/ojs.jpblhe.v2i1.1005
Valencia, R. R. (2010). Dismantling Contemporary Deficit Thinking: Educational Thought and Practice. Taylor & Francis Group. https://doi.org/10.4324/9780203853214
Warnock, J. N., & Mohammadi-Aragh, M. J. (2016). Case study: Use of problem-based learning to develop students’ technical and professional skills. European Journal of Engineering Education, 41(2), 142–153. https://doi.org/10.1080/03043797.2015.1040739
Wiggins, G. P., & McTighe, J. (2005). Understanding by Design. ASCD.
Woodcock, T., Adeleke, Y., Goeschel, C., Pronovost, P., & Dixon-Woods, M. (2020). A modified Delphi study to identify the features of high quality measurement plans for healthcare improvement projects. BMC Medical Research Methodology, 20(1), 8. https://doi.org/10.1186/s12874-019-0886-6
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c) 2025 Andrew Olewnik, Laine Schrewe, Scott Ferguson