Towards a Holistic Understanding of Evidence: A Working Paper
Keywords:
educational technologies, EdTechSynopsis
This report synthesizes reflections and recent literature to propose a rethinking of evidence generation in educational technology (EdTech). It advocates for three paradigm shifts to advance the discourse and practice in EdTech impact:
1) Evidence of impact should be understood as multidimensional, encompassing a variety of measures, not only experimental evidence.
2) Evidence collection should support an iterative and ongoing process of evaluating and improving impact over time.
3) Rigorous approaches are essential throughout the entire cycle of evidence to ensure validity and reliability.
Recognizing that stakeholders in the EdTech ecosystem have diverse incentives and metrics for success, the report argues that aligning these efforts toward a shared goal of improving educational equity can accelerate the production of sharable knowledge and foster wider, more meaningful impacts.
References
Angevine, C., Cator, K., Liberman, B., Smith, K., & Young, V. (2019). Designing a Process for Inclusive Innovation: A Radical Commitment to Equity. Version 1.0. DigitalPromise.
https://doi.org/10.51388/20.500.12265/86
Arieli-Attali M, Ward S, Thomas J, Deonovic B, von Davier AA. The Expanded Evidence-Centered Design (e-ECD) for Learning and Assessment Systems: A Framework for Incorporating Learning Goals and Processes Within Assessment Design. Front Psychol. 2019 Apr 26;10:853. PMID: 31105616; PMCID: PMC6498139.
https://doi.org/10.3389/fpsyg.2019.00853
Aubrey-Smith, F., & Twining, P. (2023). From EdTech to PedTech: Changing the Way We Think about Digital Technology. Taylor & Francis.
https://doi.org/10.4324/9781003321637
Coulson, A. (2019). Why You Need More Than "One Good Study" To Evaluate EdTech, Why You Need More Than "One Good Study" To Evaluate EdTech (mindresearch.org)
Dockterman, D. (2018). Insights from 200+ years of personalized learning. npj Science of Learning, 3(1), 15
https://doi.org/10.1038/s41539-018-0033-x
Fishman, B. J., Penuel, W. R., Allen, A. R., Cheng, B. H., & Sabelli, N. O. R. A. (2013). Design-based implementation research: An emerging model for transforming the relationship of research and practice. Teachers College Record, 115(14), 136-156. Global Education Evidence Advisory Panel: https://www.worldbank.org/en/topic/teachingandlearning/brief/globaleducation-evidence-advisory-panel
https://doi.org/10.1177/016146811311501415
Holt, L. (2024) The 5 Percent Problem: Online mathematics programs may benefit most the kids who need it least, Education Next, https://www.educationnext.org/5-percent-problem-online-mathematicsprograms-may-benefit-most-kids-who-need-it-least/
Kucirkova, N., Brod, G., & Gaab, N. (2023). Applying the science of learning to EdTech evidence evaluations using the EdTech Evidence Evaluation Routine (EVER). npj Science of Learning, 8(1), 35.
https://doi.org/10.1038/s41539-023-00186-7
Macgilchrist, F., Potter, J., & Williamson, B. (2024). Challenging the inequitable impacts of edtech. Learning, Media and Technology, 1-4.
https://doi.org/10.1080/17439884.2024.2350117
OECD (2018). The Future of Education and Skills: Education 2030. The Future We Want. Available online from: https://repository.canterbury.ac.uk/download/96f6c3f39ae6dcffa26e72cefe47684172da0c93db0a63d78668406e4f478ae8/3102592/E2030%20Position%20Paper%20%2805.04.2018%29.pdf
UNESCO (2023) Technology in education A tool on whose terms?, GEM Report, Available online from: https://www.unesco.org/gem-report/en/technology
