Consolidated Benchmark for Efficacy and Effectiveness Frameworks in EdTech
Keywords:
EdTech, evidence, efficacy, effectiveness, educational technologiesSynopsis
Post-COVID19 evaluation reports of Educational Technologies (EdTech) pointed to the imperative to scientifically evaluate EdTech’s impact on learners. The absence of a shared definition for such evaluations, coupled with the availability of diverse frameworks and criteria in the field, poses a challenge. This paper is concerned with two critical dimensions of impact on learning outcomes — efficacy and effectiveness — with a focus on teaching and learning EdTech for the K12 age range. A systematic literature search identified 65 frameworks that target the efficacy or effectiveness of K12 EdTech products. The frameworks were analysed in relation to their rigour, using the science of learning principles embedded in the EdTech Evidence Evaluation Routine (Kucirkova, Brod & Gaab, 2023). The results were synthesised into a consolidated benchmark that categorises the available frameworks at three levels based on the rigour applied to their assessments. The consolidated Effectiveness/Efficacy benchmark serves as a valuable tool for evaluating any EdTech type with available frameworks, facilitating informed decision-making in the dynamic landscape of educational technology.
Update 26th of July 2024: New file uploaded, with a different interface. No changes in content.
References
Asian Development Bank (2023). EdTech Product categorisation from Asian Development Bank 2022, adapted for Reimagine Tech-Inclusive Education: Evidence, Practices, and Road Map, Asian Development Bank, 2023.
Baloh, M., Zupanc, K., Kosir, D., Bosnić, Z., & Scepanović, S. (2015, June). A quality evaluation framework for mobile learning applications. Proceedings of the 4th Mediterranean Conference on Embedded Computing, Budva, Montenegro, 280-283.
https://doi.org/10.1109/MECO.2015.7181923
Baran, E., Uygun, E., & Altan, T. (2017). Examining preservice teachers' criteria for evaluating educational mobile apps. Journal of Educational Computing Research, 54(8), 1117-1141.
https://doi.org/10.1177/0735633116649376
Barber, M. & Rizvi, S (2013). Efficacy Framework: A Practical Approach to Improving Learner Outcomes. Pearson.
Biesta, G., Wainwright, E., & Aldridge, D. (2022). A case for diversity in educational research and educational practice. British Educational Research Journal, 48(1), 1-4.
https://doi.org/10.1002/berj.3777
Barr, R., & Kirkorian, H. (2023). Reexamining models of early learning in the digital age: Applications for learning in the wild. Journal of Applied Research in Memory and Cognition, 12(4), 457-472.
https://doi.org/10.1037/mac0000132
Bradley, J. (2012). Language-Learning Apps. (Product/service evaluation), Wired (San Francisco, Calif.), 20(10), p. 66.
Booton, S. A., Kolancali, P., & Murphy, V. A. (2023). Touchscreen apps for child creativity: An evaluation of creativity apps designed for young children. Computers & Education, 201, 104811.
https://doi.org/10.1016/j.compedu.2023.104811
Campbell, L. O., Gunter, G., & Braga, J. (2015). Utilizing the Retain Model to evaluate mobile learning applications. In D. Rutledge & D. Slykhuis (Eds.), Proceedings of the Society for Information Technology & Teacher Education International Conference, 732-736. Association for the Advancement of Computing in Education.
Chatzopoulos, A., Karaflis, A., Kalogiannakis, M., Tzerachoglou, A., Cheirchanteri, G., Sfyroera, E., & Sklavounou, E. O. (2023). Evaluation of Google Play educational apps for early childhood education. Advances in Mobile Learning Educational Research, 3(2), 770-778.
https://doi.org/10.25082/AMLER.2023.02.004
Chen, X. (2016). Evaluating language-learning mobile apps for second-language learners. Journal of Educational Technology Development and Exchange, 9(2).
https://doi.org/10.18785/jetde.0902.03
Cherner, T., Dix, J., & Lee, C. (2014). Cleaning up that mess: A framework for classifying educational apps. Contemporary Issues in Technology and Teacher Education, 14(2), 158-193. https://citejournal.org/volume-14/issue-2-14/general/cleaning-up-that-mess-a-framework-for-classifying-educational-apps/
Cherner, T., Fegely, A., Lee, C. Y., & Santaniello, L. (2016). A detailed rubric for assessing the quality of teacher resource apps. Journal of Information Technology Education: Innovations in Practice, 15(1), 117-143.
De Albuquerque, C. (2021). Bridging the Financial Gap: Investing in SDG 6, SDG Knowledge Hub Blog, https://sdg.iisd.org/commentary/guest-articles/bridging-the-financial-gap-investing-in-sdg-6/
Dekker, I., & Meeter, M. (2022). Evidence-based education: Objections and future directions. Frontiers in Education, 7, p. 941410.
https://doi.org/10.3389/feduc.2022.941410
Dore, R. A., Shirilla, M., Verdine, B. N., Zimmermann, L., Golinkoff, R. M., & Hirsh-Pasek, K. (2018). Developer meets developmentalist: improving industry-research partnerships in children's educational technology. Journal of Children and Media, 12(2), 227-235.
https://doi.org/10.1080/17482798.2018.1450086
Evidence for ESSA: Standards and Procedures, Version 2.1, November 2023, Evidence for ESSA, Available online from: www.evidenceforessa.org
Foster, D., McLemore, C., Olszewski, B., Chaudhry, A., Cooper, E., Forcier, L., & Luckin, R. (2023). EdTech Quality Frameworks and Standards Review. Department for Education, Published 14. December 2023, https://www.gov.uk/government/publications/edtech-quality-characteristics-frameworks-and-standards-review
Goodwin, K., & Kucirkova, N. (2012, March). iTouch and iLearn: An examination of educational apps. Paper presented at the Early Education and Technology for Children Conference, Salt Lake City, Utah, USA.
Global Newswire (2023). K-12 Education Technology Spend Market to hit USD 132.4 billion by 2032; Amid the growing demands for personalized and online learning, https://www.globenewswire.com/en/news-release/2023/10/17/2761545/0/en/K-12-Education-Technology-Spend-Market-to-hit-USD-132-4-billion-by-2032-Amid-the-growing-demands-for-personalized-and-online-learning.html
Gough. D. (2007). Weight of Evidence: a framework for the appraisal of the quality and relevance of evidence. Research Papers in Education, 22 (2), 213-228.
https://doi.org/10.1080/02671520701296189
Grand View Research. (2023). Global EdTech Market Size, Share & Trends Analysis Report. https://www.grandviewresearch.com/industry-analysis/edtech-market
GrandView Research (2023). Education Technology Market Size, Share & Trends Analysis Report By Sector (Preschool, K-12, Higher Education), By End-user (Business, Consumer), By Type, By Deployment, By Region, And Segment Forecasts, 2023 - 2030. Report ID: GVR-4-68038-878-7 https://www.grandviewresearch.com/industry-analysis/education-technology-market
Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91-108.
https://doi.org/10.1111/j.1471-1842.2009.00848.x
Hirsh-Pasek, K., Zosh, J. M., Golinkoff, R. M., Gray, J. H., Robb, M. B., & Kaufman, J. (2015). Putting education in "educational" apps: Lessons from the science of learning. Psychological Science in the Public Interest, 16(1), 3-34.
https://doi.org/10.1177/1529100615569721
Herodotou, C. (2021, May). MAD learn: an evidence-based affordance framework to assessing learning apps. In 2021 7th International Conference of the Immersive Learning Research Network (iLRN), 1-8. IEEE.
https://doi.org/10.23919/iLRN52045.2021.9459325
Huntington, B., Goulding, J., & Pitchford, N. (2021). Transforming global learning with digital technologies: A qualitative exploration of the use of educational technology with marginalised, out-of-school children living in remote settings. In EDULEARN21 proceedings, p. 9201. IATED.
https://doi.org/10.21125/edulearn.2021.1853
Huntington, B., Goulding, J., & Pitchford, N. J. (2023). Pedagogical features of interactive apps for effective learning of foundational skills. British Journal of Educational Technology.
https://doi.org/10.1111/bjet.13317
Hussain, A., Mkpojiogu, E. O. C., & Hassan, F. (2018). Dimensions and sub-dimensions for the evaluation of m-learning apps for children: A review. International Journal of Engineering and Technology, 7(3.20), 291-295.
https://doi.org/10.14419/ijet.v7i3.20.19168
Ibrahim, N.K. et al. (2019). Multi-Criteria Evaluation and Benchmarking for Young Learners' English Language Mobile Applications in Terms of LSRW Skills, IEEE access, 7, 146620-146651.
https://doi.org/10.1109/ACCESS.2019.2941640
Israelson, M. H. (2015). The app map: A tool for systematic evaluation of apps for early literacy learning. Reading Teacher, 69(3), 339-349.
https://doi.org/10.1002/trtr.1414
Kalogiannakis, M., & Papadakis, S. (2017, August). An evaluation of Greek educational android apps for preschoolers. Proceedings of the 12th Conference of the European Science Education Research Association, Dublin Ireland, 593-603.
Kay, R. (2018a, March). Creating a framework for selecting and evaluating educational apps. Proceedings of the 12th International Technology, Education and Development Conference, Valencia, Spain, 374-382.
https://doi.org/10.21125/inted.2018.0106
Kay, R. (2018b, October). Developing a framework to help educators select and use mobile apps in the classroom. Proceedings of E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Las Vegas, NV, USA, 1315-1320.
Kay, R., Lesage, A., & Tepylo, D. (2019, November). Evaluating the learning, design and engagement value of mobile applications: The mobile app evaluation scale. Proceedings of the 12th International Conference of Education, Research and Innovation, Seville, Spain, 1103-1107.
https://doi.org/10.21125/iceri.2019.0336
Khan, A. I., Al-Khanjari, Z., & Sarrab, M. (2017, April). Crowd sourced evaluation process for mobile learning application quality. Proceedings of the 2nd International Conference on Information Systems Engineering, Charleston, SC, USA.
https://doi.org/10.1109/ICISE.2017.17
Kolak, J., Norgate, S. H., Monaghan, P., & Taylor, G. (2021). Developing evaluation tools for assessing the educational potential of apps for preschool children in the UK. Journal of Children and Media, 15(3), 410-430.
https://doi.org/10.1080/17482798.2020.1844776
Konca, A. S., Izci, B., & Simsar, A. (2023). Evaluating popular STEM applications for young children. European early childhood education research journal, 1-17.
https://doi.org/10.1080/1350293X.2023.2221414
Kucirkova, N. (2018). A taxonomy and research framework for personalization in children's literacy apps. Educational Media International, 55(3), 255-272.
https://doi.org/10.1080/09523987.2018.1512446
Kucirkova, N. (2023a). Are EdTech companies the casualties or winners of educational evidence wars? BERA Blogs, https://www.bera.ac.uk/blog/are-edtech-companies-the-casualties-or-winners-of-educational-evidence-wars
Kucirkova, N. (2023b). How can philanthropy catalyse a system-wide change in EdTech? Alliance magazine, https://www.alliancemagazine.org/blog/how-can-philanthropy-catalyse-a-system-wide-change-in-edtech/
Kucirkova, N. I. (2023c). Ethics: fund an independent system to verify EdTech. Nature, 618(7966), 675-675.
https://doi.org/10.1038/d41586-023-01922-0
Kucirkova, N., Brod, G., & Gaab, N. (2023). Applying the science of learning to EdTech evidence evaluations using the EdTech Evidence Evaluation Routine (EVER). npj Science of Learning, 8(1), 35-42.
https://doi.org/10.1038/s41539-023-00186-7
Lee, C. Y., & Cherner, T. S. (2015). A comprehensive evaluation rubric for assessing instructional apps. Journal of Information Technology Education, 14(1), 21-53.
Lee, J. S., & Kim, S. W. (2015). Validation of a tool evaluating educational apps for smart education. Journal of Educational Computing Research, 52(3), 435-450.
https://doi.org/10.1177/0735633115571923
Lisenbee, P. S. (2018). Literacy app evaluation tool for teachers: Phonemic awareness and phonics apps rubric. https://plisenbee.wixsite.com/website/blank-page
Lubniewski, K. L., McArthur, C. L., & Harriott, W. A. (2017). Evaluating instructional apps using the app checklist for educators (ACE). International Electronic Journal of Elementary Education, 10(3), 323-329.
https://doi.org/10.26822/iejee.2018336190
Lytras, M.D. et al. (2019). Evaluation of Mobile Apps for Chinese Language Learning. In Cognitive Computing in Technology-Enhanced Learning. IGI Global, 191-205.
https://doi.org/10.4018/978-1-5225-9031-6.ch009
Mallawaarachchi, S.R. et al. (2023). Persuasive design-related motivators, ability factors and prompts in early childhood apps: A content analysis, Computers in Human Behavior, 139, 107492-.
https://doi.org/10.1016/j.chb.2022.107492
Martín-Monje, E., Arús, J., Rodríguez-Arancón, P., & Calle-Martínez, C. (2014). REALL: Rubric for the evaluation of apps in language learning. Proceedings of Jornadas Internacionales Tecnología Móvil e Innovación en el Aula: Nuevos Retos y Realidades Educativas. https://www.researchgate.com/publication/255702557_REALL_Rubric_for_the_evaluation_of_apps_in_language_learning
McManis, L. D., & Parks, J. (2011). Evaluating technology for early learners. Hatch, Inc. https://www.eschoolnews.com/files/2012/01/EvaluatingTechnology_ebook_toolkit.pdf
McQuiggan, S., McQuiggan, J., Sabourin, J., & Kosturko, L. (2015). Mobile Learning: A Handbook for Developers, Educators, and Learners. Somerset: Wiley.
https://doi.org/10.1002/9781118938942
McQuiggan, S., McQuiggan, J., Sabourin, J., & Kosturko, L. (2015). The business of educational apps. In S. McQuiggan, J. McQuiggan, J. Sabourin, & L. Kosturko, Mobile Learning: A Handbook for Developers, Educators, and Learners, 215-235. John Wiley & Sons. https://onlinelibrary.wiley.com/doi/10.1002/9781118938942.ch11
https://doi.org/10.1002/9781118938942.ch11
Meyer, M., Zosh, J. M., McLaren, C., Robb, M., McCaffery, H., Golinkoff, R. M., Hirsh-Pasek, K., & Radesky, J. (2021). How educational are "educational" apps for young children? App store content analysis using the Four Pillars of Learning framework. Journal of Children and Media, 15(4), 526-548.
https://doi.org/10.1080/17482798.2021.1882516
Montazami, A. et al. (2022). Why this app? How educators choose a good educational app. Computers and Education, 184, p. 104513-
https://doi.org/10.1016/j.compedu.2022.104513
Montazami, A. et al. (2022). Why this app? How parents choose good educational apps from app stores. British Journal of Educational Technology, 53(6), 1766-1792.
https://doi.org/10.1111/bjet.13213
Munthe, E., Erstad, O., Njå, M.B., Forsström, S., Gilje, Ø., Amdam, S., Moltudal, S., & Hagen, S.B. (2022). Digitalisering i grunnopplæring; kunnskap, trender og framtidig forskningsbehov. Kunnskapssenter for utdanning: Universitetet i Stavanger.
Mustaffa, F. Y., Salam, A. R., Muhammad, F., Bunari, G., & Asary, L. H. (2016). Literature review of educational app evaluation rubrics. Intervention in School and Clinic, 51(4), 244-252.
https://doi.org/10.1177/1053451215589179
Neumann, M., Wang, Y., Qi, G. Y., & Neumann, D. L. (2019). An evaluation of Mandarin learning apps designed for English speaking preschoolers. Journal of Interactive Learning Research, 30(2), 167-193.
Outhwaite, L. A., Early, E., Herodotou, C., & Van Herwegen, J. (2023). Understanding how educational maths apps can enhance learning: A content analysis and qualitative comparative analysis. British Journal of Educational Technology.
https://doi.org/10.1111/bjet.13339
Outhwaite, L. A., & Van Herwegen, J. (2023). Educational apps and learning: Current evidence on design and evaluation. British Journal of Educational Technology, 54(5), 1268-1272.
https://doi.org/10.1111/bjet.13360
Papadakis, S. (2021). Tools for evaluating educational apps for young children: A systematic review of the literature. Interactive Technology and Smart Education, 18(1), 18-49.
https://doi.org/10.1108/ITSE-08-2020-0127
Papadakis, S., Kalogiannakis, M., & Zaranis, N. (2017). Designing and creating an educational app rubric for preschool teachers. Education and Information Technologies, 22(6), 3147-3165.
https://doi.org/10.1007/s10639-017-9579-0
Papadakis, S., Vaiopoulou, J., Kalogiannakis, M., & Stamovlasis, D. (2020). Developing and exploring an evaluation tool for educational apps (E.T.E.A.) targeting kindergarten children. Sustainability, 12(10), 4201.
https://doi.org/10.3390/su12104201
Pearson, H.A., Montazami, A. and Dubé, A.K. (2022). Why this app: Can a video‐based intervention help parents identify quality educational apps? British Journal of Educational Technology [Preprint].
https://doi.org/10.1111/bjet.13284
Privitera, A. J., Ng, S. H. S., & Chen, S. H. A. (2023). Defining the Science of Learning: A Scoping Review. Trends in Neuroscience and Education, 100206.
https://doi.org/10.1016/j.tine.2023.100206
Rosell-Aguilar, F. (2017). State of the app: A taxonomy and framework for evaluating language learning mobile applications. CALICO Journal, 34(2), 243-258.
https://doi.org/10.1558/cj.27623
Sari, B., Takacs, Z. K., & Bus, A. G. (2019). What are we downloading for our children? Best-selling children's apps in four European countries. Journal of Early Childhood Literacy, 19(4), 515-532.
https://doi.org/10.1177/1468798417744057
Shahjad and Mustafa, K. (2022). A Systematic Literature Review on Learning Apps Evaluation, Journal of Information Technology Education, 21, 663-700.
Shoukry, L., Sturm, C., & Galal-Edeen, G. H. (2015). Pre-MEGa: A proposed framework for the design and evaluation of preschoolers' mobile educational games. In T. Sobh, & K. Elleithy, K. (Eds.), Innovations and A Systematic Literature Review on Learning Apps Evaluation Advances in Computing, Informatics, Systems Sciences, Networking and Engineering, 385-390. Springer.
https://doi.org/10.1007/978-3-319-06773-5_52
Singal, A. G., Higgins, P. D., & Waljee, A. K. (2014). A primer on effectiveness and efficacy trials. Clinical and Translational Gastroenterology, 5(1), e45.
https://doi.org/10.1038/ctg.2013.13
Streiner, D. L. (2002). The 2 "Es" of research: Efficacy and effectiveness trials. The Canadian Journal of Psychiatry, 47(6), 552-556.
https://doi.org/10.1177/070674370204700607
Sweeney, P. & Moore, C. (2012). Mobile Apps for Learning Vocabulary: Categories, Evaluation and Design Criteria for Teachers and Developers, International Journal of Computer-assisted Language Learning and Teaching, 2(4), 1-16.
https://doi.org/10.4018/ijcallt.2012100101
Tahir, R., & Arif, F. (2014). Framework for evaluating the usability of mobile educational applications for children. Proceedings of the Third International Conference on Informatics Engineering and Information Science, Lodz, Poland, 156-170. https://www.researchgate.com/publication/265852684_Framework_for_Evaluating_the_Usability_of_Mobile_Educational_Applications_for_Children
Tarcea, G., Puchala, B., Berman, T., Scorzelli, G., Pascucci, V., Taufer, M., & Allison, J. (2022). The Materials Commons Data Repository. In 2022 IEEE 18th International Conference on e-Science (e-Science), 405-406). IEEE.
https://doi.org/10.1109/eScience55777.2022.00060
Taylor, G. et al. (2022). Selecting educational apps for preschool children: How useful are website app rating systems? British Journal of Educational Technology, 53(5), 1262-1282.
https://doi.org/10.1111/bjet.13199
Tu, Y., Zou, D., & Zhang, R. (2020). A comprehensive framework for designing and evaluating vocabulary learning apps from multiple perspectives. International Journal of Mobile Learning and Organization, 14(3), 370- 397. https://doi.org/10.1504/IJMLO.2020.108199
https://doi.org/10.1504/IJMLO.2020.108199
UNESCO, 2023. Global Education Monitoring Report: Technology in Education: A tool on whose terms?, UNESCO. https://www.unesco.org/gem-report/en/technology
Urquhart, N., Lee, J., & Wood, E. (2023). Get That App!: Examining Parental Evaluations of Numeracy Apps. Journal of Research in Childhood Education, 1-15.
https://doi.org/10.1080/02568543.2023.2260433
Vackova, P., Cermakova Lindroos, A. & Kucirkova, N. (2023). Children's Digital Books: Development, Validation and Dissemination of Quality Criteria. Stavanger: University of Stavanger. https://ebooks.uis.no/index.php/USPS/catalog/book/268
https://doi.org/10.31265/USPS.268
Vaiopoulou, J., Papadakis, S., Sifaki, E., Stamovlasis, D., & Kalogiannakis, M. (2021). Parents' perceptions of educational apps use for kindergarten children: Development and validation of a new instrument (PEAU-p) and exploration of parents' profiles. Behavioral Sciences, 11(6), 82.
https://doi.org/10.3390/bs11060082
Vaiopoulou, J., Papadakis, S., Sifaki, E., Kalogiannakis, M., & Stamovlasis, D. (2022). Classification and evaluation of educational apps for early childhood: Security matters. Education and Information Technologies, 1-32.
https://doi.org/10.1007/s10639-022-11289-w
Vanbecelaere, S., Adam, T., Sieber, C., Clark-Wilson, A., Boody Adorno, K., & Haßler, B. (2023). Towards Systemic EdTech Testbeds: A Global Perspective. Global EdTech Testbeds Network.
https://doi.org/10.53832/opendeved.0285
Vázquez-Cano, E., Quicios-García, M. P., Fombona, J., & Rodríguez-Arce, J. (2023). Latent factors on the design and adoption of gamified apps in primary education. Education and Information Technologies, 1-31.
https://doi.org/10.1007/s10639-023-11797-3
Vincent, T. (n.d.a). Educational app evaluation checklist. Squarespace. https://static.squarespace.com/static/50eca855e4b0939ae8bb12d9/50ecb58ee4b0b16f176a9e7d/50ecb593e4b0b16f176aa976/1330884481041/Vincent_App_Checklist.pdf
Verwimp, C., Snellings, P., Wiers, R. W., & Tijms, J. (2023). A randomised proof‐of‐concept trial on the effectiveness of a game‐based training of phoneme‐grapheme correspondences in pre‐readers. Journal of Computer Assisted Learning (in press).
https://doi.org/10.1111/jcal.12821
Wadhwa, M., Zheng, J., & Cook, T. D. (2023). How Consistent Are Meanings of "Evidence-Based"? A Comparative Review of 12 Clearinghouses that Rate the Effectiveness of Educational Programs. Review of Educational Research.
https://doi.org/10.3102/00346543231152262
Wang, Y. Y., Wang, Y. S., Lin, H. H., & Tsai, T. H. (2019). Developing and validating a model for assessing paid mobile learning app success. Interactive Learning Environments, 27(4), 458-477.
https://doi.org/10.1080/10494820.2018.1484773
Wang, A. I., & Tahir, R. (2020). The effect of using Kahoot! for learning-A literature review. Computers & Education, 149, 103818.
https://doi.org/10.1016/j.compedu.2020.103818
XPRIZE. (2019). Global learning XPRIZE executive summary. https://www.xprize.org/prizes/global-learning/articles/glexp-executive-summary
