Consolidated Benchmark for Efficacy and Effectiveness Frameworks in EdTech


EdTech, evidence, efficacy, effectiveness, educational technologies


Post-COVID19 evaluation reports of Educational Technologies (EdTech) pointed to the imperative to scientifically evaluate EdTech’s impact on learners. The absence of a shared definition for such evaluations, coupled with the availability of diverse frameworks and criteria in the field, poses a challenge. This paper is concerned with two critical dimensions of impact on learning outcomes — efficacy and effectiveness — with a focus on teaching and learning EdTech for the K12 age range. A systematic literature search identified 65 frameworks that target the efficacy or effectiveness of K12 EdTech products. The frameworks were analysed in relation to their rigour, using the science of learning principles embedded in the EdTech Evidence Evaluation Routine (Kucirkova, Brod & Gaab, 2023). The results were synthesised into a consolidated benchmark that categorises the available frameworks at three levels based on the rigour applied to their assessments. The consolidated Effectiveness/Efficacy benchmark serves as a valuable tool for evaluating any EdTech type with available frameworks, facilitating informed decision-making in the dynamic landscape of educational technology.

Author Biographies

Natalia Ingebretsen Kucirkova

University in Stavanger
Faculty of Arts and Education
Norwegian Centre for Learning Environment and Behavioral Research in Education

Anna Lindroos Cermakova

Senior Research Associate in Corpus Linguistics and Sociomaterial Methodologies
Lancaster University
Faculty of Arts and Social Sciences

Petra Vackova

The Open University


Asian Development Bank (2023). EdTech Product categorisation from Asian Development Bank 2022, adapted for Reimagine Tech-Inclusive Education: Evidence, Practices, and Road Map, Asian Development Bank, 2023.

Baloh, M., Zupanc, K., Kosir, D., Bosnić, Z., & Scepanović, S. (2015, June). A quality evaluation framework for mobile learning applications. Proceedings of the 4th Mediterranean Conference on Embedded Computing, Budva, Montenegro, 280-283.

Baran, E., Uygun, E., & Altan, T. (2017). Examining preservice teachers' criteria for evaluating educational mobile apps. Journal of Educational Computing Research, 54(8), 1117-1141.

Barber, M. & Rizvi, S (2013). Efficacy Framework: A Practical Approach to Improving Learner Outcomes. Pearson.

Biesta, G., Wainwright, E., & Aldridge, D. (2022). A case for diversity in educational research and educational practice. British Educational Research Journal, 48(1), 1-4.

Barr, R., & Kirkorian, H. (2023). Reexamining models of early learning in the digital age: Applications for learning in the wild. Journal of Applied Research in Memory and Cognition, 12(4), 457-472.

Bradley, J. (2012). Language-Learning Apps. (Product/service evaluation), Wired (San Francisco, Calif.), 20(10), p. 66.

Booton, S. A., Kolancali, P., & Murphy, V. A. (2023). Touchscreen apps for child creativity: An evaluation of creativity apps designed for young children. Computers & Education, 201, 104811.

Campbell, L. O., Gunter, G., & Braga, J. (2015). Utilizing the Retain Model to evaluate mobile learning applications. In D. Rutledge & D. Slykhuis (Eds.), Proceedings of the Society for Information Technology & Teacher Education International Conference, 732-736. Association for the Advancement of Computing in Education.

Chatzopoulos, A., Karaflis, A., Kalogiannakis, M., Tzerachoglou, A., Cheirchanteri, G., Sfyroera, E., & Sklavounou, E. O. (2023). Evaluation of Google Play educational apps for early childhood education. Advances in Mobile Learning Educational Research, 3(2), 770-778.

Chen, X. (2016). Evaluating language-learning mobile apps for second-language learners. Journal of Educational Technology Development and Exchange, 9(2).

Cherner, T., Dix, J., & Lee, C. (2014). Cleaning up that mess: A framework for classifying educational apps. Contemporary Issues in Technology and Teacher Education, 14(2), 158-193.

Cherner, T., Fegely, A., Lee, C. Y., & Santaniello, L. (2016). A detailed rubric for assessing the quality of teacher resource apps. Journal of Information Technology Education: Innovations in Practice, 15(1), 117-143.

De Albuquerque, C. (2021). Bridging the Financial Gap: Investing in SDG 6, SDG Knowledge Hub Blog,

Dekker, I., & Meeter, M. (2022). Evidence-based education: Objections and future directions. Frontiers in Education, 7, p. 941410.

Dore, R. A., Shirilla, M., Verdine, B. N., Zimmermann, L., Golinkoff, R. M., & Hirsh-Pasek, K. (2018). Developer meets developmentalist: improving industry-research partnerships in children's educational technology. Journal of Children and Media, 12(2), 227-235.

Evidence for ESSA: Standards and Procedures, Version 2.1, November 2023, Evidence for ESSA, Available online from:

Foster, D., McLemore, C., Olszewski, B., Chaudhry, A., Cooper, E., Forcier, L., & Luckin, R. (2023). EdTech Quality Frameworks and Standards Review. Department for Education, Published 14. December 2023,

Goodwin, K., & Kucirkova, N. (2012, March). iTouch and iLearn: An examination of educational apps. Paper presented at the Early Education and Technology for Children Conference, Salt Lake City, Utah, USA.

Global Newswire (2023). K-12 Education Technology Spend Market to hit USD 132.4 billion by 2032; Amid the growing demands for personalized and online learning,

Gough. D. (2007). Weight of Evidence: a framework for the appraisal of the quality and relevance of evidence. Research Papers in Education, 22 (2), 213-228.

Grand View Research. (2023). Global EdTech Market Size, Share & Trends Analysis Report.

GrandView Research (2023). Education Technology Market Size, Share & Trends Analysis Report By Sector (Preschool, K-12, Higher Education), By End-user (Business, Consumer), By Type, By Deployment, By Region, And Segment Forecasts, 2023 - 2030. Report ID: GVR-4-68038-878-7

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91-108.

Hirsh-Pasek, K., Zosh, J. M., Golinkoff, R. M., Gray, J. H., Robb, M. B., & Kaufman, J. (2015). Putting education in "educational" apps: Lessons from the science of learning. Psychological Science in the Public Interest, 16(1), 3-34.

Herodotou, C. (2021, May). MAD learn: an evidence-based affordance framework to assessing learning apps. In 2021 7th International Conference of the Immersive Learning Research Network (iLRN), 1-8. IEEE.

Huntington, B., Goulding, J., & Pitchford, N. (2021). Transforming global learning with digital technologies: A qualitative exploration of the use of educational technology with marginalised, out-of-school children living in remote settings. In EDULEARN21 proceedings, p. 9201. IATED.

Huntington, B., Goulding, J., & Pitchford, N. J. (2023). Pedagogical features of interactive apps for effective learning of foundational skills. British Journal of Educational Technology.

Hussain, A., Mkpojiogu, E. O. C., & Hassan, F. (2018). Dimensions and sub-dimensions for the evaluation of m-learning apps for children: A review. International Journal of Engineering and Technology, 7(3.20), 291-295.

Ibrahim, N.K. et al. (2019). Multi-Criteria Evaluation and Benchmarking for Young Learners' English Language Mobile Applications in Terms of LSRW Skills, IEEE access, 7, 146620-146651.

Israelson, M. H. (2015). The app map: A tool for systematic evaluation of apps for early literacy learning. Reading Teacher, 69(3), 339-349.

Kalogiannakis, M., & Papadakis, S. (2017, August). An evaluation of Greek educational android apps for preschoolers. Proceedings of the 12th Conference of the European Science Education Research Association, Dublin Ireland, 593-603.

Kay, R. (2018a, March). Creating a framework for selecting and evaluating educational apps. Proceedings of the 12th International Technology, Education and Development Conference, Valencia, Spain, 374-382.

Kay, R. (2018b, October). Developing a framework to help educators select and use mobile apps in the classroom. Proceedings of E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Las Vegas, NV, USA, 1315-1320.

Kay, R., Lesage, A., & Tepylo, D. (2019, November). Evaluating the learning, design and engagement value of mobile applications: The mobile app evaluation scale. Proceedings of the 12th International Conference of Education, Research and Innovation, Seville, Spain, 1103-1107.

Khan, A. I., Al-Khanjari, Z., & Sarrab, M. (2017, April). Crowd sourced evaluation process for mobile learning application quality. Proceedings of the 2nd International Conference on Information Systems Engineering, Charleston, SC, USA.

Kolak, J., Norgate, S. H., Monaghan, P., & Taylor, G. (2021). Developing evaluation tools for assessing the educational potential of apps for preschool children in the UK. Journal of Children and Media, 15(3), 410-430.

Konca, A. S., Izci, B., & Simsar, A. (2023). Evaluating popular STEM applications for young children. European early childhood education research journal, 1-17.

Kucirkova, N. (2018). A taxonomy and research framework for personalization in children's literacy apps. Educational Media International, 55(3), 255-272.

Kucirkova, N. (2023a). Are EdTech companies the casualties or winners of educational evidence wars? BERA Blogs,

Kucirkova, N. (2023b). How can philanthropy catalyse a system-wide change in EdTech? Alliance magazine,

Kucirkova, N. I. (2023c). Ethics: fund an independent system to verify EdTech. Nature, 618(7966), 675-675.

Kucirkova, N., Brod, G., & Gaab, N. (2023). Applying the science of learning to EdTech evidence evaluations using the EdTech Evidence Evaluation Routine (EVER). npj Science of Learning, 8(1), 35-42.

Lee, C. Y., & Cherner, T. S. (2015). A comprehensive evaluation rubric for assessing instructional apps. Journal of Information Technology Education, 14(1), 21-53.

Lee, J. S., & Kim, S. W. (2015). Validation of a tool evaluating educational apps for smart education. Journal of Educational Computing Research, 52(3), 435-450.

Lisenbee, P. S. (2018). Literacy app evaluation tool for teachers: Phonemic awareness and phonics apps rubric.

Lubniewski, K. L., McArthur, C. L., & Harriott, W. A. (2017). Evaluating instructional apps using the app checklist for educators (ACE). International Electronic Journal of Elementary Education, 10(3), 323-329.

Lytras, M.D. et al. (2019). Evaluation of Mobile Apps for Chinese Language Learning. In Cognitive Computing in Technology-Enhanced Learning. IGI Global, 191-205.

Mallawaarachchi, S.R. et al. (2023). Persuasive design-related motivators, ability factors and prompts in early childhood apps: A content analysis, Computers in Human Behavior, 139, 107492-.

Martín-Monje, E., Arús, J., Rodríguez-Arancón, P., & Calle-Martínez, C. (2014). REALL: Rubric for the evaluation of apps in language learning. Proceedings of Jornadas Internacionales Tecnología Móvil e Innovación en el Aula: Nuevos Retos y Realidades Educativas.

McManis, L. D., & Parks, J. (2011). Evaluating technology for early learners. Hatch, Inc.

McQuiggan, S., McQuiggan, J., Sabourin, J., & Kosturko, L. (2015). Mobile Learning: A Handbook for Developers, Educators, and Learners. Somerset: Wiley.

McQuiggan, S., McQuiggan, J., Sabourin, J., & Kosturko, L. (2015). The business of educational apps. In S. McQuiggan, J. McQuiggan, J. Sabourin, & L. Kosturko, Mobile Learning: A Handbook for Developers, Educators, and Learners, 215-235. John Wiley & Sons.

Meyer, M., Zosh, J. M., McLaren, C., Robb, M., McCaffery, H., Golinkoff, R. M., Hirsh-Pasek, K., & Radesky, J. (2021). How educational are "educational" apps for young children? App store content analysis using the Four Pillars of Learning framework. Journal of Children and Media, 15(4), 526-548.

Montazami, A. et al. (2022). Why this app? How educators choose a good educational app. Computers and Education, 184, p. 104513-

Montazami, A. et al. (2022). Why this app? How parents choose good educational apps from app stores. British Journal of Educational Technology, 53(6), 1766-1792.

Munthe, E., Erstad, O., Njå, M.B., Forsström, S., Gilje, Ø., Amdam, S., Moltudal, S., & Hagen, S.B. (2022). Digitalisering i grunnopplæring; kunnskap, trender og framtidig forskningsbehov. Kunnskapssenter for utdanning: Universitetet i Stavanger.

Mustaffa, F. Y., Salam, A. R., Muhammad, F., Bunari, G., & Asary, L. H. (2016). Literature review of educational app evaluation rubrics. Intervention in School and Clinic, 51(4), 244-252.

Neumann, M., Wang, Y., Qi, G. Y., & Neumann, D. L. (2019). An evaluation of Mandarin learning apps designed for English speaking preschoolers. Journal of Interactive Learning Research, 30(2), 167-193.

Outhwaite, L. A., Early, E., Herodotou, C., & Van Herwegen, J. (2023). Understanding how educational maths apps can enhance learning: A content analysis and qualitative comparative analysis. British Journal of Educational Technology.

Outhwaite, L. A., & Van Herwegen, J. (2023). Educational apps and learning: Current evidence on design and evaluation. British Journal of Educational Technology, 54(5), 1268-1272.

Papadakis, S. (2021). Tools for evaluating educational apps for young children: A systematic review of the literature. Interactive Technology and Smart Education, 18(1), 18-49.

Papadakis, S., Kalogiannakis, M., & Zaranis, N. (2017). Designing and creating an educational app rubric for preschool teachers. Education and Information Technologies, 22(6), 3147-3165.

Papadakis, S., Vaiopoulou, J., Kalogiannakis, M., & Stamovlasis, D. (2020). Developing and exploring an evaluation tool for educational apps (E.T.E.A.) targeting kindergarten children. Sustainability, 12(10), 4201.

Pearson, H.A., Montazami, A. and Dubé, A.K. (2022). Why this app: Can a video‐based intervention help parents identify quality educational apps? British Journal of Educational Technology [Preprint].

Privitera, A. J., Ng, S. H. S., & Chen, S. H. A. (2023). Defining the Science of Learning: A Scoping Review. Trends in Neuroscience and Education, 100206.

Rosell-Aguilar, F. (2017). State of the app: A taxonomy and framework for evaluating language learning mobile applications. CALICO Journal, 34(2), 243-258.

Sari, B., Takacs, Z. K., & Bus, A. G. (2019). What are we downloading for our children? Best-selling children's apps in four European countries. Journal of Early Childhood Literacy, 19(4), 515-532.

Shahjad and Mustafa, K. (2022). A Systematic Literature Review on Learning Apps Evaluation, Journal of Information Technology Education, 21, 663-700.

Shoukry, L., Sturm, C., & Galal-Edeen, G. H. (2015). Pre-MEGa: A proposed framework for the design and evaluation of preschoolers' mobile educational games. In T. Sobh, & K. Elleithy, K. (Eds.), Innovations and A Systematic Literature Review on Learning Apps Evaluation Advances in Computing, Informatics, Systems Sciences, Networking and Engineering, 385-390. Springer.

Singal, A. G., Higgins, P. D., & Waljee, A. K. (2014). A primer on effectiveness and efficacy trials. Clinical and Translational Gastroenterology, 5(1), e45.

Streiner, D. L. (2002). The 2 "Es" of research: Efficacy and effectiveness trials. The Canadian Journal of Psychiatry, 47(6), 552-556.

Sweeney, P. & Moore, C. (2012). Mobile Apps for Learning Vocabulary: Categories, Evaluation and Design Criteria for Teachers and Developers, International Journal of Computer-assisted Language Learning and Teaching, 2(4), 1-16.

Tahir, R., & Arif, F. (2014). Framework for evaluating the usability of mobile educational applications for children. Proceedings of the Third International Conference on Informatics Engineering and Information Science, Lodz, Poland, 156-170.

Tarcea, G., Puchala, B., Berman, T., Scorzelli, G., Pascucci, V., Taufer, M., & Allison, J. (2022). The Materials Commons Data Repository. In 2022 IEEE 18th International Conference on e-Science (e-Science), 405-406). IEEE.

Taylor, G. et al. (2022). Selecting educational apps for preschool children: How useful are website app rating systems? British Journal of Educational Technology, 53(5), 1262-1282.

Tu, Y., Zou, D., & Zhang, R. (2020). A comprehensive framework for designing and evaluating vocabulary learning apps from multiple perspectives. International Journal of Mobile Learning and Organization, 14(3), 370- 397.

UNESCO, 2023. Global Education Monitoring Report: Technology in Education: A tool on whose terms?, UNESCO.

Urquhart, N., Lee, J., & Wood, E. (2023). Get That App!: Examining Parental Evaluations of Numeracy Apps. Journal of Research in Childhood Education, 1-15.

Vackova, P., Cermakova Lindroos, A. & Kucirkova, N. (2023). Children's Digital Books: Development, Validation and Dissemination of Quality Criteria. Stavanger: University of Stavanger.

Vaiopoulou, J., Papadakis, S., Sifaki, E., Stamovlasis, D., & Kalogiannakis, M. (2021). Parents' perceptions of educational apps use for kindergarten children: Development and validation of a new instrument (PEAU-p) and exploration of parents' profiles. Behavioral Sciences, 11(6), 82.

Vaiopoulou, J., Papadakis, S., Sifaki, E., Kalogiannakis, M., & Stamovlasis, D. (2022). Classification and evaluation of educational apps for early childhood: Security matters. Education and Information Technologies, 1-32.‏

Vanbecelaere, S., Adam, T., Sieber, C., Clark-Wilson, A., Boody Adorno, K., & Haßler, B. (2023). Towards Systemic EdTech Testbeds: A Global Perspective. Global EdTech Testbeds Network.

Vázquez-Cano, E., Quicios-García, M. P., Fombona, J., & Rodríguez-Arce, J. (2023). Latent factors on the design and adoption of gamified apps in primary education. Education and Information Technologies, 1-31.

Vincent, T. (n.d.a). Educational app evaluation checklist. Squarespace.

Verwimp, C., Snellings, P., Wiers, R. W., & Tijms, J. (2023). A randomised proof‐of‐concept trial on the effectiveness of a game‐based training of phoneme‐grapheme correspondences in pre‐readers. Journal of Computer Assisted Learning (in press).

Wadhwa, M., Zheng, J., & Cook, T. D. (2023). How Consistent Are Meanings of "Evidence-Based"? A Comparative Review of 12 Clearinghouses that Rate the Effectiveness of Educational Programs. Review of Educational Research.

Wang, Y. Y., Wang, Y. S., Lin, H. H., & Tsai, T. H. (2019). Developing and validating a model for assessing paid mobile learning app success. Interactive Learning Environments, 27(4), 458-477.

Wang, A. I., & Tahir, R. (2020). The effect of using Kahoot! for learning-A literature review. Computers & Education, 149, 103818.

XPRIZE. (2019). Global learning XPRIZE executive summary.




January 11, 2024


Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.