Item Analysis of Information Literacy Test for Open, Distance and Digital Education New Learners
DOI:
https://doi.org/10.57125/ELIJ.2025.09.25.02Keywords:
information literacy, information literacy assessment, SCONUL Seven Pillars of Information Literacy, reliability management, test and testingAbstract
Assessing information literacy is key to understanding students’ ability to locate, evaluate, and use information effectively. This study aimed to develop and validate multiple-choice questions (MCQs) to assess the information literacy skills of new undergraduate learners in open, distance, and digital education (ODDE), using the SCONUL Seven Pillars of Information Literacy Model as the framework for the construct. A total of 60 MCQs were tested with 388 new learners. Item analysis evaluated the difficulty index (DIF I), discrimination index (DI), and distractor effectiveness (DE). Validity was established through expert review and construct alignment, while reliability was measured using Cronbach’s alpha and KR-20. Analysis via Excel and JASP showed that 41 items (68.3%) had acceptable difficulty levels, with a mean DIF I of 55.59 ± 10.63. Two items (3.2%) were too complicated, while 17 items (28.3%) were too easy. Based on DI, 54 items (90%) were rated as acceptable to excellent, and six (10%) were rated as poor. A weak positive correlation was found between DIF I and DI (r = 0.222, p > 0.05). Educators in various ODDEsettings can adapt the refined-items instrument to assess learners’ information literacy levels based on local needs contexts. Its use supports more precise diagnostic practices, enabling targeted instructional interventions and resource planning in digital learning environments. The study not only introduces a psychometrically validated IL test based on the SCONUL model but also provides actionable evidence to guide higher education institutions and policymakers in embedding information literacy into national digital literacy strategies, curriculum design, and quality assurance frameworks for ODDE.
References
Al-Qallaf, C. L. (2019). Information literacy assessment of incoming students in an information studies graduate program. Global Knowledge, Memory and Communication, 68(3), 223–241. https://doi.org/10.1108/GKMC-07-2018-0062
American Colleague and Research Libraries (ACRL). (2000). Information literacy competency standards for higher education. https://alair.ala.org/server/api/core/bitstreams/ce62c38e-971a-4a98-a424-7c0d1fe94d34/content
Association of College and Research Libraries. (2015). Framework for information literacy for higher education. https://www.ala.org/acrl/standards/ilframework
Babo, R., & Suhonen, J. (2018). E-assessment with multiple choice questions: A qualitative study of teachers’ opinions and experience regarding the new assessment strategy. International Journal of Learning Technology, 13(3), 220-248. https://doi.org/10.1504/IJLT.2018.095964
Bent M., & Stubbings, R. (2011). The SCONUL seven pillars of information literacy core mode. The Society of College, National and University Libraries. https://senseandreference.wordpress.com/wp-content/uploads/2016/12/coremodel.pdf
Boh Podgornik, B., Dolničar, D., Šorgo, A., & Bartol, T. (2016). Development, testing, and validation of an information literacy test (ILT) for higher education. Journal of the Association for Information Science and Technology, 67(10), 2420–2436. https://doi.org/10.1002/asi.23586
Cameron, L., Wise, S. L., & Lottridge, S. M. (2007). The development and validation of the information literacy test. College & Research Libraries, 68(3), 229–237. https://doi.org/10.5860/crl.68.3.229
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
Ebel, R. L. (1954). Procedures for the analysis of classroom tests. Educational and Psychological Measurement, 14(2), 352–364. https://doi.org/10.1177/001316445401400215
Elmas, R., Bodner, G. M., Aydogdu, B., & Saban, Y. (2018). The inclusion of science process skills in multiple choice questions: Are we getting any better? European Journal of Science and Mathematics Education, 6(1), 13–23. https://doi.org/10.30935/scimath/9519
Garcia, A. N., Ebert, I. S., Sun, H.-P., Davis, J., & Wu, B. (2024). Comparison of 3-option and 4-option multiple choice questions and related issues and practices. Credentialing Insights. https://www.credentialinginsights.org/Article/comparison-of-3-option-and-4-option-multiple-choice-questions-and-related-issues-and-practices
Gross, M., & Latham, D. (2012). What’s skill got to do with it? Information literacy skills and self‐views of ability among first‐year college students. Journal of the American Society for Information Science and Technology, 63(3), 574–583. https://doi.org/10.1002/asi.21681
Haataja, E. S. H., Tolvanen, A., Vilppu, H., Kallio, M., Peltonen, J., & Metsäpelto, R.-L. (2023). Measuring higher-order cognitive skills with multiple choice questions –potentials and pitfalls of Finnish teacher education entrance. Teaching and Teacher Education, 122, Article 103943. https://doi.org/10.1016/j.tate.2022.103943
Jones, G. (2020). Designing multiple-choice test items. In P. Winke & T. Brunfaut (Eds), The Routledge Handbook of Second Language Acquisition and Language Testing (1st edn, pp. 90–101). Routledge. https://doi.org/10.4324/9781351034784-11
Jones, W. L., & Mastrorilli, T. (2022). Assessing the Impact of an Information Literacy Course on Students’ Academic Achievement: A Mixed-Methods Study. Evidence Based Library and Information Practice, 17(2), 61–87. https://doi.org/10.18438/eblip30090
Kennette, L. N., & McIntosh, E. (2022). Your information literacy practices (YILP): Information literacy and resourcefulness. Partnership: The Canadian Journal of Library and Information Practice and Research, 17(2), 1–22. https://doi.org/10.21083/partnership.v17i2.6680
Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2(3), 151–160. https://doi.org/10.1007/BF02288391
Kumar, D., Jaipurkar, R., Shekhar, A., Sikri, G., & Srinivas, V. (2021). Item analysis of multiple choice questions: A quality assurance test for an assessment tool. Medical Journal Armed Forces India, 77, S85–S89. https://doi.org/10.1016/j.mjafi.2020.11.007
Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), Article 33267. https://doi.org/10.1525/collabra.33267
Lockhart, J. (2015). Using item analysis to evaluate the validity and reliability of an existing online information literacy skills assessment instrument. South African Journal of Libraries and Information Science, 80(2), 36–42. https://catalogue.leidenuniv.nl/discovery/fulldisplay?vid=31UKB_LEU:UBL_V1&docid=alma9939086066902711&lang=en&context=L&adaptor=Local%20Search%20Engine
Malik, A., Ali, S., Batool, S. H., & Ameen, K. (2022). Linking information literacy with research productivity: A survey of mathematicians in Pakistan. Portal: Libraries and the Academy, 22(2), 475–498. https://doi.org/10.1353/pla.2022.0026
Mery, Y., Newby, J., & Peng, K. (2011). Assessing the reliability and validity of locally developed information literacy test items. Reference Services Review, 39(1), 98–122. https://doi.org/10.1108/00907321111108141
O’Connor, L. G., Radcliff, C. J., & Gedeon, J. A. (2002). Applying systems design and item response theory to the problem of measuring information literacy skills. College & Research Libraries, 63(6), 528–543. https://doi.org/10.5860/crl.63.6.528
Oakleaf, M. (2008). Dangers and Opportunities: A Conceptual Map of Information Literacy Assessment Approaches. Portal: Libraries and the Academy, 8(3), 233–253. https://doi.org/10.1353/pla.0.0011
Pinto, M., Garcia-Marco, J., Caballero, D., Manso, R., Uribe, A., & Gomez, C. (2024). Assessing information, media and data literacy in academic libraries: Approaches and challenges in the research literature on the topic. The Journal of Academic Librarianship, 50(5), Article 102920. https://doi.org/10.1016/j.acalib.2024.102920
Rosman, T., Mayer, A.-K., & Krampen, G. (2015). Combining self-assessments and achievement tests in information literacy assessment: Empirical results and recommendations for practice. Assessment & Evaluation in Higher Education, 40(5), 740–754. https://doi.org/10.1080/02602938.2014.950554
Serap Kurbanoglu, S., Akkoyunlu, B., & Umay, A. (2006). Developing the information literacy self‐efficacy scale. Journal of Documentation, 62(6), 730–743. https://doi.org/10.1108/00220410610714949
Velou, M., & Ahila, E. (2020). Refine the multiple choice questions tool with item analysis. International Archives of Integrated Medicine, 7(8), 80-85. https://www.iaimjournal.com/storage/2020/08/iaim_2020_0708_13.pdf
Walsh, A. (2009). Information literacy assessment: Where do we start? Journal of Librarianship and Information Science, 41(1), 19–28. https://doi.org/10.1177/0961000608099896
Walters, W. H., Sheehan, S. E., Handfield, A. E., López-Fitzsimmons, B. M., Markgren, S., & Paradise, L. (2020). A multi-method information literacy assessment program: Foundation and early results. Libraries and the Academy, 20(1), 101–135. https://preprint.press.jhu.edu/portal/sites/default/files/20.1walters.pdf
Walvoord, B. E. F. (2010). Assessment clear and simple: A practical guide for institutions, departments, and general education (2nd ed). Jossey-Bass.
Xu, X., Kauer, S., & Tupy, S. (2016). Multiple-choice questions: Tips for optimising assessment in-seat and online. Scholarship of Teaching and Learning in Psychology, 2(2), 147–158. https://doi.org/10.1037/stl0000062
Yu, C. (2023). Integrating information literacy and academic writing: Developing a self-assessment scale of information-based academic writing. The Journal of Academic Librarianship, 49(6), Article 102804. https://doi.org/10.1016/j.acalib.2023.102804
Zhou, X., & Jenkins, R. (2020). Dunning–Kruger effects in face perception. Cognition, 203, Article 104345. https://doi.org/10.1016/j.cognition.2020.104345
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Shahril Effendi Ibrahim, Md Rosli Ismail
This work is licensed under a Creative Commons Attribution 4.0 International License.