An Integrated Multidimensional Item Response Theory Validation Framework for a Chemistry Achievement Test
DOI:
https://doi.org/10.71291/rwykgq70Keywords:
Chemistry Achievement Assessment, DETECT Method, Differential Item Functioning, Measurement Invariance, Multidimensional Item Response TheoryAbstract
The validity of chemistry achievement assessments remains a concern due to the multidimensional nature of the construct and the continued reliance on fragmented validation approaches. This study addresses this gap by adopting an integrated psychometric validation framework grounded in multidimensional Item Response Theory (MIRT). A 150-item dichotomously scored Chemistry Achievement Test (CAT) was administered to 741 senior secondary school students. Dimensionality was examined using Stout’s DETECT procedure, revealing weak multidimensionality characterised by a dominant general factor and three secondary dimensions, thereby supporting a four-dimensional structure. Competing MIRT models (M1PL, M2PL, M3PL, and M4PL) were evaluated using likelihood-ratio tests and information criteria. The multidimensional three-parameter logistic (M3PL) model demonstrated superior fit and parsimony, underscoring the importance of modelling both item discrimination and guessing in multiple-choice contexts. Local item independence, assessed via Yen’s Q3 statistic, showed no evidence of problematic residual correlations. Item parameter screening indicated that 84% of items met established criteria, leading to the removal of poorly functioning items. Measurement invariance was evaluated through IRT likelihood-ratio Differential Item Functioning (DIF) analysis across sex and school type, resulting in a final set of 112 invariant items. The findings provide strong evidence for the structural validity, reliability, and fairness of the CAT. Beyond its empirical contribution, the study advances an integrated validation framework that systematically combines dimensionality assessment, model evaluation, local independence diagnostics, and invariance testing, offering a robust template for developing equitable and psychometrically sound science assessments.
References
Ackerman, T. A., Gierl, M. J., & Walker, C. M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37–51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x DOI: https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman.
Bichi, A. A., & Talib, R. (2018). Item response theory: An introduction to latent trait models to DOI: https://doi.org/10.11591/ijere.v7i2.12900
test and item development. International Journal of Evaluation and Research in Education, 7(2), 142–151.
Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch Analysis in the Human Sciences. DOI: https://doi.org/10.1007/978-94-007-6857-4
Jang, E. E., & Roussos, L. A. (2007). An investigation into the dimensionality of TOEFL using conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 31(4), 287–310.
Chalmers, R. P. (2018). Improving the Crossing-SIBTEST statistic for detecting non-uniform differential item functioning. Psychometrika, 83(2), 376–386. DOI: https://doi.org/10.1007/s11336-017-9583-8
Chandio, M. T., Pandhiani, S. M., & Iqbal, R. (2016). Bloom’s taxonomy: Improving assessment
and teaching-learning process. Journal of Education and Educational Development, 3(2), 203–221.
Chen, W. H., & Thissen, D. (1997). Local dependence indices for item pairs using item response DOI: https://doi.org/10.2307/1165285
theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289.
Christensen, K. B., Makransky, G., & Horton, M. (2017). Critical values for Yen's Q3: Identification of local dependence in the Rasch model using residual correlations. Applied Psychological Measurement, 41(3), 178–194. https://doi.org/10.1177/0146621616677520 DOI: https://doi.org/10.1177/0146621616677520
Cunha, W., Viegas, F., França, C., Rosa, T., Rocha, L., & Gonçalves, M. A. (2023). A comparative survey of instance selection methods applied to non-neural and transformer-based text classification. ACM Computing Surveys, 55(13s), 1–52. https://doi.org/10.1145/3582000 DOI: https://doi.org/10.1145/3582000
De Ayala, R. J. (2009). The theory and practice of item response theory. Guilford Press.
DeMars, C. (2010). Item response theory. Oxford University Press. Dogra, V., Verma, S., Kavita, Chatterjee, P., Shafi, J., Choi, J., & Ijaz, M. F. (2022). A complete process of text classification system using state-of-the-art NLP models. Computational Intelligence and Neuroscience, 2022, 1883698. https://doi.org/10.1155/2022/1883698 DOI: https://doi.org/10.1155/2022/1883698
Du, X., Yang, J., Hung, J.-L., & Shelton, B. (2020). Educational data mining: A systematic review of research and emerging trends. Information Discovery and Delivery, 48(4), 225–236. DOI: https://doi.org/10.1108/IDD-09-2019-0070
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates. DOI: https://doi.org/10.1037/10519-153
Finch, W. H., & French, B. F. (2019). Educational and psychological measurement. Routledge. https://doi.org/10.4324/9781315650951 DOI: https://doi.org/10.4324/9781315650951
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response
theory. Sage Publications.
Madhavi, A., Nagesh, A., & Govardhan, A. (2023). Efficient course recommendation using deep transformer-based ensembled attention model. EAI Endorsed Transactions on E-Learning, 9. DOI: https://doi.org/10.4108/eetel.4470
Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5–8. https://doi.org/10.1111/j.1745-3992.1995.tb00881.x DOI: https://doi.org/10.1111/j.1745-3992.1995.tb00881.x
Oladele, J. I., & Ndlovu, M. (2021). A review of standardised assessment development procedure and algorithms for computer adaptive testing: Applications and relevance for fourth industrial revolution. International Journal of Learning, Teaching and Educational Research, 20(5), 1-17. https://doi.org/10.26803/ijlter.20.5.1 DOI: https://doi.org/10.26803/ijlter.20.5.1
Penfield, R. D., & Camilli, G. (2007). Differential item functioning and item bias. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp. 125–167). Elsevier. https://doi.org/10.1016/S0169-7161(06)26005-X DOI: https://doi.org/10.1016/S0169-7161(06)26005-X
Reckase, M. D. (2009). Multidimensional item response theory. Springer. DOI: https://doi.org/10.1007/978-0-387-89976-3
Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4(3), 207–230 DOI: https://doi.org/10.3102/10769986004003207
Rios, J. A., & Wells, C. S. (2014). Validity evidence based on internal structure. Psicothema, 26(1), 108–116. DOI: https://doi.org/10.7334/psicothema2013.260
Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52(4), 589–617. https://doi.org/10.1007/BF02294821 DOI: https://doi.org/10.1007/BF02294821
Stout, W., Habing, B., Douglas, J., Kim, H. R., Roussos, L., & Zhang, J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20(4), 331–354. DOI: https://doi.org/10.1177/014662169602000403
Sung, C., Dhamecha, T. I., & Mukhi, N. (2019). Improving short answer grading using transformer-based pre-training. In Proceedings of the International Conference on Artificial Intelligence in Education (pp. 469–481) DOI: https://doi.org/10.1007/978-3-030-23204-7_39
Thissen, D., Steinberg, L., & Wainer, H. (1993). Detection of differential item functioning using
the parameters of item response models. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 67–113). Lawrence Erlbaum Associates.
Weibelzahl, S., Paramythis, A., & Masthoff, J. (2020). Evaluation of adaptive systems. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization (pp. 394–395). DOI: https://doi.org/10.1145/3340631.3398668
Woods, C. M. (2009). Evaluation of MIMIC-model methods for DIF testing with comparison to DOI: https://doi.org/10.1080/00273170802620121
two-group analysis. Multivariate Behavioral Research, 44(1), 1–27.
Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. https://doi.org/10.1177/014662168400800201 DOI: https://doi.org/10.1177/014662168400800201
Zaidi, N. L. B., Grob, K. L., Monrad, S. M., Kurtz, J. B., Tai, A., Ahmed, A. Z., Gruppen, L. D., & Santen, S. A. (2018). Pushing critical thinking skills with multiple-choice questions: Does Bloom’s taxonomy work? Academic Medicine, 93(6), 856–859. DOI: https://doi.org/10.1097/ACM.0000000000002087
Zhang, J., & Stout, W. (1999). The theoretical DETECT index of dimensionality and its
application to approximate a simple structure. Psychometrika, 64(2), 213–249.
Zhang, J. (2007). Conditional covariance-based nonparametric multidimensionality assessment. Psychometrika, 72(2), 181–198.
Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning
(DIF). National Defense Headquarters.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Abayomi Funmilayo Rebecca, Modupe M. Osokoya (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
