Cognitive Diagnostic Assessment of a Reading Comprehension Test Battery for Formative Diagnostic Feedback

Document Type : research article

Authors

University of Tehran

Abstract

A critical issue in cognitive diagnostic assessment (CDA) lies in the dearth of research in developing tests for cognitive diagnostic purposes. CDA of formative language assessment that aims to inform instruction has not been conducted in foreign language learning contexts. During recent years, researches have attempted to provide more practical results beyond a simple average grade in order to improve learning. This study explored how developing a reading comprehension test based on a cognitive framework could be used for diagnostic purposes. To validate the developed test items, a Q-matrix that specified the relationships between test items and target attributes was theoretically developed. ُThink aloud verbal protocols and content expert judgment was used to aid in development of the Q-matrix. Finally, to diagnose the participants’ strengths and weaknesses, the test was administered to 1986 students of a GE course at the University of Tehran and Fusion Model analysis was used to obtain learners’ skill mastery profiles. The obtained results can be very beneficial for formative diagnostic feedback.

Keywords

References

منابع
Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press.
Alderson, J. C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. A&C Black.
Alderson, J. C. (2010). “Cognitive Diagnosis and Q-Matrices in Language Assessment”: A Commentary, 96-103.
Birch, B. M. (2002).English L2 reading: Getting to the bottom. Routledge.
Buck, G., Tatsuoka, K., &Kostin, I. (1997). The subskills of reading: Rule-space analysis of a multiple-choice test of second language reading comprehension. Language Learning, 47(3), 423-266.
Cohen, A. D., & Upton, T. A. (2006). Strategies in responding to the new TOEFL reading tasks (TOEFL Monograph No. MS-33). Princeton, NJ: ETS.
de la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163-183.
DiBello, L. V., Stout, W. F., & Roussos, L. (1995). Unified cognitive psychometric assessment likelihood-based classification techniques. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361–390). Hillsdale, NJ: Erlbaum.
DiBello, L., & Stout, W. (2008). Arpeggio documentation and analyst manual.Chicago: Applied informative assessment research enterprises (AIARE)—LLC.
Embretson, S. E. (1990). Diagnostic testing by measuring learning processes: Psychometric considerations for dynamic testing. In N. Frederiksen, R. Glaser, A. Lesgold, & M. G. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 407-432). Hillsdale, NJ: Erlbaum.
Fletcher, J. M. (2006). Measuring reading comprehension. Scientific Studies of Reading, 10(3), 323-330.
Francis, D.J., Snow, C.E., August, D., Carlson, C.D., Miller, J., & Iglesias, A. (2006). Measures of Reading Comprehension: A Latent Variable Analysis of the Diagnostic Assessment of Reading Comprehension, Scientific Studies of Reading, 10(3), 301-322.
Gorin, J. S. (2006). Test design with cognition in mind. Educational measurement: Issues and practice, 25(4), 21-35.
Hartman, H. J. (2001). Developing students’ metacognitive knowledge and skills. In H. J. Hartman (Ed.). Metacognition in learning and instruction: Theory, Research and Practice (pp. 33-68). Dordrecht, the Netherlands: Kluwer.
Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Dissertation AbstractsInternational: Section B: The Sciences and Engineering, 63(2-B), 864.
Hartz, S. M., & Roussos, L. A. (2005). The Fusion Model for skills diagnosis: Blending theory with practice. ETS Research Report. Princeton, NJ: Educational Testing Service.
Huang, T.W., & Wu, P.C. (2013). Classroom-based Cognitive Diagnostic Model for a Teacher-made Fraction- Decimal Test. Educational Technology & Society, 16 (3), 347–361.
Jang, E. E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG-TOEFL (Unpublished doctoral dissertation). University of Illinois at Urbana Champaign.Available from ProQuest Dissertations and Theses database. (AAT 3182288)
Jang, E. E. (2009). Demystifying a Q-matrix for making diagnostic inferences about L2 reading skills. Language Assessment Quarterly, 6(3), 210-238.
Leighton, J. P., Gierl, M. J., &Hunka, S. M. (2004). The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule‐Space Approach. Journal of Educational Measurement, 41(3), 205-237.
Li, H. (2011). A cognitive diagnostic analysis of the MELAB reading test.Spaan Fellow, 17.
Li, H., &Suen, H. K. (2013). Constructing and Validating a Q-Matrix for Cognitive Diagnostic Analyses of a Reading Test. Educational Assessment, 18(1), 1-25.
Madison, M. J., & Bradshaw, L. P. (2015). The Effects of Q-Matrix Design on Classification Accuracy in the Log-Linear Cognitive Diagnosis Model.Educational and Psychological Measurement, 75(3), 491-511.
Mislevy, R. J. (1994).Evidence and inference in educational assessment.Psychometrika, 59, 439–483.
Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33(4), 379-416.
Mislevy, R. J. (2006).Cognitive psychology and educational assessment.Educational measurement, 4, 257-305.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003).On the structure of educational assessments.Measurement: Interdisciplinary Research and Perspective, 1, 3–67.
Pellegrino, J. W., &Chudowsky, N. (2003). The Foundations of Assessment.Measurement: Interdisciplinary Research and Perspectives, 1(2), 103-148.
Roussos, L. A., DiBello, L. V., Stout, W. F., Hartz, S. M., Henson, R. A., & Templin, J. H. (2007). The fusion model skills diagnostic system. In J. Leighton & M. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications(pp. 275–318). New York, NY: Cambridge University Press.
Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441-474.
Rupp, A. A., Templin, J., & Henson, R. A. (2012). Diagnostic measurement: Theory, methods, and applications. Guilford Press.
Snow, R. E., &Lohman, D. F. (1989). Implications of cognitive psychology for educational measurement.American Council on Education.
Stiggins, R., Arter, J., &Chappuis, S. (2004). Classroom assessment for student learning: Doing it right—using it well. Dover, NH: Assessment Training Institute.
Tatsuoka, K. K. (1995). Architecture of knowledge structure and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman, and R. L. Brennan (Eds.), Cognitively Diagnostic Assessment (pp. 327-361). Hillsdale, NJ: Lawrence Erlbaum Associates.
Templin, J. L. (2004). Generalized linear mixed proficiency models for cognitive diagnosis. Available from ProQuest Dissertations and Theses database. (AAT3160960)
Urquhart, S., & Weir, C. J. (1998).Reading in a second language: Process, product and practice. New York: Longman.
Von Davier, M., DiBello, L., & Yamamoto, K. Y. (2006). Reporting test outcomes with models for cognitive diagnosis (ETS Research Rep. No. RR-06-28). Princeton, NJ: ETS.
Journal of Foreign Language Research
Volume 6, Issue 2
October 2017
Pages 321-342

Files

Share

How to cite

Statistics