CEFR based test Validation and Fairness ‎Assessment

Document Type : research article

Authors

1 Professor of TEFL University of Tehran

2 Assistant Professor of TEFL University of Tehran

3 Ph.D. Candidate University of Tehran Kish International Campus

Abstract

Fairness in test assessment and evaluation is one of the most important goals in the field of ‎language learning and assessment. In recent decades, experts have focused ‎on the relationship between Validity and Fairness. However, some scholars believe that validity ‎is part of fairness. At the same time others are of the idea that achieving fairness is ‎in fact the acquisition of test validity. To achieve a sound and unified conceptualization of the relationship between these two concepts numerous qualitative and quantitative methods have been proposed and implemented.The Common European Frame of Reference (CEFR) is claimed to be a culture neutral that to the effect that all languages, regardless of the social and cultural backgrounds can be subsumed under the proposed system of descriptors for achieving educational equality and ‎assessment in teaching and learning. In this research, a test of 25 items based ‎on the CEFR descriptors was developed. In order to obtain adequate information ‎necessary to determine the test validity, 150 male and female participants took the test. The participants were all freshman in studying in various engineering courses. In addition to the mentioned test, a questionnaire of 75 items was used to distinguish ‎groups of participants based on their interests, educational, and occupational goals. The data were analyzed to determine the ‎Differential Item Functioning (DIF) of each item by Item Response Theory ‎‎(IRT). Combined together it was determined that in test validation procedure and investigation of test fairness test takers' interest also accounts.

Keywords

References

Alderson, J.C., (2007). The CEFR and the Need for More Research The Modern Language Journal, 91(4), pp. 659-663.
Ali, Z. M., Ali, F., Radzuan, N. M., Alwi, N. N. M., Abu, N. L., & Kassim, Z. (2018). Contextualising the CEFR: the Universiti Malaysia Pahang English language proficiency writing test. In 11th Annual International Conference of Education, Research and Innovation (ICERI 2018) (pp. 4892-4902).
Arrighi, J. T., & Piccoli, L. (2018). SWISSCIT Index on Citizenship Law in Swiss Cantons: Conceptualisation, Measurement, Aggregation. Université de Neuchâtel.
Arsalan, A. & Özeinci, S. (2017) A CEFR-based Curriculum Design for Tertiary Education Level International Journal of Languages’ Education and Teaching. Volume 5, Issue 3, p. 12-36
Bachman, L. F. (2003). Constructing an assessment use argument and supporting claims about test taker-assessment task interactions in evidence-centered assessment design. MEASUREMENT-LAWRENCE ERLBAUM ASSOCIATES-, 1, 63-65.
Bachman, L. F. (2005). Building and supporting a case for test use. Language Assessment Quarterly,2, 1–34.
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford, England: Oxford University Press.
Bordbar, S. (2020). Differential Item Functioning of University Entrance Exam: Using Rasch Analysis. Foreign Language Research Journal, 10(1), 37-55. doi: 10.22059/jflr.2019.278170.611
Council of Europe. (2001). Common European Framework of Reference for Languages. Cambridge, UK: Cambridge University Press.
Cummins, A. (2012). Validation of language assessments. The encyclopedia of applied linguistics.
Davies, A., Brown, A., Elder, C., Hill, K., Lumley, T., & McNamara, T. (1999). Dictionary of language testing. Cambridge, England: UCLES/Cambridge University Press.
Davis, A. (2013). Fifty Years of Language Assessments. In The Companion to Language Assessment, A. J. Kunnan (Ed). doi:10.1002/9781118411360.wbcla127
Dörnyei, Z., & Taguchi, T. (2009). Questionnaires in second language research: Construction, administration, and processing. Routledge.
Domna, A., & Zafiri, M. (2018). A Case Study of Two Groups of A1 Level Students in English. Education and Linguistic Research.
Elatia, S. (2011). Choosing language competence descriptors for language assessment: validity and fairness issues. Synergies Europe, 6, 165-175.
Fulcher, G., & Davidson, F. (Eds.). (2013). The Routledge handbook of language testing. Routledge.
Guerra, L., Gonçalves, O., Fisne, F. N., & Gungor, M. N. (2018). A CEFR-based comparison of ELT curriculum and course books used in Turkish and Portuguese primary schools.
HUANG, T., & JIA, G. (2012). The feasibility study of linking language test to the CEFR—Taking College English Test as an example [J]. Foreign Language Testing and Teaching, 1, 007.
Kane, M. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527–35.
Karami, M., Jafarigohar, M., Tajeddin, Z., & Rouhi, A. (2018). Input-induced Variation in EFL Learners’ Oral Production in Terms of Complexity, Accuracy, and Fluency. Iranian Journal of English for Academic Purposes, 6(2), 70-85.
 Karimi, F., Chalak, A., & Biria, R. (2019). Pedagogical Utility of Pre-Listening Activities for Improving Iranian Elementary EFL Learners’ Listening Comprehension. International Journal of Instruction, 12(1), 1127-1140.
Kunnan, A. (2000). Fairness and justice for all. In A. J. Kunnan (Ed.), Fairness and validation in language assessment, Vol. 9, Studies in language testing (pp. 1–14). Cambridge, UK: UCLES/CUP.
Kunnan, A. (2004). Test fairness. In M. Milanovic & C. Weir (Eds.), European language testing in a global context (pp. 27–48). Cambridge, England: UCLES/Cambridge University Press.
Kunnan, A. (2008). Towards a model of test evaluation: Using the test fairness and test context frameworks. In L. Taylor & C. Weir (Eds.), Multilingualism and assessment (pp. 229–51). Cambridge, England: UCLES/Cambridge University Press.
Le, H. T. T. (2018). Impacts of the CEFR-Aligned learning outcomes implementation on assessment practice. Hue University Journal of Science: Social Sciences and Humanities, 127(6B), 87-99.
 
Linacre, J. M. (2019). Winsteps[Computer program]. Chicago, IL: Winsteps.com.
Little, D. (2014). Learning, teaching, assessment: An exploration of their interdependence in the CEFR. In 5th international conference on teaching English as a foreign language assessment in ELT: Opportunities and challenges, FCSH, Lisbon new university, Portugal (pp. 21-22).
Lundell, F., Lindqvist, C., & Edmonds, A. (2018). Productive Collocation Knowledge at Advanced CEFR Levels: Evidence from the Development of a Test for Advanced L2 French. Canadian Modern Language Review, 74(4), 627-649.
Martyniuk, W. (Ed.). (2010). Aligning Tests with the CEFR: Reflections on Using the Council of Europe's Draft Manual (Vol. 33). Cambridge University Press.
Megyesi, B., Granstedt, L., Johansson, S., Prentice, J., Rosén, D., Schenström, C. J., ... & Volodina, E. (2018, November). Learner Corpus Anonymization in the Age of GDPR: Insights from the Creation of a Learner Corpus of Swedish. In Proceedings of the 7th Workshop on NLP for Computer Assisted Language Learning (NLP4CALL 2018) at SLTC, Stockholm, 7th November 2018 (No. 152, pp. 47-56). Linköping University Electronic Press.
McNamara, T. (2005). 21st century shibboleth: Language tests, identity and intergroup conflict. Language Policy, 4(4), 351-370.https://doi.org/10.1007/s10993-005-2886-0
McNamara, T., & Ryan, K. (2011). Fairness versus justice in language testing: The place of English literacy in the Australian Citizenship Test. Language Assessment Quarterly, 8(2), 161-178.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103).
New York: American Council on Education and Macmillan.
Nagai, N., & O’Dwyer, F. (2011). The actual and potential impacts of the CEFR on language education in Japan. Synergies Europe, 6, 141-152.
Negishi, M., Tono, Y., & Fujita, Y. (2012). A validation study of the CEFR levels of phrasal verbs in the English Vocabulary Profile. English Prole Journal, 3, e3 doi:10.1017/S2041536212000037
Nematzadeh, A. (2018). Construct Irrelevant Factors and Test Validity: Investigating the Relationship among Gender, Age, Mother Tongue, Field of Study and TOEFL IBT ® Results. Foreign Language Research Journal, 8(1), 139-166. doi: 10.22059/jflr.2018.242996.405
Peachy, W. S. (2012). şen, Y.(2012). The Common European Framework of Reference for Languages (CEFR) and English Language Teaching in Higher Education in Turkey. In International Higher Education Symposium.
Piccardo, E. (2012). Multidimensionality of assessment in the Common European Framework of References for Languages (CEFR). Les Cahiers de l’ILOB/OLBI Working Papers, 4, 37-54.
Pérez de la Calle, S. (2018). The use of advertisement as didactic resource in the foreign language classroom according to sociocultural, linguistic, sociolinguistic and pragmatic aspects.
Razavipour, K. (2019). Philosophy of Research in Language Testing: Investigating Papers Published in Iranian, Peer-reviewed, Domestic Journals from 2008 to 2018. Foreign Language Research Journal, 9(3), 831-860. doi: 10.22059/jflr.2019.262709.535
Shakirova, D. S., Zamaletdinov, R. R., & Ashrapova, A. K. (2018). The Impact of the Cefr in Testing Tatar as a Foreign Language (A2 Level). The Journal of Social Sciences Research, 4, 36-39.
Shaw, S., & Imam, H. (2013). Assessment of international students through the medium of English: Ensuring validity and fairness in content-based examinations. Language Assessment Quarterly, 10(4), 452-475.
Toulmin, S. E. (2003). The uses of argument. Cambridge university press.
Wu, J. R., & Wu, R. Y. (2007). Using the CEFR in Taiwan: The perspective of a local examination board. The Language Training and Testing Center Annual Report, 56, 1-20.
Xi, X. (2008). Methods of test validation. In E. Shohamy & N. H. Hornberger (Eds.), Encyclopediaof language and education. Vol. 7: Language testing and assessment (2nd ed., pp. 177–96). New York, NY: Springer.
Xi, X. (2010). How do we go about investigating test fairness? Language Testing, 27, 147–170
Journal of Foreign Language Research
Volume 12, Issue 1
April 2022
Pages 111-131

Files

Share

How to cite

Statistics