Designing AI-Aware Assessment Models to Measure Students’ Genuine English Proficiency

Abstract

The rapid advancement of artificial intelligence (AI) has transformed language assessment practices, offering increased efficiency and consistency in scoring. However, concerns remain regarding the validity of AI-based assessment in measuring students’ genuine English proficiency, particularly in productive language skills. This study aims to design and evaluate an AI-aware assessment model that aligns technological innovation with communicative competence frameworks. Employing a design-based research approach, the study involved 120 secondary-level EFL students and six English teachers in an authentic classroom context. The assessment model comprised four performance-based tasks—two writing and two speaking—evaluated using shared multidimensional rubrics applied by both AI-assisted scoring and human raters. Quantitative data were analyzed through descriptive statistics and correlation analysis, while qualitative data were examined thematically. The findings indicate that AI-assisted scoring demonstrates moderate to high consistency with human ratings in linguistic accuracy, lexical range, and coherence. However, discrepancies were observed in assessing pragmatic and communicative effectiveness, underscoring the limitations of fully automated evaluation. The study concludes that AI-aware assessment models are most effective when implemented within a human–AI collaborative framework. Such an approach enhances assessment efficiency and diagnostic feedback while preserving construct validity and ethical accountability in measuring genuine English proficiency.