Academic English proficiency assessment using a computerized adaptive test

Resumo

This paper describes the steps to convert a paper-and-pencil English proficiency test for academic purposes, consisting of multiple choice items administered following the Admissible Probability Measurement Procedure [24], adopted by the graduate program at the Institute of Mathematics and Computer Sciences at the University of São Paulo (ICMC-USP), Brazil, to a computerized adaptive test (CAT) based on an Item Response Theory Model (IRT). Despite the fact that the program accepts various internationally recognized tests that attest non-native speakers English proficiency, such as the Test of English as a Foreign Language (TOEFL), the International English Language Testing System (IELTS) and the Cambridge English: Proficiency (CPE), for instance, its requirement is incoherent in public universities in Brazil due to the cost, which ranges from US$ 200.00 to US$ 300.00 per exam. The TAI-PI software (Computerized Adaptive Test for English Proficiency), which was developed in Java language and SQLite, started to be used to assess the English pro?ciency of students on the program from October, 2013. The statistical methodology used was defined considering the history and aims of the test and adopted Samejima's Graded Response Model [21], the Kullback-Leibler information criterion for item selection, the a posteriori estimation method for latent trait [2] and the Shadow Test approach [29] to impose restrictions (content and test length) on the test composition of each individual. A description of the test design, the statistical methods used, and the results of a real application of TAI-PI for graduate students are presented in this paper, as well as the validation studies of the new methodology for pass or fail classification, showing the good quality of the new evaluation system and examination of improvement using the IRT and CAT methods.