Investigating an online rater training program: Product and process 
14 March 2012
12pm
Venue: Arts 1, DALSL meeting room (206-408)
Applied Language Studies and Linguistics seminar by Rosemary Erlam and Janet von Randow DALSL & DELNA, The University of Auckland.
The Diagnostic English Language Needs Assessment (DELNA) for first-year students at The University of Auckland includes a writing task that identifies students who need support in the area of academic literacy. Although this is a low stakes assessment it is important that the scripts are rated accurately and that students receive relevant feedback. Face-to-face rater training takes place before each semester but because refresher training is required between batches of scripts an online rater training program was developed in 2004 to do this. Research carried out by Elder, Barkhuizen, Knoch and von Randow(2005, 2007) evaluated the effectiveness of the program, and, as a result, changes were made to the analytic descriptors and the online program itself.
Recently the issue has arisen as to whether this program can be used to train new raters as well. This initiative is in response to the increased interest that overseas institutions have shown in licensing components of the DELNA programme for use in their own contexts, and in particular requests for assistance in the training of raters to assess the DELNA writing task. The study reported on in this paper arose out of the need to establish the validity of using the DELNA online program to train raters new to DELNA, hereafter known as novice raters. Through the use of think-aloud protocols it also incorporated in its design a focus on the process of rating, allowing for an investigation of the rating behaviours that were associated with more reliable as opposed to less reliable rating. It was also expected that raters might respond differently to the online training according to experience and so the rating behaviours of experienced (expert) and novice raters were compared.
All welcome.
