Introduction
AES offers a time-saving tool to teachers and immediate diagnostic feedback to students for writing practices. However, automation also poses operational and technical challenges in classroom integration and in summative assessment.
Research on the impact of computer-based formative and summative assessment on student learning and performance has become increasingly important. Within the U.S. an increasing number of states have adopted AES programs in school- and classroom-based writing assessment as well as in state summative writing assessment. Recent emphasis on direct writing instruction and assessment has led to increased interest in using AES for curriculum program evaluation as well as instructional and learning feedback. Internationally, as English is being transformed from a foreign language into a global language, productive communication skills of English writing and speaking are being taught and tested as basic skills. As a result there is an increasing interest in the use of automated essay scoring technology in ESL/EFL teaching and learning.
When a state decides to adopt AES for a summative writing program, a process of reviewing different companies' AES software systems takes place. The commercial AES products usually present the reliability and validity evidence produced as part of the research-development cycle of the product. The availability and suitability of writing prompts, types of diagnostic feedbacks, and functionalities that assist teachers with classroom management are factors for consideration. After the state adoption of an AES product for its summative assessment, an ongoing validation process will begin; this ongoing validation of the intended use of the AES program is very much similar to validation of other summative assessment programs. The state is interested in validating the Online Writing Assessment AES scores. The state is also interested in answering the question, "Does AES use in the classroom improve writing?"
In this paper, we research two related issues in state-adopted AES programs. First, we explore how the state validates an AES program when AES is used for summative assessment. Second, we examine how the state can find out if an AES program used in classroom formative assessment is working. We present two-year empirical data from the state of West Virginia online writing assessments to show the process of evaluating the reliability and validity of AES scores as a state summative writing program. We examine the relationship between classroom writing practices using AES to the state summative writing test results; we analyze the effects that using AES has on communities by looking into differential performance based on rural versus urban schools, gender, and ethnicity.
The research questions in this paper involve the measurement quality of the writing assessment in terms of validity, reliability, comparability, and fairness. The paper has five sections. Part I provides an overview of the West Virginia Online Writing Assessment and classroom writing assessment using Writing Roadmap. Part II describes the process of automated essay-scoring engine training and validation for WV state online writing summative assessments. Scoring-engine believability and non-scorable essays are discussed. Part III provides evidence of the construct validity of the online summative writing by analyzing the relationship between online writing scores and the state NCLB West Virginia Education Standards Test Reading/Language Arts scores. Part IV presents students scores for 2007 operational online writing administrations based upon classroom practice using an AES program. The Writing Roadmap practice effect is analyzed using a quasi-experimental design for non-equivalent groups. The score relationship is analyzed by rural versus urban school type, gender, and ethnicity for Grade 7 of the WV 2006 and 2007 assessments. Part V provides a conclusion and discussion.
Continue Reading
Introduction
AES offers a time-saving tool to teachers and immediate diagnostic feedback to students for writing practices. However, automation also poses operational and technical challenges in classroom integration and in summative assessment.
Research on the impact of computer-based formative and summative assessment on student learning and performance has become increasingly important. Within the U.S. an increasing number of states have adopted AES programs in school- and classroom-based writing assessment as well as in state summative writing assessment. Recent emphasis on direct writing instruction and assessment has led to increased interest in using AES for curriculum program evaluation as well as instructional and learning feedback. Internationally, as English is being transformed from a foreign language into a global language, productive communication skills of English writing and speaking are being taught and tested as basic skills. As a result there is an increasing interest in the use of automated essay scoring technology in ESL/EFL teaching and learning.
When a state decides to adopt AES for a summative writing program, a process of reviewing different companies' AES software systems takes place. The commercial AES products usually present the reliability and validity evidence produced as part of the research-development cycle of the product. The availability and suitability of writing prompts, types of diagnostic feedbacks, and functionalities that assist teachers with classroom management are factors for consideration. After the state adoption of an AES product for its summative assessment, an ongoing validation process will begin; this ongoing validation of the intended use of the AES program is very much similar to validation of other summative assessment programs. The state is interested in validating the Online Writing Assessment AES scores. The state is also interested in answering the question, "Does AES use in the classroom improve writing?"
In this paper, we research two related issues in state-adopted AES programs. First, we explore how the state validates an AES program when AES is used for summative assessment. Second, we examine how the state can find out if an AES program used in classroom formative assessment is working. We present two-year empirical data from the state of West Virginia online writing assessments to show the process of evaluating the reliability and validity of AES scores as a state summative writing program. We examine the relationship between classroom writing practices using AES to the state summative writing test results; we analyze the effects that using AES has on communities by looking into differential performance based on rural versus urban schools, gender, and ethnicity.
The research questions in this paper involve the measurement quality of the writing assessment in terms of validity, reliability, comparability, and fairness. The paper has five sections. Part I provides an overview of the West Virginia Online Writing Assessment and classroom writing assessment using Writing Roadmap. Part II describes the process of automated essay-scoring engine training and validation for WV state online writing summative assessments. Scoring-engine believability and non-scorable essays are discussed. Part III provides evidence of the construct validity of the online summative writing by analyzing the relationship between online writing scores and the state NCLB West Virginia Education Standards Test Reading/Language Arts scores. Part IV presents students scores for 2007 operational online writing administrations based upon classroom practice using an AES program. The Writing Roadmap practice effect is analyzed using a quasi-experimental design for non-equivalent groups. The score relationship is analyzed by rural versus urban school type, gender, and ethnicity for Grade 7 of the WV 2006 and 2007 assessments. Part V provides a conclusion and discussion.
