Machine Scoring of Student Essays: Truth and Consequences

Published on Jan 1, 2006
Patricia Freitag Ericsson2
Estimated H-index: 2
Richard H. Haswell2
Estimated H-index: 2
The current trend toward machine-scoring of student work, Ericsson and Haswell argue, has created an emerging issue with implications for higher education across the disciplines, but with particular importance for those in English departments and in administration. The academic community has been silent on the issuesome would say excluded from itwhile the commercial entities who develop essay-scoring software have been very active. Machine Scoring of Student Essays is the first volume to seriously consider the educational mechanisms and consequences of this trend, and it offers important discussions from some of the leading scholars in writing assessment.Reading and evaluating student writing is a time-consuming process, yet it is a vital part of both student placement and coursework at post-secondary institutions. In recent years, commercial computer-evaluation programs have been developed to score student essays in both of these contexts. Two-year colleges have been especially drawn to these programs, but four-year institutions are moving to them as well, because of the cost-savings they promise. Unfortunately, to a large extent, the programs have been written, and institutions are installing them, without attention to their instructional validity or adequacy.Since the education software companies are moving so rapidly into what they perceive as a promising new market, a wider discussion of machine-scoring is vital if scholars hope to influence development and-or implementation of the programs being created. What is needed, then, is a critical resource to help teachers and administrators evaluate programs they might be considering, and to more fully envision the instructional consequences of adopting them. And this is the resource that Ericsson and Haswell are providing here.
  • References (126)
  • Citations (46)
📖 Papers frequently viewed together
314 Citations
420 Citations
142 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#2William C. WardH-Index: 16
Contents: Preface. R.E. Bennett, On the Meanings of Constructed Response. R.E. Traub, On the Equivalence of the Traits Assessed by Multiple-Choice and Constructed-Response Tests. R.E. Snow, Construct Validity and Constructed-Response Tests. S. Messick, Trait Equivalence as Construct Validity of Score Interpretation Across Multiple Methods of Measurement. R.J. Mislevy, A Framework for Studying Differences Between Multiple-Choice and Free-Response Test Items. K.K. Tatsuoka, Item Construction and P...
91 Citations
#1Yigal AttaliH-Index: 17
#2Jill BursteinH-Index: 26
420 Citations
#1David Bartholomae (University of Pittsburgh)H-Index: 11
Every time a student sits down to write for us, he has to invent the university for the occasion — invent the university, that is, or a branch of it, like history or anthropology or economics or English. The student has to learn to speak our language, to speak as we do, to try on the peculiar ways of knowing, selecting, evaluating, reporting, concluding, and arguing that define the discourse of our community. Or perhaps I should say the various discourses of our community, since it is in the nat...
393 CitationsSource
#1David S. Kaufer (CMU: Carnegie Mellon University)H-Index: 18
#2Cheryl Geisler (RPI: Rensselaer Polytechnic Institute)H-Index: 10
Last. Pantelis K. Vlachos (CMU: Carnegie Mellon University)H-Index: 6
view all 4 authors...
This chapter reports on a research program that investigates language and text from a rhetorical point of view. By rhetorical, we mean an approach that features the relationship between the speaker and the audience or between the writer and the reader. Fundamental to a rhetorical approach to language is an interest in linguistic and textual agency, how speakers and writers manage to use language strategically to affect audiences; and how audiences and readers, agents in their own right, manage, ...
6 CitationsSource
#1Richard N. Matzen (UVU: Utah Valley University)H-Index: 1
#2Jeff E. HoytH-Index: 3
Since composition began as a field in the 1960s, academics who specialize in composition and literacy have described why composition instruction is inconsistent across the country in grades K through 12 and in higher education (Berlin, 1987; Bloom, Daiker, & White, 1996; Myers, 1996; Strickland & Strickland, 1993). This "writing on the wall" is larger now because of these headlines: "Writing in Schools Is Found Both Dismal and Neglected" in the April 26, 2003 New York Times (Lewin, 2003a); "Educ...
16 Citations
#1Martin Chodorow (CUNY: City University of New York)H-Index: 32
#2Jill Burstein (Princeton University)H-Index: 26
This study examines the relation between essay length and holistic scores assigned to Test of English as a Foreign Language™ (TOEFL®) essays by e-rater®, the automated essay scoring system developed by ETS. Results show that an early version of the system, e-rater99, accounted for little variance in human reader scores beyond that which could be predicted by essay length. A later version of the system, e-rater01, performs significantly better than its predecessor and is less dependent on length ...
56 CitationsSource
#1Carl Whithaus (ODU: Old Dominion University)H-Index: 3
This essay traces a distinction between computer-mediated writing environments that are tools for correcting student prose and those that are media for communication. This distinction has its roots in the influence of behavioral science on teaching machines and computer-aided writing instruction during the 1960s and 1970s. By looking at the development of the time-shared, interactive, computer-controlled, information television (TICCIT) and early human–computer interaction (HCI) research, this e...
6 CitationsSource
43 CitationsSource
#1Stuart A. SelberH-Index: 12
Multiliteracies for a Digital Age serves as a guide for composition teachers to develop effective, full-scale computer literacy programs that are also professionally responsible by emphasizing different kinds of literacies. Stuart A. Selber also proposes methods for helping students move among these literacies in strategic ways. Defining computer literacy as a domain of writing and communication, Selber addresses the questions that few other computer literacy texts consider: What should a comput...
250 Citations
#1William Condon (WSU: Washington State University)H-Index: 8
#2Diane Kelly-Riley (WSU: Washington State University)H-Index: 1
Washington State University (WSU), has developed two large-scale assessment programs to evaluate student learning outcomes. The largest, the Writing Assessment Program, diagnoses student writing abilities at entry and mid-career to determine the type of support needed to navigate the expectations of our writing-rich curriculum. The second, the Critical Thinking Project, has developed an assessment instrument, the WSU Guide to Rating Critical Thinking, adaptable by faculty to their instructional ...
75 CitationsSource
Cited By46
#1Duygu Bektik (OU: Open University)H-Index: 2
Effective written communication is an essential skill which promotes educational success for undergraduates. One of the key requirements of good academic writing in higher education is that students must develop a critical mind, and learn how to construct sound arguments in their discipline. Writing Analytics focuses on the measurement and analysis of written texts to improve the teaching and learning of writing, and is being developed at the intersection of fields such as automated assessment, ...
#1Simon Skov Fougt (Metropolitan University College)H-Index: 1
#2Amanda Siebert-Evenstone (UW: University of Wisconsin-Madison)H-Index: 2
Last. Morten Misfeldt (AAU: Aalborg University)H-Index: 8
view all 5 authors...
This paper reports on an exploratory trial of developing pedagogical visualizations of 16 students' written assignments on literary analysis using two sets of keywords and Epistemic Network Analysis (ENA). The visualizations are aimed at summative evaluation as a tool for the professor to support assessment and understanding of subject learning. Results show that ENA can visually distinguish low, middle and high performing students, but not statistically significantly. Thus, our trial provides a...
1 CitationsSource
#1Simon Knight (UTS: University of Technology, Sydney)H-Index: 9
#2Simon Buckingham Shum (UTS: University of Technology, Sydney)H-Index: 37
Last. Xiaolong Wang (UTS: University of Technology, Sydney)H-Index: 2
view all 5 authors...
Research into the teaching and assessment of student writing shows that many students find academic writing a challenge to learn, with legal writing no exception. Improving the availability and quality of timely formative feedback is an important aim. However, the time-consuming nature of assessing writing makes it impractical for instructors to provide rapid, detailed feedback on hundreds of draft texts which might be improved prior to submission. This paper describes the design of a natural la...
10 CitationsSource
#1Denise WhitelockH-Index: 20
#2Duygu BektikH-Index: 2
Large-scale assessment refers to tests that are administered to large numbers of students and are used at local, state, and national levels to measure the progress of schools with respect to educational standards. In order to have accurate and fair measurements, large-scale assessment systems need to include all available students, which means a high volume of students, with large numbers of exams to be marked. The amount of marking that is required is extensive; thus marking exams at this scale...
1 Citations
#1Duygu BektikH-Index: 2
Effective written communication is an essential skill which promotes educational success for undergraduates. Argumentation is a key requirement of successful writing, which is the most common genre that undergraduates have to write particularly in the social sciences. Therefore, when assessing student writing academic tutors look for students’ ability to present and pursue well-reasoned and strong arguments through scholarly argumentation, which is articulated by meta-discourse. Today, there are...
3 Citations
#1Todd RueckerH-Index: 5
#2Dawn ShepherdH-Index: 2
Last. Beth L. BrunkH-Index: 4
view all 4 authors...
ABSTRACTComputer technology made its way into writing instruction and assessment with spelling and grammar checkers decades ago, but more recently it has done so with automated essay evaluation (AEE) and diagnostic feedback. And although many programs and tools have been developed in the last decade, not enough research has been conducted to support or evaluate the claims of the developers. This study examined the effectiveness of automated writing instructional programs in consistent scoring of...
3 CitationsSource
#1Nancy WelchH-Index: 3
#2Tony ScottH-Index: 2
7 CitationsSource
#1Laura K. AllenH-Index: 10
Last. Danielle S. McNamaraH-Index: 50
view all 3 authors...
15 Citations
#1Elena Cotos (Iowa State University)H-Index: 8
This article aims to engage specialists in writing pedagogy, assessment, genre study, and educational technologies in a constructive dialog and joint exploration of automated writing analysis as a potent instantiation of computer-enhanced assessment for learning. It recounts the values of writing pedagogy and, from this perspective, examines legitimate concerns with automated writing analysis. Emphasis is placed on the need to substantiate the construct-driven debate with systematic empirical ev...
4 CitationsSource