Progress and Challenges for Automated Scoring and Feedback Systems for Large-Scale Assessments

Published on Feb 13, 2018
Denise Whitelock20
Estimated H-index: 20
Duygu Bektik2
Estimated H-index: 2
Large-scale assessment refers to tests that are administered to large numbers of students and are used at local, state, and national levels to measure the progress of schools with respect to educational standards. In order to have accurate and fair measurements, large-scale assessment systems need to include all available students, which means a high volume of students, with large numbers of exams to be marked. The amount of marking that is required is extensive; thus marking exams at this scale requires a lot of work, which means a high volume of exam scripts need to be marked by tens of thousands of examiners appointed by the exam boards. The need for large-scale assessments and the high cost of manual marking and limited “turn around” time have led to developments, over some years, of automated assessment and marking. This chapter reviews the history and development of automated assessment systems. It includes findings from empirical research as well as highlights the theoretical considerations that emerge from such developments. In addition, the practical aspects of developing such assessments are explored with examples primarily from the UK and USA, including the systems and tools available, the current capabilities of natural language processing (NLP) approaches, and their limitations, ethical concerns, and future potential.
  • References (46)
  • Citations (1)
📖 Papers frequently viewed together
1 Author (R.E. Bennett)
15 Citations
3 Authors (Lester Gilbert, ..., Veronica Gale)
9 Citations
3 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Duygu BektikH-Index: 2
Effective written communication is an essential skill which promotes educational success for undergraduates. Argumentation is a key requirement of successful writing, which is the most common genre that undergraduates have to write particularly in the social sciences. Therefore, when assessing student writing academic tutors look for students’ ability to present and pursue well-reasoned and strong arguments through scholarly argumentation, which is articulated by meta-discourse. Today, there are...
3 Citations
#1Rose Luckin (UCL: University College London)H-Index: 11
‘Stop and test’ assessments do not rigorously evaluate a student's understanding of a topic. Artificial intelligence-based assessment provides constant feedback to teachers, students and parents about how the student learns, the support they need and the progress they are making towards their learning goals.
4 CitationsSource
#1Eric RasH-Index: 13
#2Denise WhitelockH-Index: 20
Last. Marco KalzH-Index: 2
view all 3 authors...
The chapter introduces the current state of the art of technology for E-Assessment for Learning. A large amount of literature is reviewed with respect to formative assessment, feedback and E-Assessment for Learning. Three case studies introduced single innovative approaches of assessment for learning. These include details of OpenEssayist which is a real-time learning analytics tool using natural language processing techniques to give students feedback on draft essays. A mobile notification syst...
8 Citations
#1Denise Whitelock (OU: Open University)H-Index: 20
This paper discusses the outcomes from the building and empirical investigation of two automatic feedback systems, namely OpenMentor and OpenEssayist that can support student learning. The findings from OpenMentor and OpenEssayist usage suggest that prompt targeted feedback for time poor students can maximise student success. Both systems facilitate the users to take ownership and reflect on their own work, through provision of feedback at a point where they can build on it in subsequent tasks. ...
4 CitationsSource
#1Denise Whitelock (OU: Open University)H-Index: 20
#2Alison Twiner (OU: Open University)H-Index: 12
Last. Stephen Pulman (University of Oxford)H-Index: 29
view all 5 authors...
This paper focuses on the use of a natural language analytics engine to provide feedback to students when preparing an essay for summative assessment. OpenEssayist is a real-time learning analytics tool, which operates through the combination of a linguistic analysis engine that processes the text in the essay, and a web application that uses the output of the linguistic analysis engine to generate the feedback. We outline the system itself and present analysis of observed patterns of activity a...
22 CitationsSource
#1Einari KurvinenH-Index: 5
#2Rolf LindénH-Index: 6
Last. Mikko-Jussi LaaksoH-Index: 12
view all 4 authors...
3 Citations
#2Arthur C. Graesser (University of Oxford)H-Index: 77
Last. Zhiqiang Cai (ASU: Arizona State University)H-Index: 2
view all 4 authors...
Coh-Metrix is among the broadest and most sophisticated automated textual assessment tools available today. Automated Evaluation of Text and Discourse with Coh-Metrix describes this computational tool, as well as the wide range of language and discourse measures it provides. Section I of the book focuses on the theoretical perspectives that led to the development of Coh-Metrix, its measures, and empirical work that has been conducted using this approach. Section II shifts to the practical arena,...
158 Citations
#1Denise WhitelockH-Index: 20
#2DeborafieldH-Index: 8
Last. Nicolas Van LabekeH-Index: 8
view all 5 authors...
This paper reports the findings of an empirical investigation, which set out to test a set of rainbow essay exercises. The rainbow diagrams are pictorial representations of formal graphs that are derived automatically from student essays. They were designed to allow students to discover how key concepts in a well written essay are connected together. The students would then be able to compare a rainbow diagram of their own essay with a good essay and make changes to it before submission to their...
14 Citations
#1Mark D. ShermisH-Index: 15
#2Jill BursteinH-Index: 26
Carl Whithaus, Foreword. M. D. Shermis, J. Burstein, S. A. Bursky, Introduction to Automated Essay Evaluation. N. Elliot, A. Klobucar, Automated Essay Evaluation and the Teaching of Writing. S. C. Weigle, ESL Writing and Automated Essay Evaluation. J. Burstein, J. Tetreault, N. Madnani, The E-rater(R) Automated Essay Scoring System. P. W. Foltz, L. A. Streeter, K. E. Lochbaum, T. K Landauer, Implementation and Applications of the Intelligent Essay Assessor. M. T. Schultz, The Intellimetric(TM) A...
79 Citations
#1Danielle S. McNamara (ASU: Arizona State University)H-Index: 50
#2Scott A. Crossley (GSU: Georgia State University)H-Index: 29
Last. Rod D. Roscoe (ASU: Arizona State University)H-Index: 19
view all 3 authors...
The Writing Pal is an intelligent tutoring system that provides writing strategy training. A large part of its artificial intelligence resides in the natural language processing algorithms to assess essay quality and guide feedback to students. Because writing is often highly nuanced and subjective, the development of these algorithms must consider a broad array of linguistic, rhetorical, and contextual features. This study assesses the potential for computational indices to predict human rating...
68 CitationsSource
Cited By1
#1Mary Webb ('KCL': King's College London)H-Index: 16
#2Doreen PrasseH-Index: 1
Last. Hans LaugesenH-Index: 1
view all 10 authors...
In this article, we identify and examine opportunities for formative assessment provided by information technologies (IT) and the challenges which these opportunities present. We address some of these challenges by examining key aspects of assessment processes that can be facilitated by IT: datafication of learning; feedback and scaffolding; peer assessment and peer feedback. We then consider how these processes may be applied in relation to the assessment of horizontal, general complex 21st cen...
2 CitationsSource