Match!

Progress and Challenges for Automated Scoring and Feedback Systems for Large-Scale Assessments

Published on Feb 13, 2018
Denise Whitelock19
Estimated H-index: 19
,
Duygu Bektik1
Estimated H-index: 1
Abstract
Large-scale assessment refers to tests that are administered to large numbers of students and are used at local, state, and national levels to measure the progress of schools with respect to educational standards. In order to have accurate and fair measurements, large-scale assessment systems need to include all available students, which means a high volume of students, with large numbers of exams to be marked. The amount of marking that is required is extensive; thus marking exams at this scale requires a lot of work, which means a high volume of exam scripts need to be marked by tens of thousands of examiners appointed by the exam boards. The need for large-scale assessments and the high cost of manual marking and limited “turn around” time have led to developments, over some years, of automated assessment and marking. This chapter reviews the history and development of automated assessment systems. It includes findings from empirical research as well as highlights the theoretical considerations that emerge from such developments. In addition, the practical aspects of developing such assessments are explored with examples primarily from the UK and USA, including the systems and tools available, the current capabilities of natural language processing (NLP) approaches, and their limitations, ethical concerns, and future potential.
  • References (43)
  • Citations (1)
References43
Newest
Published on May 26, 2017
Duygu Bektik1
Estimated H-index: 1
Effective written communication is an essential skill which promotes educational success for undergraduates. Argumentation is a key requirement of successful writing, which is the most common genre that undergraduates have to write particularly in the social sciences. Therefore, when assessing student writing academic tutors look for students’ ability to present and pursue well-reasoned and strong arguments through scholarly argumentation, which is articulated by meta-discourse. Today, there are...
Published on Mar 1, 2017in Nature Human Behaviour
Rose Luckin1
Estimated H-index: 1
‘Stop and test’ assessments do not rigorously evaluate a student's understanding of a topic. Artificial intelligence-based assessment provides constant feedback to teachers, students and parents about how the student learns, the support they need and the progress they are making towards their learning goals.
Published on Jul 29, 2015
Eric Ras13
Estimated H-index: 13
,
Denise Whitelock19
Estimated H-index: 19
,
Marco Kalz1
Estimated H-index: 1
The chapter introduces the current state of the art of technology for E-Assessment for Learning. A large amount of literature is reviewed with respect to formative assessment, feedback and E-Assessment for Learning. Three case studies introduced single innovative approaches of assessment for learning. These include details of OpenEssayist which is a real-time learning analytics tool using natural language processing techniques to give students feedback on draft essays. A mobile notification syst...
Published on Jun 22, 2015
Denise Whitelock19
Estimated H-index: 19
(OU: Open University)
This paper discusses the outcomes from the building and empirical investigation of two automatic feedback systems, namely OpenMentor and OpenEssayist that can support student learning. The findings from OpenMentor and OpenEssayist usage suggest that prompt targeted feedback for time poor students can maximise student success. Both systems facilitate the users to take ownership and reflect on their own work, through provision of feedback at a point where they can build on it in subsequent tasks. ...
Published on Mar 16, 2015
Denise Whitelock19
Estimated H-index: 19
(OU: Open University),
Alison Twiner12
Estimated H-index: 12
(OU: Open University)
+ 2 AuthorsStephen Pulman28
Estimated H-index: 28
(University of Oxford)
This paper focuses on the use of a natural language analytics engine to provide feedback to students when preparing an essay for summative assessment. OpenEssayist is a real-time learning analytics tool, which operates through the combination of a linguistic analysis engine that processes the text in the essay, and a web application that uses the output of the linguistic analysis engine to generate the feedback. We outline the system itself and present analysis of observed patterns of activity a...
Published on Jan 1, 2015
Einari Kurvinen4
Estimated H-index: 4
,
Rolf Lindén4
Estimated H-index: 4
+ 1 AuthorsMikko-Jussi Laakso11
Estimated H-index: 11
Published on Mar 31, 2014
Danielle S. McNamara50
Estimated H-index: 50
,
Arthur C. Graesser77
Estimated H-index: 77
(University of Oxford)
+ 1 AuthorsZhiqiang Cai3
Estimated H-index: 3
(ASU: Arizona State University)
Coh-Metrix is among the broadest and most sophisticated automated textual assessment tools available today. Automated Evaluation of Text and Discourse with Coh-Metrix describes this computational tool, as well as the wide range of language and discourse measures it provides. Section I of the book focuses on the theoretical perspectives that led to the development of Coh-Metrix, its measures, and empirical work that has been conducted using this approach. Section II shifts to the practical arena,...
Published on Jan 1, 2014
Denise Whitelock19
Estimated H-index: 19
,
Deborafield7
Estimated H-index: 7
+ 2 AuthorsNicolas Van Labeke8
Estimated H-index: 8
This paper reports the findings of an empirical investigation, which set out to test a set of rainbow essay exercises. The rainbow diagrams are pictorial representations of formal graphs that are derived automatically from student essays. They were designed to allow students to discover how key concepts in a well written essay are connected together. The students would then be able to compare a rainbow diagram of their own essay with a good essay and make changes to it before submission to their...
Published on Jan 1, 2014
Danielle S. McNamara50
Estimated H-index: 50
,
Arthur C. Graesser77
Estimated H-index: 77
+ 1 AuthorsZhiqiang Cai3
Estimated H-index: 3
Published on Jan 1, 2014
Danielle S. McNamara50
Estimated H-index: 50
,
Arthur C. Graesser77
Estimated H-index: 77
+ 1 AuthorsZhiqiang Cai3
Estimated H-index: 3
Cited By1
Newest
Published on Oct 1, 2018in Technology, Knowledge, and Learning
Mary Webb15
Estimated H-index: 15
('KCL': King's College London),
Doreen Prasse + 7 AuthorsHans Laugesen
In this article, we identify and examine opportunities for formative assessment provided by information technologies (IT) and the challenges which these opportunities present. We address some of these challenges by examining key aspects of assessment processes that can be facilitated by IT: datafication of learning; feedback and scaffolding; peer assessment and peer feedback. We then consider how these processes may be applied in relation to the assessment of horizontal, general complex 21st cen...
View next paperTechnology for Large-Scale Assessment