Reducing Variation in the Assessment of Student Writing.

Published on Jan 1, 2011in Assessment & Evaluation in Higher Education2.473
· DOI :10.1080/02602930903215842
Kerry Hunter3
Estimated H-index: 3
(UTS: University of Technology, Sydney),
Peter Docherty5
Estimated H-index: 5
(UTS: University of Technology, Sydney)
This paper extends the literature on grader variation and the role of moderation and socialisation processes in reducing this variation. It offers a fresh categorisation of academics’ assessment beliefs and expectations, and uses this categorisation to analyse the interaction between implicit and explicit expectations in relation to grader variation and socialisation processes. A combination of quantitative and qualitative data from a large class grading moderation process was used to identify the existence and character of residual implicit grader expectations, the impact of these implicit expectations in causing grader variation, and the effectiveness of socialisation in reducing the extent of variation. An effective strategy for designing and focusing socialisation processes is also outlined.
  • References (20)
  • Citations (27)
📖 Papers frequently viewed together
124 Citations
101 Citations
238 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Kerry John Wimshurst (Griffith University)H-Index: 7
#2Richard Keith Wortley (Griffith University)H-Index: 22
Last. Troy John Allard (Griffith University)H-Index: 11
view all 4 authors...
This paper situates the topic of student assessment and the moderation of assessment within a broader context of policy debates about the quality of teaching and learning in universities. The focus and discussion grew out of a research project that aimed to investigate factors related to academic success and failure in a Faculty of Arts. The study, initially, identified a range of student demographic and biographical factors significantly related to academic success and failure. However, there w...
21 CitationsSource
124 CitationsSource
337 CitationsSource
For many years universities around the world have sought to articulate the nature of the education they offer to their students through a description of the generic qualities and skills their graduates possess. Despite the lengthy history of the rhetoric of such policy claims, universities' endeavours to describe generic attributes of graduates continue to lack a clear theoretical or conceptual base and are characterized by a plurality of view‐points. Furthermore, despite extensive funding in so...
427 CitationsSource
#1Berry O’Donovan (Oxford Brookes University)H-Index: 10
#2Margaret Price (Oxford Brookes University)H-Index: 18
Last. Chris Rust (Oxford Brookes University)H-Index: 17
view all 3 authors...
In recent years there has been an increasing emphasis in higher education on the explicit articulation of assessment standards and requirements, whether this emanates from calls for public accountability or is based on ideas of good educational practice (Ecclestone, 2001). We argue in this article that a single‐minded focus on explicit articulation, whilst currently the dominant logic of higher education, will inevitably fall short of providing students and staff with meaningful knowledge of sta...
202 CitationsSource
This paper reflects on the experiences of the authors in planning and teaching a short-course in academic literacy for students enrolled in the first year of an education degree. By conceptualising tertiary literacy as a social practice and drawing on a sociocultural approach to learning, the members of the project team were able to move beyond deficit views of individual students towards a consideration of their own teaching practices and how they could best help students expand their literate ...
30 Citations
This article reviews the findings of a small‐scale investigation into the criteria used by a number of SACWG departments for assessing final‐year project modules in business and history and other written history assignments. The findings provide the basis for a broader discussion of the issues relating to the formulation and use of assessment criteria. Assessment entails academics making professional judgements about the standards and quality of students' work. However, for the educational value...
68 CitationsSource
#1Chris Rust (Oxford Brookes University)H-Index: 17
#2Margaret Price (Oxford Brookes University)H-Index: 18
Last. Berry O’Donovan (Oxford Brookes University)H-Index: 10
view all 3 authors...
427 CitationsSource
#1Richard HigginsH-Index: 2
#2Peter HartleyH-Index: 2
Last. Alan SkeltonH-Index: 12
view all 3 authors...
This article reports the initial findings of a 3-year research project investigating the meaning and impact of assessment feedback for students in higher education. Adopting aspects of a constructivist theory of learning, it is seen that formative assessment feedback is essential to encourage the kind of 'deep' learning desired by tutors. There are a number of barriers to the utility of feedback outside the sphere of control of individual students, including those relating to the quality, quanti...
488 CitationsSource
In at least one fundamental respect, our universities are failing to clarify what they expect of students and what students and other stakeholders can expect of them. This failure concerns communication skills. The present paper identifies particular problems of academic literacy and proposes a coordinated institutional approach to their solution. Standards of oral and written communication among those who enter universities are often deplored by academics. Standards of oral and written communic...
12 CitationsSource
Cited By27
#1Stephen R. George-Williams (Monash University, Clayton campus)H-Index: 1
#2Mary-Rose Carroll (Monash University, Clayton campus)
Last. Tina Overton (Monash University, Clayton campus)H-Index: 12
view all 5 authors...
AbstractVariation in marks awarded, alongside quality of feedback, is an issue whenever large-scale assessment is undertaken. In particular, variation between sessional teaching staff has been studied for decades resulting in many recorded efforts to overcome this issue. Attempts to curtail variation range from moderation meetings, extended training programmes, electronic tools, automated feedback or even audio/video feedback. Decreased marking variation was observed whenever automated marking w...
Abstract In writing assessment, the inconsistency of teachers’ scorings is among the frequently reported concerns regarding the validity and the reliability of assessment. The study aimed to find out to what extent participating in a community of assessment practice (CAP) can impact the discrepancies among raters’ scorings. Adopting one group pretest-posttest design, patterns in the teachers’ scoring judgments were explored based on both quantitative and qualitative data. The results indicate si...
1 CitationsSource
#1Adam Aitken (UTS: University of Technology, Sydney)H-Index: 2
#2Darrall Thompson (UTS: University of Technology, Sydney)H-Index: 8
First year undergraduate design students have found difficulties in realising the standards expected for academic writing at university level. An assessment initiative was used to engage students with criteria and standards for a core interdisciplinary design subject notable for its demanding assessment of academic writing. The same graduate attribute categories linked to assessment criteria and web-based software (REVIEW™) were used for assessing students’ other design assignments. Students eng...
#1Whitney Alicia Zimmerman (PSU: Pennsylvania State University)H-Index: 3
#2Hyun Bin Kang (PSU: Pennsylvania State University)H-Index: 1
Last. Fan Zhang (Penn State College of Information Sciences and Technology)H-Index: 1
view all 7 authors...
ABSTRACTOver two semesters short essay prompts were developed for use with the Graphical Interface for Knowledge Structure (GIKS), an automated essay scoring system. Participants were students in an undergraduate-level online introductory statistics course. The GIKS compares students' writing samples with an expert's to produce keyword occurrence and links in common scores which can be used to construct a visual representation of an individual's knowledge structure. Each semester, students respo...
3 CitationsSource
#1Susan Bloxham (University of Cumbria)H-Index: 13
#2Birgit den-Outer (Oxford Brookes University)H-Index: 1
Last. Margaret Price (Oxford Brookes University)H-Index: 18
view all 4 authors...
Unreliability in marking is well documented, yet we lack studies that have investigated assessors’ detailed use of assessment criteria. This project used a form of Kelly’s repertory grid method to examine the characteristics that 24 experienced UK assessors notice in distinguishing between students’ performance in four contrasting subject disciplines: that is their implicit assessment criteria. Variation in the choice, ranking and scoring of criteria was evident. Inspection of the individual con...
35 CitationsSource
#1Sally O’Hagan (University of Melbourne)H-Index: 4
#2Gillian Wigglesworth (University of Melbourne)H-Index: 23
Assessment is a crucial factor in higher education where marks gained can determine future study and career options. Increasing student numbers, and an increasing proportion of international students, raises concerns regarding marking practices, and whether the same criteria are used to mark both native-speaker (NS) and non-native-speaker (NNS) essays. This paper examines the essays of five native and five non-native speakers written in English for the same undergraduate subject. The essays were...
11 CitationsSource
#1Jinrong Li (GS: Georgia Southern University)H-Index: 4
#2Peggy J. Lindsey (GS: Georgia Southern University)H-Index: 2
Abstract While rubrics have their limitations, many studies show that they can clarify teacher expectations, and in comparison to a simple score or a letter grade, provide more information about the strengths and weaknesses of students’ writing. Few studies, however, have explored the variations between students’ and teachers’ readings of rubrics and how such differences affect student writing. This article describes the findings of a mixed-methods research study designed to identify discrepanci...
10 CitationsSource
This paper presents an effort to create a unified model for conducting and assessing undergraduate dissertations, shared by all disciplines involved in computer game development at a Swedish university. Computer game development includes technology-oriented disciplines as well as disciplines with aesthetical traditions. The challenge has been to create a unified process and assessment procedure while maintaining the individual disciplines’ academic standards. The unification has been achieved th...
3 CitationsSource
#1Jeffrey Chi Hoe Mok (NUS: National University of Singapore)H-Index: 1
#2Anita Ann Lee Toh (NUS: National University of Singapore)H-Index: 1
Purpose – This paper aims to investigate the use of blind marking to increase the ability of criterion-referenced marking to discriminate students’ varied levels of knowledge and skill mastery in a business communication skills course. Design/methodology/approach – The business communication course in this study involved more than 10 teachers and 350 students each semester. Data were collected from four semesters of assignment grades to compare the distribution of grades in semesters that used b...
2 CitationsSource
In higher education, assessment integrity is pivotal to student learning and satisfaction, and, therefore, a particularly important target of continuous quality improvement. This paper reports on the preliminary development and application of a process of recording and analysing current assessment moderation practices, with the aim of identifying areas in need of improvement. Specifically, survey software was used to create a record keeping strategy in which unit coordinators documented the asse...
3 CitationsSource