Designing Academic Writing Analytics for Civil Law Student Self-Assessment

Published on Mar 1, 2018
· DOI :10.1007/s40593-016-0121-0
Simon Knight9
Estimated H-index: 9
(UTS: University of Technology, Sydney),
Simon Buckingham Shum37
Estimated H-index: 37
(UTS: University of Technology, Sydney)
+ 2 AuthorsXiaolong Wang2
Estimated H-index: 2
(UTS: University of Technology, Sydney)
Research into the teaching and assessment of student writing shows that many students find academic writing a challenge to learn, with legal writing no exception. Improving the availability and quality of timely formative feedback is an important aim. However, the time-consuming nature of assessing writing makes it impractical for instructors to provide rapid, detailed feedback on hundreds of draft texts which might be improved prior to submission. This paper describes the design of a natural language processing (NLP) tool to provide such support. We report progress in the development of a web application called AWA (Academic Writing Analytics), which has been piloted in a Civil Law degree. We describe: the underlying NLP platform and the participatory design process through which the law academic and analytics team tested and refined an existing rhetorical parser for the discipline; the user interface design and evaluation process; and feedback from students, which was broadly positive, but also identifies important issues to address. We discuss how our approach is positioned in relation to concerns regarding automated essay grading, and ways in which AWA might provide more actionable feedback to students. We conclude by considering how this design process addresses the challenge of making explicit to learners and educators the underlying mode of action in analytic devices such as our rhetorical parser, which we term algorithmic accountability.
Figures & Tables
  • References (50)
  • Citations (10)
📖 Papers frequently viewed together
1 Author (Duygu Bektik)
8 Citations
2016LAK: Learning Analytics and Knowledge
20 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
Apr 25, 2016 in LAK (Learning Analytics and Knowledge)
#2Ágnes Sándor (Xerox)H-Index: 10
Last. Mindy McWilliams (Georgetown University)H-Index: 3
view all 6 authors...
When used effectively, reflective writing tasks can deepen learners' understanding of key concepts, help them critically appraise their developing professional identity, and build qualities for lifelong learning. As such, reflecting writing is attracting substantial interest from universities concerned with experiential learning, reflective practice, and developing a holistic conception of the learner. However, reflective writing is for many students a novel genre to compose in, and tutors may b...
20 CitationsSource
#1Nicholas Diakopoulos (UMD: University of Maryland, College Park)H-Index: 1
Every day automated algorithms make decisions that can amplify the power of businesses and governments. Yet as algorithms come to regulate more aspects of our lives, the contours of their power can remain difficult to grasp. This paper studies the notion of algorithmic accountability reporting as a mechanism for elucidating and articulating the power structures, biases, and influences that computational artifacts exercise in society. A framework for algorithmic power based on autonomous decision...
145 CitationsSource
#1Duygu Simsek (OU: Open University)H-Index: 4
#2Ágnes Sándor (Xerox)H-Index: 10
Last. Denise Whitelock (OU: Open University)H-Index: 20
view all 6 authors...
When assessing student essays, educators look for the students' ability to present and pursue well-reasoned and strong arguments. Such scholarly argumentation is often articulated by rhetorical metadiscourse. Educators will be necessarily examining metadiscourse in students' writing as signals of the intellectual moves that make their reasoning visible. Therefore students and educators could benefit from available powerful automated textual analysis that is able to detect rhetorical metadiscours...
6 CitationsSource
#1David Boud (UTS: University of Technology, Sydney)H-Index: 58
#2Romy Lawson (UOW: University of Wollongong)H-Index: 8
Last. Darrall Thompson (UTS: University of Technology, Sydney)H-Index: 8
view all 3 authors...
Can extended opportunities for self-assessment over time help students develop the capacity to make better judgements about their work? Using evidence gathered through students' voluntary self-assessment of their performance with respect to assessment tasks in two different disciplines at two Australian universities, the paper focuses on the effects of sequences of units of study and the use of different types of assessment task (written, oral, analysis, and project) in the development of studen...
42 CitationsSource
#2Arthur C. Graesser (University of Oxford)H-Index: 77
Last. Zhiqiang Cai (ASU: Arizona State University)H-Index: 2
view all 4 authors...
Coh-Metrix is among the broadest and most sophisticated automated textual assessment tools available today. Automated Evaluation of Text and Discourse with Coh-Metrix describes this computational tool, as well as the wide range of language and discourse measures it provides. Section I of the book focuses on the theoretical perspectives that led to the development of Coh-Metrix, its measures, and empirical work that has been conducted using this approach. Section II shifts to the practical arena,...
187 Citations
#1David Boud (UTS: University of Technology, Sydney)H-Index: 58
#2Romy Lawson (JCU: James Cook University)H-Index: 8
Last. Darrall Thompson (UTS: University of Technology, Sydney)H-Index: 8
view all 3 authors...
One of the implicit aims of higher education is to enable students to become better judges of their own work. This paper examines whether students who voluntarily engage in self-assessment improve in their capacity to make those judgements. The study utilises data from a web-based marking system that provides students with the opportunity to assess themselves on each criterion for each assessment task throughout a programme of study. Student marks were compared with those from tutors to plot cha...
66 CitationsSource
#1Anthony Niedwiecki (John Marshall Law School)H-Index: 2
With the recent criticisms that law schools do not do enough to prepare students for the practice of law, law schools are beginning to develop ways to better prepare their students. These criticisms have focused on the lack of teaching students to be lifelong learners. In fact, a recent report by the Carnegie Foundation stated that law schools need to teach students to be more "metacognitive about their learning." Generally, metacognition refers to having both an awareness of and control over on...
5 Citations
#1Anthony NiedwieckiH-Index: 2
This article will detail the concept of metacognition, how current law school teaching does not teach metacognitive skills, and how legal educators can incorporate metacognitive learning into the law school curriculum to help students better transfer knowledge and skills to the practice of law. Teaching metacognitive skills to law students should focus on explaining learning theory and modeling appropriate planning, monitoring, and evaluating techniques across the curriculum.
6 Citations
#1Mark D. ShermisH-Index: 15
#2Jill BursteinH-Index: 26
Carl Whithaus, Foreword. M. D. Shermis, J. Burstein, S. A. Bursky, Introduction to Automated Essay Evaluation. N. Elliot, A. Klobucar, Automated Essay Evaluation and the Teaching of Writing. S. C. Weigle, ESL Writing and Automated Essay Evaluation. J. Burstein, J. Tetreault, N. Madnani, The E-rater(R) Automated Essay Scoring System. P. W. Foltz, L. A. Streeter, K. E. Lochbaum, T. K Landauer, Implementation and Applications of the Intelligent Essay Assessor. M. T. Schultz, The Intellimetric(TM) A...
129 CitationsSource
#1Solon Barocas (Microsoft)H-Index: 12
#2Sophie Hood (NYU: New York University)H-Index: 1
Last. Malte Ziewitz (Cornell University)H-Index: 6
view all 3 authors...
Algorithms have developed into somewhat of a modern myth. They “compet[e] for our living rooms” (Slavin 2011), “determine how a billion plus people get where they’re going” (McGee 2011), “have already written symphonies as moving as those composed by Beethoven” (Steiner 2012), and “free us from sorting through multitudes of irrelevant results” (Spring 2011). Nevertheless, the nature and implications of such orderings are far from clear. What exactly is it that algorithms “do”? What is the role a...
41 CitationsSource
Cited By10
#1Shibani Antonette (UTS: University of Technology, Sydney)
#1Antonette Shibani (UTS: University of Technology, Sydney)H-Index: 3
Last. Simon Buckingham Shum (UTS: University of Technology, Sydney)H-Index: 37
view all 3 authors...
Abstract Failing to understand the perspectives of educators, and the constraints under which they work, is a hallmark of many educational technology innovations' failure to achieve usage in authentic contexts, and sustained adoption. Learning Analytics (LA) is no exception, and there are increasingly recognised policy and implementation challenges in higher education for educators to integrate LA into their teaching. This paper contributes a detailed analysis of interviews with educators who in...
1 CitationsSource
#1Simon Knight (UTS: University of Technology, Sydney)H-Index: 9
#2Andrew Gibson (QUT: Queensland University of Technology)H-Index: 5
Last. Antonette Shibani (UTS: University of Technology, Sydney)H-Index: 3
view all 3 authors...
Abstract Learning analytics has the potential to impact student learning, at scale. Embedded in that claim are a set of assumptions and tensions around the nature of scale, impact on student learning, and the scope of infrastructure encompassed by ‘learning analytics’ as a socio-technical field. Drawing on our design experience of developing learning analytics and inducting others into its use, we present a model that we have used to address five key challenges we have encountered. In developing...
2 CitationsSource
#1Yeonji Jung (NYU: New York University)H-Index: 1
#2Alyssa Friend Wise (NYU: New York University)H-Index: 20
#1Rafael Ferreira‐Mello (UFRPE: Universidade Federal Rural de Pernambuco)H-Index: 1
#2Máverick André (UFRPE: Universidade Federal Rural de Pernambuco)H-Index: 1
Last. Cristóbal Romero (UCO: University of Córdoba (Spain))H-Index: 31
view all 5 authors...
1 CitationsSource
#1Jalal Nouri (Stockholm University)H-Index: 7
#2Ken Larsson (Stockholm University)H-Index: 5
Last. Mohammed Saqr (University of Eastern Finland)H-Index: 1
view all 3 authors...
The master thesis is the last formal step in most universities around the world. However, all students do not finish their master thesis. Thus, it is reasonable to assume that the non-completion of the master thesis should be viewed as a substantial problem that requires serious attention and proactive planning. This learning analytics study aims to understand better factors that influence completion and non-completion of master thesis projects. More specifically, we ask: which student and super...
#1Diego Dermeval (UFAL: Federal University of Alagoas)H-Index: 7
#2Josmário Albuquerque (UFAL: Federal University of Alagoas)H-Index: 3
Last. Julita Vassileva (U of S: University of Saskatchewan)H-Index: 38
view all 6 authors...
Intelligent Tutoring Systems (ITSs) are concerned with the use of artificial intelligence techniques for performing adaptive tutoring to learners' according to what they know about the domain. Researchers are increasingly interested in applying gamification in e-learning systems to engage students and to drive desired learning behaviors. However, little attention has been drawn to the effective application of gamification in ITS, and how to connect theories of both concepts in a standard and for...
#1Joan JolinstonH-Index: 11
#2Henry Phillips (Naval Air Warfare Center Training Systems Division)H-Index: 1
Last. Sean M. Fitzhugh (ARL: United States Army Research Laboratory)H-Index: 9
view all 9 authors...
Recent advances in the science of teams have provided much insight into the important attitudes (e.g., team cohesion and efficacy), cognitions (e.g., shared team cognition), and behaviors (e.g., teamwork communications) of high performing teams and how these competencies emerge as team members interact, and appropriate measurement methods for tracking development. Numerous training interventions have been found to effectively improve these competencies, and more recently have begun addressing th...
#1Ming Liu (UTS: University of Technology, Sydney)H-Index: 11
#2Simon Buckingham Shum (UTS: University of Technology, Sydney)H-Index: 37
Last. Cherie Lucas (UTS: University of Technology, Sydney)H-Index: 3
view all 4 authors...
Reflective writing is widely acknowledged to be one of the most effective learning activities for promoting students’ self-reflection and critical thinking. However, manually assessing and giving feedback on reflective writing is time consuming, and known to be challenging for educators. There is little work investigating the potential of automated analysis of reflective writing, and even less on machine learning approaches which offer potential advantages over rule-based approaches. This study ...
#1Kathryn S. McCarthy (GSU: Georgia State University)H-Index: 3
#2Rod D. Roscoe (ASU: Arizona State University)H-Index: 19
Last. Danielle S. McNamara (ASU: Arizona State University)H-Index: 50
view all 4 authors...
This study investigated the effect of incorporating spelling and grammar checking tools within an automated writing tutoring system, Writing Pal. High school students (n = 119) wrote and revised six persuasive essays. After initial drafts, all students received formative feedback about writing strategies. Half of the participants were also given access to spelling and grammar checking tools during the writing and revision periods. Linear mixed effects models revealed that essay quality for stude...