A case study of the issues arising as lecturers use computer-aided assessment in mathematics modules


CETL-MSOR Conference 2012
12-13 July 2012
University of Sheffield, Sheffield


Recent research has developed a wealth of knowledge on the development of computer-aided assessment (CAA) systems (e.g. Greenhow et al. 2003, Pidcock et al. 2004, Sangwin 2007) and CAA has become a useful tool for lecturers to assess and give immediate feedback to large cohorts of students.

This case study examines the use of a CAA system using the QuestionMark Perception software at a UK university with a strong engineering tradition. This system has been used with around 1000 mathematics and engineering first-year undergraduates each year since the start of its development in 2002.

Though it would appear that the system is firmly established as an efficient teaching and learning resource, it has not been without problem. For example, large group sizes have precluded invigilating the online tests. Also, the questions were developed to test students’ ability to carry out common mathematical procedures, so their conceptual knowledge is not tested to the same extent.

Lecturers have responded to issues such as these and, today, a diverse range of practices aimed at mitigating these issues has emerged. Questionnaires completed by nine lecturers at the target institution highlighted stark differences in approach: from non-invigilated online tests taken by students at the time and location of their choosing, to replacement paper tests that are invigilated and incorporate questions designed to assess conceptual understanding.

Six lecturers took part in follow up interviews that sought to address the following questions:

  • What issues do lecturers identify with the use of CAA?
  • How are lecturers responding to these issues?
  • What impact will this have on their future use of CAA?

The interviews suggest that these lecturers are experiencing further issues while using CAA. For example, the paper tests delay the return of feedback and require lecturer time to mark. As for the non-invigilated test, the lecturers have no means to monitor the work that students do and have gathered anecdotes of students using mathematical software, groups of friends or prepared solutions in order to gain high marks in the tests.

Further changes have been made in order to mitigate the effect of these issues too. Some lecturers have attempted to develop new CAA questions that seek students’ conceptual understanding; and some lecturers have reduced the allocation of module marks given to CAA to minimise the advantages gained by abusing the system.

This case study aims to serve as a starting point from which to discuss good practice with computer-aided assessment. Moreover it will outline future plans at this university for better addressing some of the issues concerned.