e-Marking
Last updated on: 12/03/2008
Background
Prior to the recent innovation in the use of electronic systems, the process for marking GCSE and A level exam scripts had changed relatively little for a significant number of years.
In the conventional marking process:
- Examiners are standardised by senior colleagues regarding the application of a mark scheme to its associated question paper.
- Examiners receive an allocation of whole exam scripts by post, which they mark by hand using a paper copy of the mark scheme, and annotate where relevant.
- Examiners submit their marked scripts to the awarding body by post, and a senior examiner checks a number of these to ensure that the mark scheme has been applied appropriately.
- Examiners complete marking sheets by hand for their allocation, which detail the total mark achieved by each script; they submit these by post to the awarding body where they are processed.
- Senior examiners consider a range of annotated hard copy scripts in determining the specific marks at which grades will be awarded.
Developing electronic marking
Each of the unitary awarding bodies addressed the need to modernise the system for marking exam scripts by developing an electronic marking process. They submitted proposals to us, giving details of the design features of their intended processes, as well as outlining their plans for trialling and adopting them.
The electronic marking processes developed by the unitary awarding bodies contained a number of common features:
- Hard copy original scripts are scanned at dedicated centres, with each item response identified separately and stored as an image on a secure website. (Note: an item may be a whole question or a designated part of a question, depending on its size.)
- Examiners are standardised by senior colleagues on the application of a mark scheme to its associated question paper, though not necessarily for every item.
- Examiners access the secure website and mark a number of candidate responses to a selected item; they view an image of the candidate response on-screen, and use an on-screen version of the mark scheme.
- Examiners enter the mark achieved by a candidate response on-screen immediately after marking it and the marks are processed automatically.
- Marking is monitored through the use of items that have been marked by a senior examiner to exemplify the marking standard; examiners must mark a number of these to the required standard before marking a given item, and again at regular intervals in the course of marking the item
- Senior examiners, in determining the marks at which grades will be awarded, either: consider a range of unannotated hardcopy scripts reassimiliated from the website; consider a range of unannotated scripts online using computer terminals.
Aside from these common features, there are some variations between the electronic marking processes developed. A brief description of the main characteristics of each of the awarding body processes is provided below.
AQA
AQA uses the DRS e-Marker® system. This system contains a number of separate processes related to electronic marking and electronic mark capture, of which the most sophisticated is CMI+. The CMI+ process enables the division of marking into expert, general and computer-assisted expert marking strands. Expert and general marking each follow the common process detailed above, although expert marking involves subject specialists while general marking involves non-subject specialists because items marked in this way are short and factual. Computer-assisted expert marking is notably different from these processes and is used only for items requiring a single-word or number response. At a data bureau, all the candidate responses to a given item are entered onto a database, being double-keyed for accuracy. The senior examiner responsible for the question paper then views every response offered together with its frequency. The computer then allocates marks to those responses that have been deemed worthy of credit.
Edexcel
Edexcel uses the ePen system. Students sit a traditional exam paper using handwriting and pens. Scripts are sent to a central scanning facility where each question is digitised. Markers, trained to the same standard as traditional markers, are now trained to specialise on a number of questions, rather than marking entire papers. This allows for general markers to mark the lower value questions on a paper and the experienced subject experts to mark higher value questions. The most senior examiners on the assessment team determine whether the questions should be marked by general or expert markers. All markers receive the same training and all markers are subject to real-time quality control as they unknowingly mark questions that have been pre-marked by the senior team. Any mismatches trigger alerts, and markers can quickly be retrained or removed from marking.
OCR
OCR uses the DOMS (Digital Object Marking System). With this system senior examiners attend a standardisation set-up meeting but the standardisation of other examiners occurs online rather than at a dedicated meeting. All examiners are subject specialists and mark whole candidate scripts, although the marks awarded to candidates’ responses for each question are entered separately.
Completed electronic marking
The unitary awarding bodies are at different stages with the implementation of electronic marking processes.
AQA
Following successful trial activities beginning in November 2003, AQA used electronic marking and capture for 2.4 million scripts across 80 question papers in the June 2005 examination series and again used e-Marker® to mark and electronically capture marks from 3.6 million scripts across 180 question papers including 1.6 million imaged scripts in the June 2006 examination series
Edexcel
As part of a steady increase in electronic marking since this was first trialled in November 2003 and with an increase of 3% on 2005, Edexcel marked approximately 3 million scripts electronically, across 227 question papers, in the June 2006 examination series. Edexcel will mark 90% of GCSEs and A levels (3.8 million) scripts onscreen across its GCSEs and A levels this summer. And when Edexcel's other qualifications are included, the number of questions being marked across the summer series totals over 200 million.
OCR
Following offline trials in April and December 2005, OCR undertook a live trial in March 2006. OCR then expanded electronic marking in the June 2006 examination series. OCR has a fully e-assessed GCSE - Environment and Land Based Science.
Remote standardisation
During the summer 2006 examination series both Edexcel and OCR conducted trials in standardising examiners remotely, rather than requiring them to attend a face-to-face standardisation meeting.
Edexcel trialled remote standardisation in three of the GCSE specifications being marked electronically, with the examiners who took part representing 25% of the examiners marking each paper. There was no standardisation meeting held for these examiners and they were instead standardised onscreen remotely. However, all the usual aspects of a standardisation meeting were replicated during this process: an explanation of the standardisation process, guidance on how to apply the mark scheme and the marking of a number of candidate responses sufficient to consolidate a common understanding.
OCR piloted onscreen standardisation in units of four GCSE specifications featuring electronic marking. With this system senior examiners attend a standardisation set-up meeting but the standardisation of examiners occurs on-line rather than at a dedicated meeting. Examiners are provided with a rationale explaining how the mark scheme should be applied and standardised onscreen through the marking of practice and standardisation scripts that have already been selected and marked by the principal examiner and team leaders. Training on the software used is provided to examiners on a CD-Rom.
AQA will trial remote standardisation in two components in summer 2007. Senior examiners will attend a pre-standardisation meeting. Examiners will then be standardised remotely using software accessed via the internet.
All three systems allow examiners to raise queries with a more senior examiner. The awarding bodies which trialled remote standardisation in summer 2006 found that it had no detrimental effect on the accuracy of marking and both Edexcel and OCR plan to extend remote standardisation to more subjects in future examination series.