Comparability

Thursday, 04 March 2010 16:36

Examination standards are what our country’s education system — and those who work within it — are judged against. For the system to be fair to all, examinations need to be comparable between different versions, different awarding organisations, different years and different subjects. To stimulate discussion on comparability we commissioned a book, Techniques for Monitoring the Comparability of Examination Standards, which examines the techniques used to monitor comparability over the past half century.

Range and scope of comparability studies

We undertake comparability studies for a wide range of qualifications, based on the evaluation of assessment practices and student work, and have recently commenced a programme of work comparing qualifications offered in other countries to those available to students in England, Northern Ireland and Wales. We also carry out studies that compare the same qualification in different years to make sure that GCE and GCSE standards are maintained over time and across different awarding organisations.

International Comparisons

In an age when learners cross international boundaries to study it is important that the qualifications we regulate are comparable with those studied by learners in other high-performing countries, since UK learners will often be competing with international learners for higher education and employment opportunities.

In 2010 Ofqual began a rolling programme to look at the demand of assessments and qualifications internationally. In the first study we are comparing the demand of pre-university qualifications available to candidates in England, Wales and Northern Ireland, against a range of qualifications available to similar learners in other parts of the world. As part of the study, reviewers from a range of backgrounds will be asked to analyse a cross-section of all the qualifications in the study over the next few months.

The summary report of our findings was published in May 2012.

In February 2011 we published a progress report describing the context and parameters of the study, lists the comparator jurisdictions taking part and outlines the progress made to date.

In November 2010 we published the forms that are being used by reviewers for comment. You can download the draft forms below:

Overview of the proposed methodology

ICoSSE A Form Generic

ICoSSE B Form Generic

ICoSSE C Form Generic

ICoSSE CRAS Form Generic

GCSE, AS and A levels

Our inter-subject comparability studies are conducted by subject experts with a background in assessment. They analyse candidate work and assessment materials across two or more related subjects, drawing comparisons and highlighting differences.

The methodology and approach of four studies covering GCSE, AS and A levels is explained here. You can also see the forms used for the analysis and the outcomes of Study 1a: GCSE, AS and A level geography and history; Study 1b: GCSE, AS and A level sciences; Study 2a: A level biology, psychology and sociology; and Study 2b: A level English literature, history and media studies.

GCSEs in vocational subjects have also been reviewed.

Further compatibility studies for GCSE French and the first award of the short GCSE course in citizenship studies have been undertaken. Another review took place to determine whether the IGCSE was suitable for use in state maintained schools in England. A similar study examined the comparability between GCE and the International Baccalaureate.

Whilst a degree of consistency in examinations is necessary, it is equally important that exam papers are not predictable. To monitor this, we worked with a group of independent consultants — examining three GCSEs and six GCEs to identify where predictability might become an issue. You can view the report here.

The following forms have been used to gather information for the standards over time review GCE/GCSE/:

CRAS - Analysis record form

Form A - Factual analysis of syllabuses, question papers and mark schemes

Form B - Differences in demand between qualifications: Syllabuses, question papers and mark schemes

Form C - Summary of judgements

Form P - Progression between levels within a single subject in 2009

National Vocational Qualifications (NVQs)

Our studies of NVQ assessment practice looked at the consistency and quality of the assessment practices associated with each qualification. Awarding organisations were asked to respond and indicate how they intended to address any highlighted issues of concern in units and qualifications for Lifting and Transferring units; Call handling operations; BTEC National qualifications; Personal licence holder qualifications; and Door supervision qualifications.

Qualification monitoring reports (VQs)

These reports examine the monitoring taking place in relation to vocational qualifications — including National Vocational Qualifications and GCSEvs.

The following two reports scrutinise the grading of GCSEvs; looking at whether students are being awarded lower grades than appropriate and exploring the factors that may result in apparent differences in attainment.

Performance in GCSE examinations in vocational subjects (GCSEvs) 2004-5: CA report

Performance in GCSE examinations in vocational subjects (GCSEvs) 2004-5: Appendix E

The report monitoring NVQs in Administration examines the testing around the implementation of an assessment strategy for these awards and the effectiveness of the quality assurance in relation to their assessment and verification across the awarding organisations concerned.

National Curriculum Research

We commissioned a report into into different models of assessment for home languages (the official language of a country or state) for children aged 11-17. We looked at Australia, Canada, Hungary, Italy, New Zealand, Northern Ireland, Scotland, Spain and Sweden.

Evaluation of assessor and verifier awards (A1, A2, V1)

Assessor and verifier awards are qualifications for people involved in assessing and verifying others. They certify that assessors are competent to assess candidates and their performance — using a range of methods — and that verifiers are proficient in conducting internal and external quality assurance of the assessment process. You can view an evaluation of the awards here.

Last Updated on Thursday, 10 May 2012 16:39 Tags: assessment, assessors, Comparability, evaluation, EVs, external verifiers, GCEs, GCSEs, GGSEvs, internal verifiers, monitoring, NVQs, predictability, quality assurance, verifiers