Enter keywords:

Isabel Nisbet at FAB conference 2008

14 October 2008

Ricoh Arena, Coventry, UK

Isabel Nisbet talks about the role of Ofqual and innovation in assessment at the annual conference of the Federation of Awarding Bodies.

Transcription

Good morning, and it’s a pleasure on behalf of Ofqual to join you today.  We’ve been very pleased indeed to be one of the sponsors of this important conference.  I think it’s the second one that we’ve sponsored since we were born, and we’re hoping this is going to go rather better.  The first one we did was the Plagiarism Conference, and every delegate that came in was given a raffia bag to take away that said, “Ofqual Plagiarism” in enormous letters on it, and we felt somewhat ashamed to take them anywhere after that, so it’s at the bottom of my suitcase at the moment. 

I’d also bring greetings from Kathleen Tattersall, our Chair.  One of the first public presences that Kathleen had during the summer of results was at the VQ day on 23rd July, and it was very important for her to do that first before the GCSE and A Level results.  And she visited a couple of FE colleges and also talked to Ministers about that event, and it was a very important part of our setting up.  So we’re very pleased to be here, we’re pleased to be in this interesting venue.  I did a bit of prep by going onto the Coventry City website, including the fan Oggy Oggy Bloggy, which I half-heartedly recommend to you.  [Laughter]. 

What I’m going to do is to say a little bit about Ofqual, particularly about how our roles and those of the QCA are developing to be distinct in areas which we know are of particular interest to you, and I’m going to talk particularly about qualifications, developments and criteria, a little bit about the market, a bit about technology and innovation and, with terror, I’m going to try and use the ask the audience buttons but we’ll see how we get on.  And then, to end up with, some challenges for all of us, including challenges for Ofqual.  Some of you may remember that two years ago speaking from QCA I suggested some challenges that we’re all facing, and we’ve revisited those two years later to suggest what new challenges we need to look at.  All of these slides will be on our website but also the FAB website, so there’ll be a chance for you to take the challenges away.

So quickly, a little bit about Ofqual.  The main thing about Ofqual, which was the reason that Ministers set it up, was to be independent.  To be independent of Government, so that we’re not part of the aim to try to improve achievement, what we are is part of the standard which ensures that that achievement is meaningful and reliable.  We’re also independent of the QCA, that the QCA develops qualifications and it does work in curriculum development, we are the ones who ensure that the quality is met, and we do that with a distinct and separate role.  Ofqual will report to Parliament rather than to Ministers once the legislation goes through to set us up, and we’re there to ensure that standards are secure in exams, qualifications and tests, covering all qualifications that are regulated.  Ed Balls announced our conception, if that’s the right word, almost exactly a year ago at the Labour Party Conference, and a year on we’re now in place still within the QCA but with a distinct role and a distinct voice. 

We were launched in shadow form in April of this year when Kathleen was appointed as our Chair, and the Government is intending to legislate in the forthcoming session to set up Ofqual and the QCDA in their new roles.  Meantime however, we are acting now as a distinct and separate voice in qualifications within QCA.  Now that requires a little bit of fleetness of foot but everybody, including many people in this room, have indicated that that’s what they want us to do, and I believe that our effect will depend on what difference we make in the next year, most of which will be still within QCA. 

And we, like the QCA eventually, are based in Coventry.  Our headquarters is in Coventry Business Park, which is about eight miles or so from here.  I personally moved there last March, March of this year, we’ve now got fifty staff based in Spring Place, and by the end of May all of Ofqual will be there and there’ll be no Ofqual posts based in London.  And that’s a big change for us, and it’s also an opportunity for us to be a distinctive educational voice in the West Midlands, which is what we want to be.  We had a public launch, in the Motorcycle Museum for some obscure reason, not that far from here in the pouring rain in May, and from then it’s always gone uphill. 

Our mission as a regulator is similar to that of the regulatory task of QCA.  We make no objection about that because our mission continues.  It isn’t to set up some fancy new organisation, it is to do these things:  to regulate awarding bodies, qualifications, exams and tests; to ensure that the qualifications market is fit for purpose, coming back to that, that qualifications are fair, that standards are secure, that public confidence is sustained and that Ofqual acts as a public champion of the learner.  There’s continuity there from the regulatory role of QCA and our job is to do that.  At the same time we have to set up a new Ofqual, but our eye is on that mission.  And we have strategic priorities which were set for Ofqual by the QCA Board and endorsed by the committee that now oversees Ofqual. 

And they’re these.  Only the last one is to establish and run an effective new organisation, the others are all the core of the day job, about maintaining standards, about recognising organisations that are fit to be in the national system and making sure that they’re effective.  And, as many of you know, our aim in regulation is to move up the strategic levels to be much more concentrating on making sure that organisations can control their own quality and much less ticking of the detail by the regulator.  And that strategic direction will continue.  To secure an effective and efficient qualifications market, and I’ll be coming back to that.  To act in the public interest when things go wrong, whether it’s malpractice or complaints about some of the awarding bodies, or even a failure in delivery in the system, and we’ve been working very hard over this summer to deal with a major and very, very worrying failure in the delivery of National Curriculum tests.  And also, the last one, to establish and run an effective independent interim regulator, that’s Ofqual, within QCA. 

And this is the way I want you to see us, and the challenge to me is that if you feel we’re not acting in this way you need to come up to us in the street and say, “Just a minute, that’s not how you’re supposed to behave.”  We want to be visibly independent from the Government and from the QCA.  If it looks to you that we’re too cosy, and we were accused by the Select Committee recently when Kathleen and I were before them, they said, “Aren’t you too cosy up with the QCA at the moment?” and we have to answer that challenge to show that we’re being visibly independent and speaking our minds.  That we’re about standards and confidence, that we’re at the end of the day rooting for the learner.  So if you look back for some of the statements that Kathleen or I made during the summer about the National Curriculum tests, the first thing we were saying was what the effect of these problems were on the students who were sitting the tests.  That must be our starting point. 

Regulating a fair and effective market.  That market  regulation role is at the heart of what Ofqual does.  That all our pronouncements are evidence-based.  As an independent regulator we should reject the temptation to make trendy statements that are based on hunch.  This is about evidence, and where there isn’t evidence we will say so.  So, for example, we were pushed and pushed to be able to say that the marking of the National Curriculum tests was rubbish.  Now what had happened was there were enormous problems with the delivery of the tests but there was no evidence to say that the marking was rubbish so we said that, and we’ll always act on evidence. 

That we will be respected and trusted by those whom we regulate.  And Kathleen has done a lot of work in the last few months to make it clear that she expects a relationship of respect and trust with awarding bodies, but also with the Government, who will not see us as part of their delivery chain but as an equal partner with a distinct role, and by the wider public.  That we’re open and transparent.  We have a new publicity protocol which means that a lot of our letters and publications are immediately on the website, and I’ll be telling you about one of those that is coming on today.  And also that we want to be a leading member of the educational community in this part of the country.  So that’s how we want you to see us, not a small task. 

Now, in our distinctness from QCDA, as QCA has been developing into its development agency role, one of the most difficult things to unpick, and something that you’ve asked me to talk about, is about our respective roles in developing criteria for qualifications and leading onto their accreditation.  And a lot of good work’s been done on this and Teresa Bergen, who’s here with me today, will be able to talk a bit further with you over lunch about that as well.  But this is some headlines about it, about what the emerging roles are. 

The particular thing is that the common criteria for all qualifications and particular criteria for particular groups of qualifications are regulatory instruments.  They’re approved and adopted by the regulators.  That’s Ofqual and our fellow regulators in Wales and Northern Ireland.  The QCDA will develop criteria, will work them up for some qualifications that they’re particularly asked to do, and those particularly would include a lot of 14-19 qualifications for example.   The three regulators, that’s us and Wales and Northern Ireland, would adopt these criteria when we are content with them, and there will be a genuine process for deciding that.  The criteria development process will include points where QCDA will check with the regulator that the emerging criteria are going to be regulatable, because there’s no point in going through a huge amount of work with many people in this room then at the end of the day the regulator sitting on her high horse and saying, “No, no, we don’t agree with those.”  You want to have a clear idea from the beginning of what the regulator’s requirements are going to be.

But the decision by the regulators to adopt criteria is a substantive one based on a thorough consideration of QCDA’s proposals.  And the same would go for any proposals by Government.  We cannot be directed by Government to adopt these criteria.  This is an independent decision by the regulator based on evidence and our independent judgement.  If the regulator decides that the criteria that QCDA develops need to be revised then, rather than becoming another QCDA and doing it all again ourselves, the idea is that this would go back to QCDA to do further work.  And that’s the theme, the development side is QCDA, so a lot of the discussions that you’ll be having at the development stage will be with QCDA but there’ll be key points at which the regulator will say these are the criteria, these are the standards that we’ll be applying when we make our decisions on these.  Yes, we think this is regulatable, there may be problems with that so you’d better think about this.  And we hope that that’s going to make what might be a complicated position a bit easier.

Ofqual accredits qualifications as well as doing a lot of important monitoring work, but we don’t develop them.  And the qualifications are actually developed by you from awarding bodies, those of you from awarding bodies, with the support of many others like QCDA, from the Sector Skills Councils or other organisations.  When the awarding body has a full draft qualification that you believe if you’re an awarding body meets the criteria, the regulator’s criteria, you’ll then submit it to Ofqual, and that’s what’ll happen.  And what Ofqual will do is make a decision on how we’re going to do, make the accreditation decision based on risk.  All regulators are encouraged to be proportionate and to focus their activity where the greatest risk is, and clearly qualifications which are new, are going to affect large amounts of people, are high stakes, involve young people at key stages of the National Curriculum, those kind of things would be risk indicators and we may well ask for reports on these proposals from the relevant experts development body.  So for a lot of 14-19 qualifications it would be QCDA, so we would ask QCDA for a report on this proposal, and then if we felt in the light of that report and other consideration by us that the qualification didn’t meet our regulatory criteria, we would then feed it back to the awarding body.

So that’s broadly how the roles are divided up, both for development and accreditation.  Now I know that that sounds a bit theological at the moment, but I’d encourage you over these two days to take the chance to think it through a little, do talk to people from both organisations that are here, and I hope that it’ll enable it to become a little bit clearer for you. So that’s about qualifications development and accreditation.  I want to move on now to the other part of how I, looking back to how we want to be seen, this idea of regulating and fair and effective market.  This is a market in the production of qualifications and their sale to centres and to learners. 

First of all, as a market regulator, Ofqual is determined to develop an even better understanding of how markets work, including how the money flows, the drivers for various activities by not just on the supply side but also on the demand side, so that we can regulate it more effectively and also avoid unintended side effects of any of our interventions.  Our baseline is a willingness to allow the market to work.  There must be a reason for stepping in, and we would never step in unless there was a reason for our intervention.  Now one of the things we’ve done to strengthen our position in all of this work is we have appointed a new Director of Regulation for Ofqual, and she is Fiona Pethick who’s sitting at that table there, and Fiona’s going to be here today so please knobble her with alacrity.  Fiona came to us from being a Director from Ofwat, the water regulator.  She has an economic and statistical background, is an expert in working with organisations on price mechanisms and on market regulation.  And Fiona’s appointment is a signal to you that we take this seriously and we’re going to be professional about it.  There were some concerns that we were amateurs dabbling in markets.  This is a serious part of what Ofqual does and we’ve got an expert to help us to do it.

We have remits from Government.  With Ofqual they’re a bit embarrassed about giving us remits so they’re sort of agreements I think Government would call them, to look at two things; to look at the efficiency of the qualifications market, and to look at fees, and particularly we’ve already looked at fees for A Levels and our report on those is on the website, on the Ofqual website, we’ve been asked to look at the fees for some other widely used qualifications.  Now there’s been quite a lot of exchanges, including some very useful input from FAB, on these two remits and one or two headlines I want to give you today as, particularly looking at efficiency, we’ve had a first meeting of the steering group, which Paul was at, and because of the importance of this work Kathleen is chairing this steering group herself.  What was made clear is that we’re doing this work in phases and we’re not just looking at the efficiency of the supply side of qualifications, in other words the production of them by awarding bodies and the processes around those, but we’re also looking at efficiency on the demand side as you asked us to do. 

You asked us to look at are schools efficient and colleges in the way they use qualifications.  Is it efficient for a bright pupil to do ten GCSEs?  It may not be, it may be over-consumption of the product.  So we are looking at all of those aspects and not just at some crude idea about fees for qualifications.  But that work is going on, it is of high importance, the fees work is part of the implementation of FE White Paper, and we’re pressing on with those and we’re doing it in a collaborative and transparent way.  The other thing that we’re committed to be is vigilant for anti-competitive behaviour, and that means not just vigilant among awarding bodies but particularly vigilant among the wider world in which you work.  And I can give you my assurance this morning, if we see behaviour, whether it’s by Government, QCA or Sector Skills Councils, that calls into question the ability of you to take part in a fair and competitive market, then from our independent point of view we will say so, and we’ll say so publicly.   The other side of it is that it behoves all of us to work together for the good of the learner, and that’s the bottom line, but we will be clear that there are dangers of anti-competitive behaviour in quite a lot of this.  Discussions and collaboration are one thing, pushing out market entrants and competitors is another, and we’re clear about the difference.

Another example, which is a more specific one of our work on markets, all of you from awarding bodies have had a communication from us about the effect of the withdrawal of the £10 levy on NVQs which has now been withdrawn.  Hooray I should say, it’s about time too.   But what as a regulator we would expect was the benefit of this to pass on to the schools and colleges and learners and not to be absorbed, and we’ve asked all of you to show us that that’s what you’re doing.  This is an important thing.  You might say it’s only a small part of the system, but it’s a symbolic one.  I know that many awarding bodies have treated that levy in different ways in your cost profiles, but the beneficiaries should be the schools and the colleges and the learners, so we hope we can see that that happens. 

Now, I’m going to move onto technology, and this is the terrifying moment when I’m going to ask you to do the ask the audience bit.  But just before you press anything, I’d just like to say a little bit about Ofqual’s role in relation to technology.  It’s a challenge to use to be on the side of innovation.  Yesterday I was present, as Fiona was, at part of the rather elaborate induction course that we’ve proudly developed for our new cohort of recruits to Ofqual, and that was a morning taken by Professor Alison Wolf from King’s College London talking about regulation, and she was saying that regulators are by definition opposed to innovation.  They’re about standardisation and rules and caution and risk aversion, and it’s really a challenge to them to be sufficiently flexible to promote innovation.  And one of the issues that’s challenging us at the moment, and one of the things we’ve been acting on recently, is how to try to make the world we regulate one that supports and promotes greater use of technology. 

And so what I want you to do is to pick up the voting doodahs and to have a look at – can I ask, the first question, can I ask those who are not awarding bodies not to answer this, you can do the next one but, if you’ll forgive me.  So just the awarding bodies this time.  Now, the question I’m asking you is why don’t you use technology more than you do to deliver your qualifications?  And I’ve got six possible reasons there.  Now anybody who comes from Ofqual or from parts of some awarding bodies will immediately say that these are not valid and that they duplicate and they’re not reliable and that you don’t want to vote for any of them, so could you put that bit of you aside and just pretend that you think that the answers are okay.  I should say, trying to do staff appraisals with a group of assessment regulators is a disaster.  [Laughter]

But these are some of the reasons that are given to us.  One is the cost of the use of technology, and I mean the use of technology more both in marking and in assessment.  Another is that our regulations, Ofqual’s regulations are thought not to allow this in the ways that you would want to do it.  That it’s not your skill-set as an awarding body.  That your customers for the particular skills and knowledge that you’re testing don’t want that or it’s not appropriate for them.  That you don’t know what technology you can use, or you’re rejecting the question saying you already make extensive use of technology to deliver your qualifications.  So can I now ask for the audience to say which they want?  Done.  Okay, so what have we got?  Cost is the first one, not appropriate for our customers, and then quite a lot saying you already make extensive use of technology, which is an interesting one and I want to come back to that in a second. 

Can I now go onto the second one, and this is for everybody, not just for awarding bodies.  And here we have some negative arguments, some arguments against greater use of e-assessment, this is particularly about e-assessment in qualifications.  Now which of these do you think is the strongest?  One is that it would encourage too much use of multiple choice questions.  Another is that the technology might not work.  There’s not enough educational know-how in the world of e-assessment.  It’s too expensive for users.  The public or your customers will not have confidence in it, or that e-assessment can’t test practical skills.  Do you want to have a go at those?  And again with apologies to those who would reject all six.  So any joy with that one?  Here we are.  Can’t test practical skills, which I expected to be quite a big part of it, then I would say that there are really innovative ways in which technology can test some practical skills but I do understand the issues there.  And encourage too much use of multiple choice questions is another one, and I want to come back to that as well. 

Thank you for those.  Those are very important issues because some of those are in danger of landing us with an assessment and qualification system that is almost a generation behind the curriculum system, whether for vocational or other qualifications.  Can we move on?  Thanks very much. 

Now, one of the first things Ofqual has done is we’ve sent this week an open letter to all awarding bodies, and this is the first of a series, and the idea of these open letters is that we send them to you and then they’re going to be on, immediately on our website so that everybody can see them.  And with luck, if the technology works, you should have that back in your office when you go back after this conference.  And what’s it saying?  The first is to reassure you on one point; it is saying that Ofqual is not going to be requiring a single national technology platform for delivering assessments, because we do know from you that some of you have been standing back from investing because what happens if the regulators then impose some other technology that you haven’t invested in.  But we will encourage you to operate interoperable systems. 

If you’re worried about clashes with our codes of practice, particularly detailed points, don’t assume the worst, come and ask.  Some of these codes of practice were written before a lot of the developments in technology now, and we’ve had examples of awarding bodies who’ve said oh well, we’ve assumed from page 22 line 16 that we couldn’t possibly have assessment when ready using technology.  Come and ask, because the fact is we may decide that it’s in the learners’ interests and the interests of a good qualification system to enable you to do that and to support you. 

The technology should be the servant of good assessment, not the other way round, and it’s up to people in this room to say to the techies in the e-world we want good quality assessment, don’t tell us all we can do is multiple choice, there’s no reason to say that.  The technology must deliver the assessment that you and your clients need to have good, modern, sophisticated, sensitive assessment of the skills that you want to develop.  Coming from medical regulation there’s some really exciting examples in diagnostic testing which are not about multiple choice questions, they’re about working through a diagnosis, doing various kinds of interventions, including scans and x-rays, and seeing the results and working it through.  Now that’s good quality assessment, it’s not simple multiple choice. We will promote good practice and stimulate innovation.  If you feel that we’re doing the opposite you must challenge us. 

Now, talking about challenges and to finish with, I said to you that I wanted to return to some of the challenging questions for the various groups of us in this room and revisit some of the ones that I talked about two years ago and bring them up to date.  So we thought it was best for me to start with some challenges for ourselves.  Challenges that I believe that you could legitimately ask to ours.  And again you’ll see those on the version of this that you see on the website.  One of the questions which we need to answer is this one.  How can we really be independent when we’re still part of QCA, and the story I told about the criteria roles.  It was quite difficult to work through, how can we really be independent?  And the answer is that we’ve got to try, and I today can tell you that I have the strong support of Ken Boston and Tony Greener from QCA in doing that.   We’re encouraged by them to develop our role and develop our independent voice, and they’re supporting us in doing that.  But if you think we’re too cosy you’ve got to ask us.

Are we really implying the same standards to all awarding organisations, including employers, or because there’s a policy drive to get more employers into the system are we letting them through with easier criteria?  Now the answer is that we are quite clear that we are applying the same standards, but if you believe that we’re not you must challenge us.  How can we assure standards in such big new things as principal learning in diplomas when they haven’t been piloted yet?  And this is a big, big issue for our standards side, and we’re working very closely with those of you who are working in diplomas to try to make sure that we can do that in the best way we can in the first year.  But some things will go wrong in the first year, and you’ve got to have patience, and as an evidence-based regulator you’ve got to encourage Ministers and the public to have patience to allow it to work through. 

Aren’t our new rules too complicated?  All this stuff about strategic regulation that I’ve been saying.  You’ll see on our excellent stand which I hope you’re all going to visit, the final version of the regulatory criteria for the QCF, and also the diploma monitoring rules.  Now you may feel that they’re too complicated.  We’ve tried to move them up a level and set out the principles of what we’re trying to achieve, but if you feel they’re too complicated still challenge us, because it’s actually more difficult to write simple rules than it is to write complicated rules, so we may not have got that quite right yet.  Couldn’t we trust awarding bodies more, and certainly Kathleen Tattersall’s very clear that our baseline for our relationship with you should be one of trust.  Couldn’t we be less risk averse, because we seem to panic when there’s anything new at all.  And the last one for us is do we really believe in three-country regulation?  And the answer is that we do.  We believe we’ve got the message from learners and from employers that that’s what they want from us.  There are national initiatives that are pushing the countries apart and it falls to Ofqual as a national regulator to make clear that everybody knows it may be right to different directions but the cost of that would be a reduction in three-country regulation, which our customers say that they expect us to produce.

So those are some challenges for us.  I’m quickly going to go through those, but I know we’ve got some centres and employers here.  How much do centres, schools, colleges, employers, know about what they get for the money they spend on qualifications?  A lot of complaints that it’s too much.  On our website we have the report on A Level fees, and it’s got one of those little pie charts that says what each bit is spent on.  How much do you really know about that?  Or is it like my council tax when all I know is it’s too much but I don’t really know what I get for it. 

Is the awarding body burden which we still talk about really support and help, and how can you as a centre be part of making sure that you’re getting the right kind of support?  How can your learners have confidence in the qualifications you offer?  Now I would say to you that the quality assurance provided by an independent regulator is part of that.  And what can you do to be more of a discerning and demanding customer and purchaser of the qualifications you offer?  I sometimes talk to exam officers who say oh, we don’t like the systems that awarding bodies have.  Have you ever said that?  No.  You know.  And it just seems to be a poor, not a strong purchaser in that respect.  So how can the centres be more demanding purchasers?  That may make life more difficult for the awarding bodies but I think for most of you that’s what you would actually want. 

And lastly, here’s some questions which I’ll put up together, and I’m hoping that you may want to take those away and look at them on the web.  Why can’t awarding bodies trust each other more as we’re moving into this area of reform?  Are we really, are you really willing to lend some of your own property to make the qualifications and credit framework a success?  Now I know that here’s been very good work led by FAB to try and make that a reality, a commercial reality for awarding bodies, but at root there is a choice there.  There is a bit of sacrifice of individuality for the sake of the gain of being part of a flexible national system.  Can you converge your administrative systems more than you do for the schools and colleges who do find the differences very difficult to cope with?  Why do some of you mind that we are going to publish the certification data for your particular qualifications?  Those of you who offer GCSEs and A Levels have done that for years.  There was an issue about cleansing the data which turned out to be full of rubbish, we’ve cleansed it rather slowly but it’s now cleansed and it’s ready to be published, why do you some of you mind about that?  Most producers of products would expect that’s just a normal part of commercial life.

The 14-19 strategy, you’re going to be hearing from Jeremy Benson about that tomorrow.  Now this is a challenge, that’s been the Government being a demanding purchaser and saying we’re going to purchase things in particular suites.  How as a community are you going to respond for that about, instead of wishing it wasn’t there?  How are you going to respond as a producer to this kind of new demanding customer and make sure that you get the best out of that system?  And what can you do to improve your relationship with SSCs?  I’ve made it clear that we’re going to speak out against anti-competitive behaviour but it takes two to make a relationship.  And I have to say at this point that we in Ofqual very warmly welcomed John’s appointment, I think it’s an enormous step forward in building this relationship of equals, and I think those of you who are in awarding bodies who work with the SSCs should hold your head up and feel that you’re part of a relationship of equals.  Now these are all challenges to you but you can certainly say that we’re quite busy with the challenges for ourselves as well. 

So, my conclusions.  We as independent regulator want a relationship with awarding bodies that’s based on clarity, fairness and mutual trust.  We’re working out our role boundaries with QCDA, and you’ve heard an example of that about the criteria and the accreditation.  But we’ll continue to do that, and if you’re unclear please ask.  Our values and objectives are clear, we’ve set out how we want you to see us.  If you feel we’re not that please come back and challenge us.  We want to stimulate you to innovate more, particularly in the use of technology in your qualifications, and some of the issues that came up in the question and answer illustrations of that.  And lastly, we’re very proud to be based in Coventry, come and visit us, it’s really quite a nice place.  Thank you very much.  [Applause].