Good news: Work Experience works

Six weeks ago I wrote here and here that:

"the debate on work experience seemed almost entirely divorced from the evidence of whether work experience actually improved the employment opportunities of jobless young people. "

and

"DWP [the Department for Work and Pensions] should attempt to produce a proper counterfactual analysis that would allow us to come to a considered judgement on the programme's success. "

DWP has now done just that, publishing today an impact assessment of the Work Experience programme.  The analysis compares participants on the work experience programme with "comparable" (as determined by sophisticated statistical techniques) non-participants. 



The details are highly technical, but the key findings are:

"estimates suggest that WE had a strong and beneficial impact on the likelihood of a participant receiving benefit compared to the impact if they had not participated - participants became much less likely to be on benefit than non-participants; the central estimate is -6 percentage points by week 13"

and 

"work experience had a strong and beneficial impact on the likelihood of a participant being in employment compared to non-participants - participants were more likely to be in employment than non-participants: the central estimate reached +8 percentage points."

Here is a chart illustrating the first point:

A team here at NIESR, led by David Wilkinson and including me, performed a peer review of the analysis. As the DWP publication says:

"NIESR concluded that the methodology was sound, and that the key findings and conclusions - in particular the finding that work experience had a significant impact reducing benefit receipt and increasing employment for participants, compared to otherwise comparable non-recipients – appeared robust."

This is excellent news, for two reasons. First, it shows that work experience works.  I argued in my original article:

"My view - shared by most of those who study and evaluate such programmes - is that work experience is likely to improve outcomes for young people, albeit not as much as implied by Ministers' claims. " 

This is precisely what the impact assessment says. The impacts are not huge: 16 weeks after starting the programme, 46 percent of participants are off benefit, compared to 40 percent of comparable non-participants; 35 percent are recorded as being in employment, compared to 27 percent of non-participants (the figures for employment are probably somewhat less reliable, for data reasons).  However, the impact is very clearly positive, and given that the programme is relatively cheap, it is almost certainly a worthwhile investment. 

Second, DWP - both those who commissioned this work, and the extremely professional and competent analysts who  performed it -  should be congratulated on performing a detailed and robust impact analysis, subjecting it to outside scrutiny, and publishing the results. Of course, this should happen, where feasible (and it usually is in some form) for all major social programmes; and it is a pity it took a political row to get us here.  Nevertheless, they did the right thing, and they deserve credit. 

What, then, should we conclude from the recent debate about the effectiveness or otherwise of work experience?

First, the question I asked above "whether work experience actually improves the employment opportunities of jobless young people?" has been answered; it does. As our review suggested, and as the impact analysis rightly acknowledges, there is much more analysis that could be be done, and many important issues and questions remain; but in my view it is highly unlikely to overturn this basic point. Critics of the programme should accept this, and move on to, as the Commission on Youth Unemployment suggested, consider how it could be improved. 


Second, these events show that not only do Ministers have a responsibility to ensure that policies are properly evaluated, and that the debate takes place on the basis of the best available evidence, but the government would actually be better off if they did so.  If DWP had never published this initial, highly selective and hence misleading, analysis, then the debate about work experience would have been hugely more positive; a good programme would not have been tarnished in public because the lack of proper evidence meant both sides were able to make misleading claims. 

Third, the fact that the actual impact of the programme is much smaller than implied by Ministers' claims at the time - but at the same time that it is clearly a good and cost-effective programme - should lead to a greater sense of perspective about the impact of policies.  Most social programmes don't help everyone who participates - either because their outcomes remain poor despite the programme, or because they would have had a good outcome anyway. 


That is clearly true here; the estimate suggests that despite the work experience programme, 54% of participants were still on benefit afterwards; and of the 46% who left benefit, 40% would have anyway.  This implies that about  86% of the observed impact is "deadweight"; this, of course, was precisely the argument that the government used to justify the abolition of the Educational Maintenance Allowance (EMA).  


As I and a number of other economists pointed out at the time, this was a misuse of the evidence in order to construct a spurious argument against EMA; equally, it would be spurious to use the fact that most of those leaving benefit are "deadweight" to criticise the work experience programme.  It is the costs and benefits of programmes that matter. It is to be hoped that when using today's analysis, as I am sure the government will (and indeed should) to justify the programme, that Ministers will recognise that by the same token the arguments they made to justify the decision to abolish EMA were misleading, and that this will not happen again. 


Finally, Ministers should admit it when they get the facts wrong. I noted six weeks ago that the Prime Minister's statement, at Prime Minister's Questions, that of those young people going on to the programme

"around half of them are actually getting work at the end of these schemes."

was simply false (Sarah Teather subsequently made the same mistake).  As far as I know, the Prime Minister has not corrected this error, even though it was extensively analysed by Fullfact.  Now that a government publication has set out the facts clearly, it is to be hoped that the Prime Minister will comply with the Ministerial Code, which states

"It is of paramount importance that Ministers give accurate and truthful information to Parliament, correcting any inadvertent error at the earliest opportunity."

I think this is important, as a signal not only of respect for the Code, but for the accurate and proper use of evidence in the public policy debate.