Lessons of Evaluation
29th April — 3rd May, 2013 Stanislav Zaichenko and Sergey Bredikhin, researchers at the HSE Institute for Statistical Studies and Economics of Knowledge (ISSEK), took part in the Executive Education Course “Evaluation of Science and Innovation Policies” organised by the Manchester Institute of Innovation Research (MIoIR), University of Manchester, Great Britain.
Evaluation is a crucial element of science and technology policy, as it is applied to measure the effectiveness of research teams, organisations, development institutions, programmes of excellence, etc. The course, which is taught under the guidance and direct involvement of Professor Jakob Edler, Executive Director at the Manchester Business School (MBS) MioIR, and with participation of the Institute’s leading members, provides participants with an integrated view of the basic aspects and contemporary discourse around the theory, methods, and evaluation practices in the international S&T policy.
The introductory lecture by Jakob Edler summarised the key issues, practical problems, the main approaches to the evaluation at different levels of S&T policy. Kate Barker, Senior Lecturer at the MIoIR, presented an overview of the development of evaluation in policy and programme management, as well as the logic chart — the universal procedure of planning evaluation projects — and marked its theoretical basis in terms of management, politics and economics. Then the review of evaluation methods was continued by Dr. Paul Cunningham. He considered the principles for the selection and application of various indicators, taking into account their strengths and weaknesses, focusing on the need to weigh all the possibilities and limitations of the indicators used — no matter how convenient they are for analysts.
Paul Cunningham further described the role of evaluation within S&T policy framework and evaluation effectiveness criteria. A practitioner’s point of view on this topic was presented by Dr. Rosa M. Fernández, Economic Adviser at the UK Department for Business, Innovation and Skills, who provided the course participants with information on current system of financing R&D in the UK and application of evaluation tools for allocating public funds among programmes and institutions in the field of S&T. The lecture by Dr. John Rigby, Senior Research Fellow at the MIoIR, was focused on the basic principles, functions, key concepts and directions of bibliometric analysis, its practical application as well as the known limitations. Such an important and interesting aspect of evaluation process as the study of additionalities was described in the presentation of Jakob Edler and Abdullah Gök, Research Fellow at the MIoIR. The goal here was to model the alternative scenarios of the policy objects’ behaviour (new tools, programmes and reforms) in the hypothetical situation of lack of the policy’s impact. Under this approach, policy objects are not seen as "black boxes", but as a set of behaviour models that can be analysed.
Kate Barker’s lecture on the so-called Research Assessment Exercise and the Research Excellence Framework demonstrated the principles of functioning and effects of the university evaluation system established in the UK. Then three lectures were delivered by Dr. Maria Nedeva, Senior Lecturer at the MIoIR . In the first one the emphasis was placed on the activities of agencies that fund research projects. In the second lecture Maria Nedeva introduced the effects of management decisions on social phenomena. In the third lecture she described the logic of peer review process, which is one of the key elements of any evaluation exercise, and the main problems of expert panels’ work and implementation of its results. A rare opportunity to learn the details of such a large-scale project as the Czech Research Audit was provided by an invited speaker Dr. Erik Arnold, Managing Director at Technopolis in the UK and Chairman of the Technopolis Group. After describing evaluation techniques selected for this project, he outlined the problems faced by the audit team, and submitted practical recommendations, drawn from the results of the study.
Two lectures were presented by Professor Philip Shapira, a long-time partner of ISSEK Foresight Center. He devoted the first one to the evaluation of economic impacts of research, technology and innovation programmes, and introduced the basic concepts and methods of measuring them. Then he addressed the question of evaluation of relatively new tools of science and innovation policy — the so-called innovative "X-Prize". The course participants were able to fully realise the possibilities of such an unusual instrument of policy, as well as the problems of assessing the results of it, in the format of a business game.
The final lecture was presented by Jakob Edler. He summarised the material covered previously and listed the principles, issues and approaches used in evaluating various types of public policy, gave an overview of the demand-based policy instruments and challenges emerging in the process of evaluation of this kind of policy measures. After that, five groups of the course students had an opportunity to present the results of their work on small evaluation projects, which were defined at the beginning of the course, and discuss their findings with the lecturers.
The material has been prepared by Stanislav Zaichenko and Sergey Bredikhin