GLOSSARY AND ABBREVIATIONS
Grade B
mapped from publisher markup. A grade says how a document's structure was obtained, not how reliable the law is.
Address https://gelex.eu/eli/EU/doc/http://publications.europa.eu/resource/cellar/a75c6029-dcae-44bb-ab42-8d306dcefa7a.0006.01@2015-01-21/en
Annual Report on the European Union’s Development and External Assistance Policies and their Implementation (AR) This report is requested in the regulations establishing the financing instruments for the EU’s external assistance and has to be submitted to the European Parliament, the Council, the European Economic and Social Committee and to the Committee of the Regions. It is a report on the achievement of the objectives of each financing instrument, is based on indicators and measurements of the results delivered and the efficiency of the relevant instruments. It can be found at: http://ec.europa.eu/europeaid/annualreport-2014-europeanunions developmentandexternalassistancepoliciesandtheir_en Annual Activity Report (AAR) This report presents the achievements reached and resources used by the Directorate-General. It is the ‘mirror’ of the annual work programme. It is also a management report of the DirectorGeneral to the Commission on the performance of his duties as Authorising Officer by delegation, in the management of the EU budget. This report is also communicated to the European Parliament and the Council. It can be found at: http://ec.europa.eu/atwork/keydocuments/index_en.htm Article 318 evaluation report Article 318 of the Treaty on the Functioning of the European Union requires the Commission to ‘submit to the European Parliament and to the Council an evaluative report on the Union’s finances based on the results achieved’. The report can be found at: http://ec.europa.eu/smartregulation/evaluation/documents_en.htm Budget support This method of aid delivery involves transferring funds from an external financing agency to the partner government’s national treasury. The funds thus transferred are managed in accordance with the recipient’s budgetary procedures. Budget support programmes usually include capacity development measures to help partner countries strengthen their policymaking and management, as well as to promote dialogue on the design, implementation and results of national and sectorial policies. External Assistance Management Report EU delegations’ main reporting document to EuropeAid. It has to be made available to the European Parliament and the Council. European Investment Bank (EIB) The EIB is the EU’s bank. It is owned by and represents the interests of the EU Member States. It works closely with other EU institutions to implement EU policy. It is the largest multilateral borrower and lender by volume in the world. More than 90 % of its activity is focused on Europe, but it also supports the EU’s external and development policies. European Development Funds (EDFs) The EDFs are the main instrument for providing EU aid for development cooperation to African, Caribbean and Pacific (ACP) States and overseas countries and territories (OCTs). The partnership agreement signed in Cotonou on 23 June 2000 for a period of 20 years is the current framework for the EU’s relations with ACP States and OCTs. Its main focus is on reducing and eventually eradicating poverty. The EDFs are funded by the EU Member States, governed by their own financial regulations and managed, by the Commission, outside the EU general budget. European External Action Service (EEAS) The EEAS was launched in December 2010 under the Treaty of Lisbon. Among its activities, it prepares country and regional allocations and EU cooperation strategies jointly with relevant Commission departments, as part of the programming cycle for most external action instruments. EuropeAid Within the European Commission, the DirectorateGeneral for Development and Cooperation - EuropeAid is responsible for:
formulating EU development policy and defining sectoral policies in the field of external aid;
drawing up the multiannual programming of the external aid instruments together with the European External Action Service (EEAS);
fostering coordination between the EU and the Member States on development cooperation and externally representing the EU in this field.
Evaluation findings For the purpose of this report, this term covers conclusions, recommendations and lessons learned from an evaluation. EU delegations The EU is represented through 139 EU delegations and offices around the world which present, explain and implement EU policies (e.g. the development cooperation policy), analyse and report on policies and developments of the countries to which they are accredited, and conduct negotiations in accordance with a given mandate. Internal Audit Capability is a unit of a Commission’s directorategeneral which reports directly to the directorgeneral. Its task is to provide independent assurance on the effectiveness of the internal control system with a view to improving the directorategeneral’s operations. Financing instruments These are regulations, usually issued by the European Parliament and the Council. They provide the financing needed to support an EU policy. The EU’s development policy is supported by several financing instruments. Financial perspectives A multiannual financial planning framework which sets limits on EU expenditure. It puts the priorities set for the Union’s policies into financial terms and is at the same time an instrument of budgetary discipline and planning. The current financial perspectives cover the period from 2014 to 2020; the previous one covered the period from 2007 to 2013. OECDDAC Development Assistance Committee. A committee within the OECD (Organisation for Economic Cooperation and Development) dealing specifically with development assistance. Result This term covers:
outputs: that is produced or accomplished with the resources allocated to an intervention (e.g. grants distributed to farmers, training courses delivered to unemployed people, road built);
outcomes: the immediate changes resulting from an intervention (e.g. improved accessibility to an area due to the construction of a road, trainees who have found employment); and
impacts: longerterm socioeconomic consequences, direct or indirect, positive or negative, that can be observed a certain period after the completion of an intervention.
Within the Commission’s decentralised organisational framework, the DirectorateGeneral for Development and Cooperation — EuropeAid (EuropeAid) has set up its own results accountability framework which comprises the monitoring, evaluation and reporting of its activities. This report examines two of the key elements of this framework, namely evaluations and results oriented monitoring (ROM). Evaluation is the systematic and objective assessment of an on- going or completed programme or policy, its design, implementation and results. ROM is a standardised external review, specific to external aid, designed to look at programmes’ performance. The main purposes of these parts of the accountability framework are to improve the implementation of ongoing programmes and the design of future programmes and policies through feedback and lessons learned, and to provide a basis for accountability. The Court found that EuropeAid’s evaluation and ROM systems are not sufficiently reliable. Overall, EuropeAid’s evaluation and ROM functions are well organised but there is a lack of overall supervision of programme evaluation activities. Also, insufficient attention is paid to the efficient use of evaluation and ROM resources. The evaluation and ROM systems do not sufficiently ensure that relevant and robust findings are produced. Programme evaluation plans are based on insufficiently clear prioritisation criteria. There is no monitoring system to identify and address frequent deviations from evaluation plans. Quality control procedures are not implemented consistently for ROM and programme evaluations. The evaluation and ROM systems do not ensure that maximum use is made of findings because proper mechanisms are not in place to monitor their followup and dissemination. The evaluation and ROM systems do not provide adequate information on results achieved due to insufficiently welldefined objectives and indicators, the limited proportion of ex post evaluations and ROMs and inherent limitations in the evaluation methodology for budget support. These factors limit considerably EuropeAid’s capacity to account for the actual results achieved. The Court provides recommendations on the efficient use of evaluation and ROM resources, the prioritisation and monitoring of evaluations, the implementation of quality control procedures, the demonstration of results achieved and the followup and dissemination of evaluation and ROM findings. In the current economic and political context, there is an increased need for the Commission to ensure and demonstrate that EU funds are used in an economic, efficient and effective manner. The European Parliament and the Council have indicated a need for a clearer view of the results achieved with regard to the EU’s main objectives. Information on results achieved can only be obtained with a results accountability framework comprising three elements: monitoring, evaluation and reporting. Evaluation is the systematic and objective assessment of an ongoing or completed programme or policy, its design, implementation and results. The main purposes are to ascertain whether the objectives have been met and to formulate recommendations with a view to improving future interventions so as to support decisionmaking by the Commission, the European Parliament and the Council. Evaluation is essential for planning, designing and implementing EU policies and interventions, and enhancing transparency and democratic accountability. This is emphasised by Article 318 of the Treaty on the Functioning of the European Union, which requires the Commission to submit annually to the European Parliament and the Council an evaluation report on the Union’s finances on the basis of the results achieved (Article 318 evaluation report). As regards the first three reports produced by the Commission, in February 2014, the European Parliament pointed out that they did not provide the budgetary authority with a clear view of the real extent to which the Union’s main objectives have been achieved. The Commission’s evaluation framework is set out in a number of policy and regulatory documents:
(a) the financial regulations applicable to the EU budget;
(b) the communication on reinforcing the use of evaluation;
(c) the communication on strategic evaluation; and
(d) the communication on strengthening the foundation of Smart Regulation — improving evaluation.
Individual directorates-general are responsible for evaluating their own activities and can structure their evaluation systems to suit their needs and requirements, in compliance with Commission-defined standards concerning the organisation, the selection, the design and the use of evaluations findings. EuropeAid’s evaluation and results oriented monitoring (ROM) systems are part of its results accountability framework which is based on the policy and regulatory documents listed in paragraph 4 as well as in policy and regulatory documents more specific to the EU’s development aid:
(a) the financial regulations applicable to the EDF and the other financing instruments supporting the EU’s development policy;
(b) the Paris Declaration, the Accra Agenda for Action and the Busan Partnership Agreement;
(c) the European Consensus on Development;
(d) the communication on the ‘Agenda for Change’.
EuropeAid manages two types of evaluations, which are carried out by external contractors:
(a) strategic evaluations, which comprise geographic evaluations (covering a country or region) and thematic evaluations, which cover a sector (e.g. education, food security) or a method of aid delivery (e.g. budget support, cooperation with development banks); and
(b) programme evaluations, that can be carried out during the implementation (midterm evaluation), on completion (final evaluation) or after completion (ex post evaluation) of an intervention.
These evaluations assess performance against five criteria developed by the Development Assistance Committee of the Organisation for Economic Cooperation and Development (OECDDAC), along with the international donor community: relevance, effectiveness, efficiency, sustainability and impact. In addition, evaluations have to assess the EU’s comparative advantage and the coherence with other EU policies and programmes, other donors’ actions and partner countries’ policies. The ROM system is used by EuropeAid to monitor, on the spot, the quality of its programmes. It is based on short onsite visits to programmes by external experts who rate the programme’s performance (see Annex I) in respect of the above mentioned five OECDDAC evaluation criteria (see paragraph 8). The grading aims to provide a comparison between different programmes and for the aggregation of overall portfolio performance data. Compared to evaluations, ROM is a lighter tool which provides a quick overview of the performance of programmes. Like programme evaluations, they can be carried out during or after the implementation of programmes. EuropeAid produces the following key reports:
(a) the half yearly External Assistance Management Report (EAMR), EU Delegations’ main reporting document;
(b) the reports of EuropeAid’s directors about the management of the funds delegated to them;
(c) the Annual Activity Report (AAR) submitted by EuropeAid’s Director General to the Commissioner as well as to the European Parliament and the Council;
(d) the Annual Report (AR) on the European Union’s development and external assistance policies submitted by the Commission to the European Parliament and the Council and;
(e) the contribution to the Article 318 evaluation report submitted by the Commission to the European Parliament and the Council.
The Court examined whether EuropeAid has reliable evaluation and ROM systems. The audit focused on three main questions:
(a) Are EuropeAid’s evaluation and ROM functions appropriately organised and resourced?
(b) Do EuropeAid’s evaluation and ROM systems ensure the provision of relevant and robust findings?
(c) Do the evaluation and ROM systems ensure the maximum use of the findings made?
The audit covered the evaluation and ROM systems in place in the period from 2007 to 2013. Where adequate information was available and could be checked by the Court, it took account of changes adopted or planned by EuropeAid for the period from 2014 to 2020. The audit was carried out between November 2013 and March 2014 and included:
(a) a review of EuropeAid’s evaluation and ROM guidelines and of the main documents and processes in the management of evaluations;
(b) interviews with officials from eight units at EuropeAid;
(c) visits to the EU delegations in Benin and India, involving interviews with the delegations staff and a review of relevant documentation;
(d) a review of programming documents (see Annex II), programme financing agreements (see Annex III), strategic and programme evaluations (see Annex IV), and ROM reports (see Annex V); and
(e) a survey sent to 50 EU delegations to which 41 answers were received (see Annex VI).
The Court examined whether the evaluation and ROM systems provide for a sound distribution of responsibilities and ensure the efficient use of financial and staff resources. EuropeAid’s organisation successfully combines centralised and decentralised responsibilities. As regards centrally managed responsibilities:
(a) the Evaluation unit manages strategic evaluations, which requires it to have a comprehensive view of EuropeAid’s activities. This unit is responsible for developing the evaluation methodology and for coordinating and monitoring EuropeAid’s evaluation activities. It also has a central role in the reporting of strategic evaluations findings in the AAR and AR and in EuropeAid’s contribution to the Commission’s Article 318 evaluation report;
(b) the Quality and Results unit supervises the ROM system, including overall quality assurance common to the whole organisation, the coherence between ROM and programme evaluations, and contributes to the analysis of ROM and programme evaluation data. It provides the ROM aggregated performance ratings for the AAR and AR;
(c) ROM coordinators at EuropeAid supervise the ROM system for their respective areas of intervention. In EuropeAid’s operational units and EU delegations, ROM focal points provide support to the experts in organising field visits.
The Evaluation unit and the Quality and Results unit report directly to a deputy directorgeneral. This is a sufficiently senior position in the organisation to ensure that they receive support from senior levels of management for their roles. The management of programme evaluations is decentralised to EuropeAid’s operational units and EU delegations. This has potential benefits in terms of ensuring the takeup of evaluation results. However, programme managers have a considerable degree of autonomy in managing programme evaluations as they are responsible at all stages of the process from the initiation of the evaluations until the followup of results. In the absence of overall coordination and monitoring of this activity EuropeAid has no overview and there is little supervision by senior management. The Commission’s evaluation standards require that evaluation activities are appropriately resourced to meet their purposes. In particular, each directorategeneral must ensure that staff and financial resources are clearly identified and proportionately allocated for evaluation activities to be carried out. EuropeAid has insufficient information on the resources used by its evaluation and ROM systems. Information is available as regards the annual cost of strategic evaluations, which ranges between 5 and 6 million euro, and of ROM, which amounted to 17 million euro over the 2007–13 period. According to estimates made in 2010 and 2013 by EuropeAid, the annual cost of programme evaluations ranges between 12 and 15 million euro. The total annual cost of evaluations and ROM would thus amount to 34–38 million euro. However, as EuropeAid has no reliable information on the number of programme evaluations and since their cost cannot be easily identified in the IT management information system, the Court cannot confirm the accuracy of these estimates. There are 16 staff in the Evaluation unit and about one fulltime equivalent staff member involved in ROM at the Quality and Results unit. But there is no management information on staff resources involved in programme evaluations and ROM reviews at the level of programme managers, ROM focal points and ROM coordinators. On the basis of its survey, the Court estimates that there are approximately 60 full time equivalent staff dealing with ROM and programme evaluations in the EU delegations. The total number of full-time equivalent staff dealing with evaluations and ROM is thus likely to be well above 80. Due to the lack of reliable information EuropeAid cannot ensure that staff and financial resources are appropriate and proportionately allocated to the various evaluation and ROM activities. The Court found inadequate coordination between programme evaluations and ROMs, which is likely to lead to inefficiencies. Programme evaluations and ROMs follow completely different planning approaches and criteria (see paragraphs 26 and 30). Programmes can be over- or under-assessed due to an inappropriate combination of ROM and evaluations, or sometimes not assessed at all, as is frequently the case at the post completion stage (see paragraphs 43 to 46). This is confirmed by the Court’s survey: only half of the replies consider that programme evaluations and ROMs complement each other well. The Court examined whether evalu- ations and ROMs were selected on the basis of appropriate criteria and are carried out on time. Furthermore, it examined whether they were carried out by sufficiently qualified experts, whether quality control procedures were applied effectively and whether they provided sufficient evidence on the results achieved by EuropeAid’s interventions. The indicative multiannual evaluation plan for the 2007–13 period aimed to achieve full geographic coverage with a view to drawing lessons for the subsequent programming process. Thematic evaluations were selected on the basis of the materiality of funds allocated and the importance of sectors in countrylevel and regional EU cooperation strategies. The objective was also to increase the number of evaluations of aid delivery methods (e.g. budget support, cooperation with the EIB, development banks, cooperation with UN organisations). For the rolling 2014–18 work programme, the selection criteria were further refined. The objective was to provide sufficient material to report, in 2017, to the European Parliament and the Council on the results and impacts of the financial instruments, as required by the common rules and procedures for the implementation of the Union’s instruments for financing external action. Thematic evaluations were selected on the basis of priority themes of the Agenda for Change
, and the aim was to ensure a proportional balance between regions (Africa, Latin America and Asia) and types of countries (middleincome countries, fragile states and countries with significant budget support programmes). The main reason for carrying out a programme evaluation is that it is an obligation or a possibility set out in the financing agreement signed with the beneficiary or implementing partner. The provision for evalu-ations in these agreements is decided on a casebycase basis at the design phase of the programme but no guidance is given on how to make this decision. The Court’s review of 20 recent programmes (see Annex III) identified significant and unexplained differences in approach:
(a) midterm evaluations were provided for in all programmes but were compulsory in only half of them;
(b) ex post evaluations were provided for in only two thirds of the programmes and were compulsory in only one of them.
EuropeAid has not set up a proper system to monitor the implementation of strategic and programme evaluation plans, analyse the reasons for deviations, and address the systemic causes of delay. Strategic evaluations often took much more time than planned. This contributed to the incomplete execution of the indicative multiannual evaluation programme for the 2007–13 per-iod; only 70 % of the selected evaluations were completed. According to EuropeAid, this was particularly due to staff resource constraints in the Evaluation unit. Even though programme evaluations are usually carried out according to their timetable, the Court’s survey found that, in the period 2010–13, there was a significant difference between the evaluations planned and carried out. The provisions set out in the financing agreements were not systematically applied during programme implementation as evalu-ations were sometimes postponed or cancelled. The Court’s survey indicates that EU delegations take such decisions on the basis of factors like limited staff resources, the needs of evaluation users and problematic programme implementation. The reasons for not carrying out planned evaluations were not documented. This increases the risk that decisions not to carry out an evaluation are not taken for sound reasons, or at the appropriate level. The selection of ROM reviews is based on clearly defined standard criteria and is made in consultation between the ROM contractors and programme managers. ROM reviews are usually carried out according to plan and reports are usually delivered without significant delays. However, in its Special Report No 1/2011 the Court pointed to key limitations which reduce the usefulness of ROM as an indicator for the overall aid portfolio, notably that the sample of projects selected is not established on a statistical but rather on a judgemental basis. In 2011, EuropeAid’s Internal Audit Capability also found that, as the selection is not riskbased, ROM reviews do not provide the feedback to programme managers where it is most needed to improve programme implementation. Evaluations are carried out by external consultants contracted by EuropeAid through public procurement procedures. Proposals submitted by consultancy firms are scored on the basis of specific requirements, in particular those concerning the experts’ professional qualification. Team leaders and senior experts usually have 10 to 15 years of professional experience in the field of intervention, plus 5 years of experience in evaluation for team leaders. ROM reviews are also carried out by external consultancy firms (ROM contractors) selected through public procurement procedures. The required experience of development cooperation in the given region is 10 years for senior experts, 5 to 10 years for midlevel experts and up to 5 years for junior experts. For contracts to be signed in 2014, EuropeAid required ROM contractors to hire specialised experts on long-term assignments in order to further enhance the level of expertise. The Commission’s evaluations standards stipulate that, save in duly justified cases, a reference group must be set up for each evaluation to advise on the terms of reference, support the evaluation work and take part in assessing the quality of the evaluation with appropriate regularity. EuropeAid’s evaluation guidelines stress that quality assurance at each step of the process is a key task of the evaluation manager, together with the reference group, which should include representatives of the main stakeholders concerned. Reference groups are systematically set up for strategic evaluations. They meet on a regular basis and their work, including quality review at all stages, is documented in meeting minutes. This was not the case for programme evaluations. According to the Court’s survey, only 39 % of EU delegations set up a reference group always or most of the time, and a reference group was rarely formally set up by EuropeAid’s operational units and EU delegations visited by the Court. Some consultations may take place between the programme manager and other stakeholders at various stages of the evaluation process (e.g. for drafting the terms of reference or reviewing the draft evaluation report). However, in the absence of appropriate documentation, it was not possible for the Court to assess if this was frequent practice. EuropeAid’s evaluation guidelines provide that the quality of the draft and final evaluation reports must be assessed on the basis of nine welldefined criteria. For this purpose, a quality grid must be filled by two persons who discuss the nine criteria. For strategic evaluations, the quality grid was filled systematically and published together with the evalu-ation report on EuropeAid’s website. For programme evaluations, according to the Court’s survey, only 71 % of EU delegations assess the quality of the evaluation report and only 37 % fill the assessment grid. Programme managers are required to assess the quality of ROM reports by filling a compulsory response sheet. This was done for less than 60 % of ROM reports delivered over the period from 2011 to 2013. To enable an evidencebased assessment of achievements to be carried out, there must be specific, measur-able and timebound objectives for programmes with objectively verifiable indicators. The objectives mentioned in high level policy documents and legal texts establishing the financing instruments are prepared in a political context and, while they should provide a basis, are often not focused enough to be useful at implementing level. They need to be translated into more specific operational objectives when programming aid for a country, region or sector, and designing individual interventions. Out of the 15 thematic, regional and countrylevel programming documents reviewed by the Court (see Annex II), five contained clearly described objectives and indicators, but set no baselines or targets against which to assess performance. For the other 10, objectives were unclear and indicators often lacking (see Box 1). Concerning individual programmes, several Court’s reports have highlighted that objectives are insufficiently clear and that no measurable indicators are defined. Out of the 10 programme evaluation reports reviewed by the Court, four mentioned unclear objectives and indicators, as well as the absence of baselines and unavailability of monitoring data, as major weaknesses (see Box 2). Only two evaluations made positive comments in this respect. In the absence of clearly defined objectives and indicators, it is not possible for an evaluation to establish evidence-based performance assessments. This issue was also mentioned in a 2011 study commissioned by EuropeAid. Investing in People: Midterm Review of Strategy Paper for Thematic Programme (2007–13) and Multiannual Indicative Programming 2011–13 As regards the strengthening of health systems and universal access to basic health care: ‘In terms of delivery, the thematic action should focus on supporting activities that:
stimulate effective leverage of resources for inclusive health system strengthening and universal coverage of health service delivery;
promote and support the application of innovative ways and good practices to address weaknesses in health service delivery and to improve quality of care;
support the role of civil society, professional associations and other stakeholders in their advocacy, policy monitoring and advisory role in the area of health;
support platforms for policy dialogue and exchange of good practices on different aspects of health at regional and global levels;
support partner countries in the process of implementing international legal commitments in the area of health.’
CSP/NIP Bolivia 2007–13 ‘Strengthening the productive sector comprising microenterprises and SMEs, and improving the institutions framework in order to increase their productivity and competitiveness, which will lead to better market access and integration with the purpose of creating decent and sustainable work.’ EuroMed Migration II 2008–11 ‘Strengthen cooperation in the management or migration so as to build the MEDA partners’ capacities to provide an effective, targeted and comprehensive solution for the various forms of migration.’ African Peace Facility 2003–13 ‘Support the operationalisation of the African Peace and Security Architecture (APSA) and to increase the effectiveness of the African Union and the African regional organisations on the conduct of peace support operations in Africa.’ EuropeAid’s evaluation guidelines indicate that:
(a) midterm and final evaluations are intended to draw lessons from the first years of the intervention implementation, adjust the ongoing intervention where necessary and provide an analysis of the first results and impacts achieved;
(b) ex post evaluations are mainly concerned with checking achieved impacts and assessing the sustainability of the intervention’s benefit, report to the institutions that have allocated the resources and help to transfer acquired experience to other countries or sectors.
Ex post programme evaluations are seldom carried out. For the two EU delegations visited, only three out of the 17 evaluations carried out in the 2010–13 period were ex post evalu-ations. Financing agreements rarely set an obligation to perform such evaluations (see paragraph 26(b)). Another reason for the lack of ex post evalu-ations is that programme evaluations are funded from the programme’s own budget and are therefore constrained by its completion date. According to EuropeAid, additional contributing factors are the unavailability of monitoring data after programme completion and the possible evolutions in the partner country’s context that may affect the programme’s effectiveness. The Court does not consider these to be valid arguments: the monitoring system can be designed to continue collecting the data deemed necessary during a certain period after programme completion (see paragraph 47) and the purpose of evaluation is also to identify and assess the impact of unexpected factors. The ROM guidelines indicate that ex post ROMs allow solid information to be gathered about a project’s design, real impact and sustainability which can only be fully appreciated after a project has ended. But ex post ROMs represented on average only 11 % of the ROMs carried out in the 2008–12 period. EuropeAid decided to discontinue with them from 2014. It considered that the methodology for ex post ROM was inappropriate for assessing impact and that the revised ROM system aims only to support programme management and implementation. In contrast with the limited proportion of ex post programme evaluations and ROMs, the Results Measurement framework (ReM) implemented at the European Investment Bank (EIB) since 2012 includes ex post monitoring until 3 years after project completion and the EIB intends to use the data thus collected for ex post evaluations. Under the lead of EuropeAid’s Evaluation unit, the OECD DAC Network on Development Evaluation produced a document called ‘Evaluating Budget Support — Methodological Approach’ which was published in 2012. It is a common evaluation framework to evaluate budget support operations and provides guidelines on how to organise and carry out these evaluations. The document recognises that in most cases, it will be difficult to quantify the budget support contribution, which will need to be assessed in a qualitative manner. While qualitative assessments are, of course, considered in all evaluations, the difficulty in establishing unambiguous logical causeeffect links between results observed and the budget support programmes weakens the reliability of evidence provided by the evaluation. This inherent limitation is not mentioned either in budget support evaluation reports or in EuropeAid’s reports. EuropeAid’s capacity to assess and report on results achieved is significantly reduced by insufficiently clear objectives and indicators set for operational programmes and by the focus put on evaluations and ROM during the implementation of programmes. As the Court mentioned in several reports, there is little information on the actual results of the aid in the EU delegations’ EAMRs and EuropeAid’s two key annual reports which tend to focus more on activities implemented, rather than on policy achievements. This is also the case with EuropeAid’s contribution to the Commission’s Article 318 evaluation reports. In its 2012 annual report, the Court found that, despite some positive developments, these reports did not provide sufficient, relevant and reliable evidence on what the EU’s policies have achieved and that they were therefore not suitable for their intended use in the discharge procedure. The Court examined whether EuropeAid’s evaluation and ROM systems ensured that the findings made were properly followed up and disseminated to all intended users. Actions to be taken following a strategic evaluation are described in a document published together with the evaluation report on EuropeAid’s website. While this is sound practice, most of such documents reviewed by the Court did not provide sufficiently detailed information on the planned actions, their timetable and the allocation of responsibilities. The followup of the action plan by the Evaluation unit is limited to 1 year after the publication of the evaluation report. This is often too short to implement actions plans which involve amendments to sectorial policies and country or regional EU cooperation strategies. In 2012, an OECD DAC Peer Review also found that EuropeAid’s Evaluation unit did not have a clear overview of the extent to which recommendations made on the strength of strategic evaluations were accepted and used to inform new policies and programmes. In 2014, a study commissioned by the Evaluation unit on the uptake of strategic evaluation results found that they were not used effectively. Programme evaluation and ROM findings are usually discussed within operational units or EU delegations and programme managers are responsible for ensuring that appropriate actions are taken. Unlike for strategic evaluations, and as the Court found in the two EU delegations visited, documen-ted action plans are not systematically prepared and monitored. According to the Court’s survey, only 44 % of EU delegations usually draw up action plans. Concerning ROM reviews, the response sheets reviewed by the Court described actions to be taken. But response sheets were completed for only 60 % of ROM reviews carried out in the period from 2011 to 2013. In more than half of the cases reviewed, the Court found evidence of use being made of some findings of the programme evaluations and ROMs. In the absence of an appropriate followup system, EuropeAid cannot ensure that findings and recommendations from programme evaluations and ROM reviews are used effectively. Strategic evaluation and ROM reports are easily accessible to all EuropeAid’s units and EU delegations. Strategic evaluation reports are also available to external stakeholders. They are published on the websites of EuropeAid and the OECD, and are sent to the European Parliament, the Council and EU Member States. The reports and lessons learned are also disseminated in a variety of ways (e.g. meetings, workshops and seminars) adapted to the target audience (e.g. Commission departments, EEAS, EU delegations, European institutions, external partners, and networks of experts). EuropeAid presents a synthesis of strategic evaluation results and the key lessons learned, as well as the aggregated ROM performance data in its AAR and AR. It also contributes to the preparation of the Commission’s Article 318 evaluation report which provides summaries of strategic evaluation findings. Programme evaluation reports are generally shared and discussed by programme managers with directly interested stakeholders, i.e. mainly EuropeAid’s geographical and thematic units and the implementing partner. They are rarely shared with beneficiaries and other donors. Delegations are required to report on lessons learned from programme evaluations in their halfyearly EAMRs to EuropeAid. EAMRs reviewed by the Court contained no or little information in this respect. For programme evaluations managed by EuropeAid’s operational units, there was no similar reporting requirement until 2012. In EuropeAid’s 2013 AAR a summary of 15 of these programme evaluations is provided; however, what this represents in terms of total evaluations completed in 2013 is not indicated. As the Court has pointed out, programme evaluation reports cannot be systematically used because there is no central database for holding such evaluations. This reduces significantly EuropeAid’s capacity for analysing and disseminating this information internally and to external stakeholders and for providing a basis for accountability. EuropeAid has stated its intention to develop a new IT tool to manage, store and analyse ROM and programme evaluations, which would serve as a central storage point for all reports and allow users to perform advanced searches and analysis. At the time of the Court’s audit, the decision to develop the part of the application for programme evaluations had not yet been made, even though report accessibility was a more crucial issue for programme evaluations. The Court concludes that EuropeAid evaluation and ROM systems are not sufficiently reliable. Overall, the evaluation and ROM functions are well organised, with an appropriate combination of centralised and decentralised responsibilities which aim to promote a culture of evaluation and give evaluation a prominent role in the aid management process. However, there is inadequate overall supervision of programme evaluations by senior management. In the absence of proper management information on staff and financial resources used for programme evaluations, EuropeAid cannot ensure that they are appropriate and are efficiently allocated to the various evaluation activities. The evaluation and ROM systems do not sufficiently ensure the delivery of relevant and robust evaluations and ROM findings. Evaluation plans, drawn up on the basis of insufficiently clear criteria, do not guarantee that priority is given to those evaluations which are most useful for decisionmaking. In addition, the absence of monitoring prevents EuropeAid from identifying and addressing the causes of frequent deviations from these plans. Procurement procedures enable external experts with the required skills and knowledge to be selected. Quality control procedures for ROM and programme evaluations are not applied consistently within EuropeAid. Maximum use is not made of evaluations and ROM findings. Proper followup mechanisms are not in place to ensure that evaluation and ROM findings are appropriately used. Strategic evaluations and ROM findings are widely disseminated through a variety of communication tools to a wide range of internal and external stakeholders. However, the lack of access to programme evaluations and the absence of search tools are obstacles to the proper sharing of knowledge. The evaluation and ROM systems do not provide adequate information on results achieved due to insufficiently welldefined objectives and indicators, the limited proportion of ex post evaluations and ROMs and inherent limitations in the evaluation methodology for budget support. As a result, EuropeAid’s capacity to account for its activities is limited and its reporting gives few indications of the actual results achieved. Recently, the Commission and EuropeAid launched initiatives to improve their capacity to assess and report on policy achievements. These initiatives were at too early a stage for the Court to assess whether they were effectively addressing the issues raised in the present report. EuropeAid should maintain adequate management information and perform needs assessments on a regular basis in order to ensure an informed allocation of financial and staff resources between programme evaluations and ROM. In order to ensure that evaluations carried out reflect the priorities of the organisation, EuropeAid should:
define clear selection criteria to prioritise programme evaluations and document how they were applied in establishing the evaluation plans, taking into account the complementarity with ROM;
significantly strengthen its system to monitor and report on the implementation of evaluation plans, including an analysis of the reasons for delays and the description of measures adopted to address them;
strengthen the overall supervision of programme evaluation activities by EuropeAid.
In order to ensure the quality of programme evaluations and ROMs, EuropeAid should:
insist that operational units and delegations apply the quality control requirements, including for programme evaluations the use of a reference group and documentation of quality controls performed;
check, on a regular basis, the application of these controls.
In order to enhance the evaluation system’s capability to provide adequate information on results achieved, EuropeAid should:
apply more rigorously the regulatory provisions requiring the use of SMART objectives and verifiable indicators;
modify the monitoring system so that it continues to provide data on programmes until at least 3 years after their completion; and
increase significantly the proportion of ex post programme evaluations.
In order to ensure the maximum use of evaluation and ROM findings, EuropeAid should:
extend the followup period for strategic evaluations;
develop a central database for programme evaluations to store evaluations reports, action plans and their followup, with search functionalities which permit analysis.
This Report was adopted by Chamber III, headed by Mr Karel PINXTEN, Member of the Court of Auditors, in Luxembourg at its meeting of 21 October 2014. For the Court of Auditors Vítor Manuel da Silva Caldeira President
ANNEX
ANNEX I ROM GRADING SYSTEM
ROM assesses a programme performance in respect of the five evaluation criteria (relevance, efficiency, effectiveness, impact prospects and potential sustainability) according to the following grading system: Grading Numerical Qualitative Explanation for the grade A 4 Very good The situation is considered highly satisfactory, largely above average and potentially a reference for good practice. Recommendations focus on the need to adopt these good practices in other operations. B 3 Good The situation is considered satisfactory, but there is room for improvements. Recommendations are useful, but not vital for the operation. C 2 Problems There are issues which need to be addressed; otherwise the global performance of the operation may be negatively affected. Necessary improvements however do not require major revisions of the operations’ strategy. D 1 Serious deficiencies There are deficiencies which are so serious that, if not addressed, they can lead to failure of the operation. Major adjustments and revision of the strategy are necessary. Each programme is then categorised, according to the number of A, B, C and Ds it has received. There are four categories in which a programme can be classified. ‘I’ indicates very good performance; ‘II’ indicates good performance; ‘III’ indicates that a programme is performing with problems; and ‘IV’ indicates that a programme is not performing, or that it is experiencing major difficulties. EuropeAid then determines the percentage of programmes in the ROM portfolio falling under each category. This is done for the whole portfolio, per region and per sector within a region. This is the basis for its reporting on the portfolio performance.
ANNEX II LIST OF PROGRAMMING DOCUMENTS REVIEWED
Thematic Regional Country
Environment and Natural Resources Thematic Programme (ENRTP): strategy paper and multiannual indicative programme (MIP) 2011–13
Food security thematic programme, strategy paper and MIP 2011–13
Investing in people: Midterm Review of strategy paper for Thematic Programme 2007–13 and MIP 2011–13
Central Asia Regional Strategy Paper 2007–13
Latin America Regional Indicative Programme 2007–13
Asia MIP 2007–10
Intra ACP cooperation 10th EDF, MIP 2008–13
Country Strategic Paper (CSP) Laos 2007–13
CSP Bangladesh 2007–13
CSP India 2007–13
CSP/National Indicative Programme (NIP) Benin 2008–13
CSP/NIP Chad 2008–13
CSP/NIP Zambia 2008–13
CSP/NIP Uruguay 2007–13
CSP/NIP Bolivia 2007–13
ANNEX III LIST OF FINANCING AGREEMENTS REVIEWED
(euro) Country Decision number Title EU contribution Africa Madagascar FED/2012/023-503 (EC) Amélioration de la Sécurité Alimentaire et Augmentation des Revenus Agricoles (ASARA) 40000000 Uganda FED/2013/023-513 (EG) Small and Medium Agribusiness Development Fund — A Public–Private Partnership. 15000000 Ethiopia FED/2011/023-531 (CA) Capacity development and governance support for scaling up renewable energy services in rural and peri-urban areas of Ethiopia — CfP 129364 Prop 353 1176000 Somalia FED/2012/023-536 (EC) Support to Democratisation 14000000 Papua New Guinea FED/2012/023-538 (EG) Non State Actors (NSAs) Support Programme in Papua New Guinea 4538000 Nigeria FED/2013/023-551 (EG) Energising Access to Sustainable Energy in Nigeria (EASE) 27000000 Kenya FED/2012/023-566 (EC) Standards for Market Access Programme (SMAP) 12100000 Liberia FED/2012/023-568 (EC) Support to trade facilitation in the customs sector 3000000 8 Financing Agreements TOTAL 116814000 Latin America Honduras DCI-ALA/2013/023-720 (EC) Promoting fair and accessible justice in Honduras (EuroJusticia) 27500000 Costa Rica DCI-ALA/2010/021-501 (EC) Comprehensive strategy to reduce school drop-out rates in public secondary education 8500000 Dominican Republic BAN/2012/024-100 (EC) Banana accompanying measure (BAM) Dominican Republic 16340000 Uruguay DCI-ALA/2011/022-498 (EC) Support for the reform of the Uruguayan penal justice and penitentiary systems for the improvement of the quality of life and social and labour reintegration of prison inmates 5000000 4 Financing Agreements TOTAL 57340000 Neighbourhood Egypt ENPI/2012/024-080 (EC) Emergency Employment Investment Project 70000000 Morocco ENPI/2012/024-196 (EC) Programme d’appui à l’Initiative Nationale de Développement Humain (INDH) — phase II 25000000 Tunisia ENPI/2011/022-772 (EC) Programme de Modernisation des services 20000000 West Bank and Gaza Strip ENPI/2010/022-476 (EC) PEGASE: Private Sector Development 11000000 4 Financing Agreements TOTAL 126000000 Asia and Pacific Afghanistan DCI-ASIE/2013/024-844 (EG) Support to credible and transparent elections 15000000 Tonga FED/2013/024-766 (EG) Technical Cooperation Facility, Good Governance and NSA programme II 472000 Vietnam DCI-ASIE/2013/024-370 (EG) EU-HSPSP 2 — EU Health Sector Policy Support Programme phase 2: Towards Equity and Quality of Health Services in Vietnam 114000000 Bangladesh DCI-ASIE/2012/022-446 (EC) Poverty Reduction Through Inclusive and Sustainable Markets (PRISM) 30000000 4 Financing Agreements TOTAL 159472000 20 Financing Agreements TOTAL 459626000
ANNEX IV LIST OF EVALUATIONS REVIEWED
(euro) Year Programme or Strategic Evaluations Evaluation type Programme/theme/country evaluated Financing instrument EU contribution Evaluation cost 2012 Strategic Country European Union’s cooperation with Nepal — Country Level Evaluation (covered period 2002–10) Mainly DCI geographic and thematic and EIDHR 58000000 201711 2011 Strategic Thematic European Commission support in the education sector (covered period 2000–07) DCI geographic and thematic, EDF, ENPI 1900000000 457265 2012 Programme Final Food Facility — AU-IBAR projects (LEISOM and VACNADA) Food facility 2507860 131870 2013 Programme but similar to Strategic Final Final evaluation of the whole Food Facility (2009–12) Food facility 1000000000 399720 2013 Programme but similar to Strategic Final Three Year Action Programme for the African Peace Facility for the period 2011–13. Mainly EDF 549345726 478022 2012 Programme Mid term ‘Invest in MED’ ENPI South 9000000 119778 2013 Programme ex post ‘Euromed Migration II’ ENPI South 4994000 148580 2011 Programme Final Appui à l’Entretien Périodique du Réseau Routier Classé EDF 36709425 72848 2012 Programme Final Programme d’appui à la décentralisation au Bénin EDF 13785026 47120 2010 Programme Final Erasmus Mundus, External Cooperation Window for India DCI geographic 28600000 147745 2014 Programme Final EC assisted State Partnership Programme to the states of Chhattisgarh and Rajasthan (budget support) DCI geographic 158590819 1934700 2011 Programme Final 20 projects selected among 100 on EU Funded Environment Projects in India DCI geographic 340000000 118008
ANNEX V LIST OF ROMS REVIEWED
(euro) Programme Title Country/region/ sector EU contribution Report type Report date Food Facility — AU-IBAR projects (LEISOM and VACNADA) Fighting hunger 2507860 Ongoing June 2011 FOOD FACILITY Fighting hunger 1000000000 Ongoing May 2011 Public Administration Capacity Building Facility Libya Libya 4500000 Ongoing October 2013 PROJECT ‘Euromed Migration II’ Neighbourhood South 4994000 Ongoing October 2010 Appui à l’Entretien Périodique du Réseau Routier Classé Benin 36709425 Ongoing November 2008 Programme d’appui à la décentralisation au Bénin Benin 13785026 Ongoing February 2012 Programme Société Civile et Culture Benin 14400000 Ongoing May 2013 Sustainable Textiles for Sustainable Development India 1672945 Ongoing July 2012 European Business and Technology Centre in India (EBTC) India 6586578 Ongoing July 2012
ANNEX VI LIST OF DELEGATIONS WHICH REPLIED TO THE SURVEY
Country/Delegation Africa Caribbean Pacific Latin America Angola Bolivia Botswana Chile Burkina Faso El Salvador Burundi Honduras Cape Verde Nicaragua Congo (Brazzaville) Paraguay Djibouti Uruguay Ethiopia Neighbourhood Madagascar Algeria Nigeria Armenia Rwanda Azerbaijan Togo Belarus Dominican Republic Georgia Guyana Israel Solomon Islands (Del. Papua New Guinea) Jordan Asia Lebanon Bangladesh Morocco Cambodia Tunisia India Ukraine Laos, Thailand Nepal Pakistan Philippines Vietnam