[Edited to update links August 2016]
Here's the question: Will bilateral aid agencies hold the multilaterals to account for results or not? UN agencies’ results reporting is inconsistent, and the results frameworks of SIDA, AUSAID and DANIDA remain ambiguous. This post reviews results frameworks from USAID, DFID, CIDA and EuropeAid.
Background: Ambiguous results chains at the UN, and some bilateral agencies.
The USAID Results Framework
Length: 4 documents, 248 p.
“…focuses mainly on “physical deliverables” (e.g. numbers of schools, numbers of clinics, etc.). With the new orientation in US foreign policy, there is an opportunity to measure development assistance performance more in outcomes than in physical deliverables.”
“A tangible, immediate, and intended product or consequence of an activity within USAID control. Examples of outputs include people fed, personnel trained, better technologies developed, and new construction.” [p. 67]
“…it is important to understand the difference between Outputs and results….In differentiating outputs from results, it can be useful to think of results as developmentally significant changes that affect a broad segment of society, while outputs are lower-level steps that are essential in achieving these changes.” [p. 27].
A similar distinction between results and Outputs occurs in the USAID Introduction to Programming Policy where results are defined as:
“A significant, intended, and measurable change in the condition of a customer, or a change in the host country, institutions, or other entities that will affect the customer directly or indirectly. Results are typically broader than USAID-funded outputs….”[p. 70]
This suggest that simply reporting on completed activities or products would not be acceptable within the evolving USAID context. Given that the United States provides such a large percentage of the assistance available to agencies such as UNDP, it remains to be seen whether this focus on results has in any way been communicated to the UN agencies.
The DFID Results Framework
“The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.”
“Outputs are the specific, direct deliverables of the project. These will provide the conditions necessary to achieve the Purpose”.
“DFID has a strong results-based management framework, and this – combined with a purpose and performance-driven organisational culture and cohesion at the senior level – is important in ensuring effective delivery of the aid programme.”
“…DFID will have to work with the fact that multiple partners mean differences in terminology and approaches.
DFID has played a leading role in ensuring harmonisation of approaches, and is committed to continuing in this vein. However, it is important that in pursuing a harmonisation agenda, we do not relax our requirements for robust monitoring and evaluation tools.
Differences in language and approach should not be an excuse for gaps in information. In fact, the revised logframe format has already been used by DFID teams when negotiating with partners. DFID needs the information in the logframe in order to report to UK taxpayers that funds are being used in the best possible way and delivering measurable results.” [p. 16]
It will be interesting to see if this applies to multilateral agencies using DFID funds. [Update note, August 13, 2011: The DFID Multilateral Aid Review was completed in March 2011.]
The Global Affairs Canada (CIDA) Results Framework
“Clearly splits development results from products and services (outputs). This distinction should strengthen performance reporting by partners, given that it is now clear they will have to report on both outputs and outcomes”
“Results are the same as outcomes. An outcome is a describable or measurable change that is derived from an initiative's outputs or lower-level outcomes. [ p. 8]
- Immediate Outcomes are near-term results phrased as changes – increases in understanding, skills or access.
- Intermediate Outcomes are mid-term changes, “expected to logically occur” within the life of a project, if Immediate Outcomes are achieved, including things such as increased use of clean water, or improved trust in government.
- Ultimate Outcomes are hoped for long-term changes, the justification for the project, but unlikely to be achieved during the life of a project. These refer to things such as improved health status, or reduced vulnerability of children in conflict areas.
These results categories are illustrated in much more detail in the Global Affairs RBM Guide.
“...cumbersome, with limited differentiation in the indicators required and the processes involved for large and small programmes. While this helps to compare results among different activities, efficiency is compromised. The system might also be used to justify risk aversion rather than risk management, especially in those areas where it is more challenging to articulate measurable results (e.g. in governance)” [p. 49]
“ For example, an application for a small workshop organised by an NGO in Canada has to set out development results as if it were equivalent to a major bilateral programme in a partner country, with requirements to provide an impact evaluation. While providing discipline for NGO proposals may appear reasonable in theory, the practice can appear unnecessarily burdensome to the applicant.” [footnote 37]
It will also be interesting to see if CIDA applies its standards on accountability and results reporting to multilateral agencies.
The EuropeAid Results Frameworks
[2016 Edit: Many of the EuropeAid documents have moved, but if readers use the "advanced search" tool in the "Library" tab, and narrow the search further by opening the "categories tab under that, it is possible to still find many, although not all, of the documents referred to below]
"Because the Community functions both as a donor agency and as a multilateral recipient of Member State funds it is understandable that it does not allocate a large proportion of its funds to other multilateral institutions...." [p. 42]
EuropeAid Impacts, defined as “Further long term change attributable to the intervention (e.g. development of trade)”.
“Monitors have to fully understand the concepts and terminology used in ROM and to apply them in the correct and coherent manner. This is specially true for ‘efficiency’, ‘effectiveness’, “outcomes’ and ‘outputs’ as these terms might be used differently in other management and M&E systems.” [ p. 48]
The irony is that this document did use some words differently than those defined in the EuropeAid Glossary. While the Glossary refers only to Outputs, Results and Impact, distinguishing between Outputs and Results, the Handbook on Results Oriented Monitoring said of Outputs that they are:
“the goods and services produced; e.g. children vaccinated. In the EC’s Logframe structure these are referred to as ‘results’;” .[p. 29].
- EuropeAid Outputs as completed activities (eg. Training sessions”),
- EuropeAid Outcomes as intermediate results (“improved capacity of those who attended the training”) and
- EuropeAid Purpose, or longer-term results, (“improvements in area of intervention due to the improved capacity of the target group”) [p. 65], which are the “specific, central highest ranking objective of the project” [p. 71], the highest level on which a project reports, but not necessarily the highest level on which it is monitored.
- EuropeAid Impact as the overarching result to which a project may contribute, and justification for a project.
The European Community's aid volume is huge. As the 2007 OECD DAC peer review noted the "volume of Community ODA alone is larger than that of the World Bank’s International Development Association and several times that of the United Nations Development Programme" [p. 12] and much of it is administered or influence by the work of EuropeAID. EuropeAid should, therefore, have a very big influence on how multilateral aid agencies treat results reporting.
The bottom line: Holding UN agencies to account
Further reading on Results-Based Management at DFID, USAID, CIDA and EuropeAid :
Edited to update links in March 2011, January 2012. and August 2016