Improving Efficiency Measurement in Evaluation - A Critical Look, New Insights and an Alternative Approach
⛶ Full screen
This challenge addresses the usefulness of current approaches to measuring efficiency, analyses their limitations, discusses the complexity of this criterion, and suggests an alternative approach. A prevailing view is that evaluating efficiency is usually weaker and not substantive enough compared to assessing relevance, effectiveness, impact and sustainability in most project, program, and policy evaluations. This challenge takes a fresh look at the issue, tries to provide some new insights, and suggests an alternative approach to measuring efficiency. Efficiency is commonly understood to mean getting the maximum output for a given input or budget or expending the least inputs to produce the desired output. It is synonymous with productivity. Quoting Wikipedia “Efficiency is the (often measurable) ability to avoid wasting materials, energy, efforts, money, and time in doing something or in producing a desired result. In a more general sense, it is the ability to do things well, successfully, and without waste. In more mathematical or scientific terms, it is a measure of the extent to which input is well used for an intended task or function (output). It often specifically comprises the capability of a specific application of effort to produce a specific outcome with a minimum amount or quantity of waste, expense, or unnecessary effort. Efficiency refers to very different inputs and outputs in different fields and industries.
Efficiency is very often confused with effectiveness. In general, efficiency is a measurable concept, quantitatively determined by the ratio of useful output to total input. Effectiveness is the simpler concept of being able to achieve a desired result. A common but confusing way of distinguishing between efficiency and effectiveness is the saying "Efficiency is doing things right, while effectiveness is doing the right things." This saying indirectly emphasizes that the selection of objectives of a production process is just as important as the quality of that process. This saying however obscures the more common sense of "effectiveness", which would/should produce the following mnemonic: "Efficiency is doing things right; effectiveness is getting things done." This makes it clear that effectiveness, for example large production numbers, can also be achieved through inefficient processes if, for example, workers are willing or used to working longer hours or with greater physical effort than in other companies or countries or if they can be forced to do so. Similarly, a company can achieve effectiveness, for example large production numbers, through inefficient processes if it can afford to use more energy per product, for example if energy prices or labor costs or both are lower than for its competitors.”
Efficiency in the Context of Evaluation
Efficiency is the second of the quintet of OECD criteria used in evaluation. It is somewhat of a step child in the sense that evaluators don’t give the importance it deserves because assessing efficiency is difficult in a complex project and difficult to undertake from a retrospective perspective. Results may be case specific, with limited generalizable learning cutting across many sectors. Also the OECD definition does not provide metrics and the only commonly used metric, DFID’s Value for Money (VFM) is inadequate and fails to grasp the dynamics of the criterion. Efficiency gets attention in evaluations when the project or program suffers large time delays or cost overruns. Evaluators then search for reasons for the failures that led to these overruns. Often these are ad hoc, as there are no detailed implementation plans or a critical path activity analysis to guide them. This is particularly applicable to social or soft sector projects which often lack the rigor of planning that characterizes a major industrial investment. Efficiency in the evaluation context is the bridge between inputs and the other three evaluation criteria. It is an intangible input or the catalyst that produces the maximum values of outputs, outcomes and sustainability. Without efficiency, projects or programs are destined to poor outcomes or failure. To measure efficiency in the evaluation context, one needs a clear understanding of the concept.
The OECD (2019) defines efficiency as follows: "Efficiency: how well are resources being used? The extent to which the intervention delivers, or is likely to deliver, results in an economic and timely way. Note: “Economic” is the conversion of inputs (funds, expertise, natural resources, time, etc.) into outputs, outcomes, and impacts, in the most cost-effective way possible, as compared to feasible alternatives in the context. “Timely” delivery is within the intended timeframe, or a timeframe reasonably adjusted to the demands of the evolving context. This may include assessing operational efficiency (how well the intervention was managed)".
For a completed project, the actual cost versus planned cost and on- time completion denotes efficient performance. If a project suffers cost and time overruns or fails to deliver outputs within the budgeted plan, the performance is inefficient. Efficient procurement presumably has resulted in the lowest cost, so the client does not benefit from cost savings. Projects have numerous tasks to be performed at the required specifications or standards. Performance metrics can be defined for each task or phase which is an indicator of efficiency. It can also be defined for completion of the aggregation of tasks or at the project or program level. I think the OECD definition does not fully capture the productivity ethos in its definition. And it does not provide any metrics for measuring efficiency.
A major gap in the OECD evaluation criteria is the absence of or insufficient attention to consideration of risk as a factor in the project’s outcome. Risk analysis is always undertaken in the appraisal of private sector projects but in donor financed projects, this is often missing, particularly in the social sectors. Multilateral banks assess risk at project appraisal but during evaluation this factor is hardly considered and lessons not internalized. The risk of a project failing to meet its objective or goal depends on many factors, some within the sponsor’s control such as implementation efficiency, others outside such as changes in regulations or world prices for inputs or outputs. Careful planning and application of high efficiency technologies and project management can reduce risks from internal factors. Evaluation should also include an assessment of whether the risk assessment and implementation plans at appraisal adequately mitigated the risks and if not, what were the gaps.
DFID uses an alternative definition of efficiency using the criteria Value for Money (VFM). After some thought, this begs a lot of questions. Value is produced through out a project's useful life and is not reflected by the investment cost. It is closer to a rate of return than a measure of implementation efficiency. Value is measured by net benefits of a project measured at project completion but by the discounted value of the stream of benefits generated by the project over its useful life time which is best captured by the economic rate of return (ERR). Note that ERR calculations at the physical completion of a project uses a revised set of estimated benefits projections and actual investment costs to calculate the ex-post ERR which is still an estimate of the project's value. Alternatively, the VFM is the net present value of a project re-estimated by the evaluator at the physical completion of the project based on the opportunity cost of capital for the donor and beneficiary. It is a measure for outcome achievement, not a measure of efficiency. I think that is an inherent limitation of the VFM as an efficiency indicator. Evaluators have used the VFM simplistically which does not shed much light on project efficiency. Where does one go from here? Let me try spell out an alternative approach.
Efficiency in the simplest definition is the value of outputs generated divided by the cost of inputs. But this is taking a black box view without paying attention to the processes involved in the transformation. What is the value of the intervention outcome and how is it measured? Should one look at effectiveness/efficacy of the outputs and the resulting outcomes and impacts or is it sufficient to just focus on outputs? Should one consider project /program design alternatives in looking at this transformation. The outputs could have been produced at a huge cost because the wrong design alternative was chosen. Few evaluations focus on the issue of alternative strategies /designs to achieve a desired outcome. Producing more output may not be the most efficient strategy if there is marginal diminishing returns. There is need to look at the individual processes involved in the transformation of inputs into outputs and not just treat it as a black box. One should break the transformation of inputs to outputs into the individual tasks and examine the efficiency of each task.
Thus formulating metrics for efficiency is more complex for projects with heterogenous elements/pathways often with branch looping networks than those that involve processes that are linear and homogenous. Let me give a generic example. A project involving expansion of steel capacity for a country. This would involve a feasibility study to examine the viability of the project which is measured by the economic and financial rate of returns which involves inter alia analysis of a host of factors including world price trends twenty years into the future, technologies to be used, competitiveness of production vis-a vis other low cost producers, market and demand trends, demand for specific types of steel products and trends in the market and many other factors. It could take many months and hundreds of thousands of dollars. If one rushes the feasibility study by consultants or the appraisal of the project by financiers, it could lead to wrong choices and painful outcomes. If an appraisal justifies viability and financing is available, efficient procurement is the next step. Usually this involves international competitive bidding to select the contractor, a process which usually takes many months. Finalizing the project documents and contracts can also take many months. Then the financiers have to release funds in tranches and so on.
Projects of this magnitude involving billions of dollars in investment and many parties require close and complex supervision and monitoring often assigned to another consulting firm. Careful implementation plans including critical path analysis and any slippage is detected with early warning systems. Simple time and cost overruns are not adequate in the management of these projects. I could elaborate more but I think this is sufficient to illustrate the point I am making. Devising metrics involves a lot more than examining output/input ratios. So one needs to develop a whole range of metrics to examine the efficiency criteria broken down by each phase of project implementation and the processes involved.
In my view, Efficiency is the most actionable dynamic element of project implementation and while we need a better understanding of how to measure efficiency in social sector interventions, an action plan to monitor whether agreed targets are being achieved during implementation is central to its measurement. Efficiency is the invisible bridge between inputs and the other three elements in the results framework - outputs, outcomes, and impacts- linking the chain together. The three key parameters to measure efficiency are (1) the project task/activity schedule with key milestones, (2) the expenditure pattern and commensurate defined outputs achievement , and (3) metrics to measure the extent to which the quality of outputs of the project is being achieved. The action plan to achieve the efficiency goal comprises three linked elements: (1) the project implementation plan with a clearly defined critical path of activities/actions and precise monitorable time schedule, (2) the logistics and technical plan including defining the supply chain and (3) the overall project management organization and plan to exercise oversight of the entire implementation process.
Beneficiaries and donors often do not focus on these elements, preferring to take a black box approach without assessing whether the contractor or consulting firm has taken adequate measures and developed the project plans and organization structure to implement the project efficiently. The approach I summarized above deals with this issue in a proactive way. This is the central thesis of my presentation.
Efficiency is the (often measurable) ability to avoid wasting materials, energy, efforts, money, and time in doing something or in producing a desired result. In a more general sense, it is the ability to do things well, successfully, and without waste. In more mathematical or scientific terms, it is a measure of the extent to which input is well used for an intended task or function (output). It often specifically comprises the capability of a specific application of effort to produce a specific outcome with a minimum amount or quantity of waste, expense, or unnecessary effort. Efficiency refers to very different inputs and outputs in different fields and industries.
Effectiveness is the capability of producing a desired result or the ability to produce desired output. When something is deemed effective, it means it has an intended or expected outcome, or produces a deep, vivid impression.