Supporting
95

Think Efficiency


⛶ Full screen

We are the "Think Efficiency" group of Alena, Hanan, Adeola, Eloisa, Zeina and Traore.

08.07.2020 13:15 We had a call and finalized 2 tasks from Day 1.

This is our Challenge Definition: HOW CAN WE come up with innovative approaches FOR measuring sectoral efficiency in evaluations IN ORDER TO SOLVE the challenges in sustainable developments? 08.07.2020 13:17 09.07.2020

Issue Map

Donor: Issues

  • Timely/quality reporting: need for specific data/ different donor requirements/insufficient capacities, weak M&E systems lack of funds for quality improvements and staff allocation
  • Accountability: donors need to report on the good use of money/ showing results to taxpayers - need for finance tracking and transparency
  • Value for Money: what method is acceptable and delivers the message needed [for decision making (is this a better project than another project?) or for accountability? (Is this the best project we could invest on to solve the problem? What is the opportunity cost?) ……
  • How and whether we should go beyond least costs approach, integration of social, environmental concerns and financial concerns yet not fall short below established benchmarks in terms of economy; Worry: application of “upfront costing” methods vs “Life cycle costing” is more typical and easier in terms of reporting.

 

  • What should be used: Transformational efficiency ratios [inputs/results or costs/benefits?] versus Optimization efficiency which are based on Net benefits concepts (costs-benefits expressed in NPV - IRR) – should we depend mainly on quantitative data to measure efficiency or qualitative data, as well.
  • How much efforts should be engaged in quantifying the soft/intangible benefits? How can intangible benefits be quantified? How important is this for your constituents?
  • Sustainability of socio-economic benefits (tangible and intangible benefits)

Community (beneficiaries): Issues

  • Outcomes/ Financial benefits: financial literacy is needed to ensure that when benefits from a project are financial, they can differentiate between incomes and benefits (incomes-costs) to respond to surveys in a correct way.
  • When benefits are NOT financial, how do projects capture and quantify intangible benefits? {access to clean water}
  • Socio-economic benefits: benefits at socio-economic level are generally relevant for State actors or Development agencies actors not so much for beneficiaries, at the beneficiary level they are part of the intangible part.
  • The monetary value assigned from project financial resources per beneficiary - $xxx per capita of beneficiaries
  • Benefit meet their needs; sustainability of benefits; equitable and inclusive distribution of benefits.

Project staff: issues

  • Resources: efficiency measures require good data collection and standardization or systems that can aggregate and access databases – Rationalization of data collection (the less the better); collecting data more than needed would thin project resources and render it less cost-efficient – In capacity building interventions, training should not be for the sake of training (to achieve performance targets) but need to be more focused and directed to targeted beneficiaries with sound and measurable learning outcomes.
  • Plan: planning and identifying concrete tasks is key to calculate needed resources
  • Technical support and Time Efficiency: MEAL technical capacities (human & institutional) to streamline the M&E function – Not assigning a dedicated project staff to the M&E function, adding it as a task besides other operational/managerial tasks, or having a single M&E person for all projects managed by an organization, which leads to completion of all tasks on project staff valuable time resource (Time Efficiency).
  • Advocacy:

State actors: issues

  • Political will
  • Capacities: MEAL technical capacities (human & institutional) to streamline the M&E function (Technical &Time Efficiency).

Non state actors: issues

  • Resources
  • Capacities MEAL technical capacities (human & institutional) to streamline the M&E function (Technical &Time Efficiency).

 

Possible Overall question/solution …. IDEAS 😊

  • Standard requirements for efficiency reporting [from donors] so that institutions and country can develop systems and databases that can deliver timely and quality indicators
  • Databased with country-based validates unit costs - agreed for reporting

Agreed methodology for efficiency calculation by type of project/sectors

Dashboards (such as the one build up by on COVID19) that can be consulted to see real time data – Creating Efficiency Dashboard (country-specific, regional – global) – Creating Donor-specific Dashboards (benchmarking efficiency measures “financial, technical, & time efficiencies

Creating a “” Think-Efficiency Community of Practice”” where all knowledge exchanges, wealth of efficiency resources (grey literature, papers, publications, research work, etc.), capacity building w/s, interactions with think-tanks, social media communication channels/presence etc can be done.

We have defined the Issue, undertaken interviews and redefine the problem statement. We are now looking for ideas on how to solve the problem. `10.07.2020 09:20` --- `10.07.2020 14:01` The team has completed the following tasks: Brain Dump, Issue Map, Conducted two interviews. Currently, the group is in its Ideation stage. We are working on solutions and selecting the best ones. LOVE Miro!

View post on imgur.com
--- `10.07.2020 19:24` 12.07.2020

Pitching Efficiency Solution (Video)

 

  1. The Problem

Why is Efficiency the weakest link in evaluation? 

While those willing and able to make an impact need to be accountable and report back on what was achieved with their money!

Those in need have very few resources to set up M&E systems capable to explain how resources were transform into results. They barely can manage to count their beans

How can we, the Evaluation Community, help them to close this gap? How could they have a clear guide to choose the right approach? How could they agree on valid data sources per sector and donor? How could they easily access these data sources?

Let’s make it available

Let’s keep it simple

Let’s be practical!!!

….. and attainable!

 

  1. Our Solution
  • What’s unique about or prototype: This is a tool that helps finding the right approach of efficiency measurement
  • Every sector, every donor, institution and size of investment require a different approach. This tool will help you to navigate this process and make a decision for you. The tool could be presented as a feature easily accessible in the CoP or even an App!
  • This will allow practitioners:
    • Chose the right methodology
    • Access practical tips by sector
    • Know exact donors’ requirements
    • Help create reliable and up to date datasets by county and sector
  • The tool will be:
    • Created and updated by the community through
      • Crowd sourcing tested solutions
      • Feedback mechanism by voting system of practitioners
      • Validation and curation[1] by the Board of Experts and representative sample of practitioners. Curation could be done through five models: aggregation, distillation, elevation, mashup and chronology[2].
  • How will it work:
    • Simple application
    • Easy to use decision trees that guide users depending on their needs, suitable solutions for different barriers to efficiency measurement and different problem areas of the measuring for different types of stakeholders (NGOs, donors, etc) and different types of projects (capital infrastructure versus small campaigns, etc.)
    • A comparative advantage of the tool is a shortcut solution to a long consultation process with multiple stakeholders from various sectors and institutions.
  1. Our Target Stakeholders:
    • Established Evaluation Community of Practice, who could champion the process of content development and curation. For, example, Better Evaluation.
  1. The Business Model:
    • Subscriptions to be included roaster of experts.
  1. Stakeholders Acquisition/Adoption:
    • Champions from Evaluation Networks, Think-Tanks, Evaluation Influencers, some of which are in the Curation Board, to support dissemination and communication.
  1. Our Team:
    • Hanan Kwinana, Eloisa De-Villalabos, Adeola Awogbemi, Zeina Fahed, Traore Alladari and Alena Lappo.

Join us in this Efficiency solution!!!

 

 

[1] Content Curation is a term that describes the act of finding, grouping, organizing or sharing the best and most relevant content on a specific issue

[2] https://www.rohitbhargava.com/2011/03/the-5-models-of-content-curation.html

--- `12.07.2020 22:52`

More Resources from the "Think Evaluation" Team

Which Efficiency Methods defines your Institution?

 

What best defines your Institution:

  • Development Agency/UN
  • MDB
  • Donor
  • NGO
  • Government
  • Other

In which sector are you looking to measure efficiency?

  • Infrastructure Development
  • Agriculture
  • Health
  • Education
  • Nutrition
  • Governance
  • Energy
  • Capacity Building
  • Natural Resource Management
  • Other

In which region/country do you operate?

  • Country/region drop down menu

What is the size of the investment/project you intend to assess?

  • USD 0 - 50 000
  • 50 000 - 100 000
  • 100 000 - 500 000
  • 500 000 - 1 000 000
  • 1 000 000 - 3 000 000
  • 3 000 000 - 10 000 000
  • 10 000 000 - 50 000 000
  • 50 000 000 - 100 000 000
  • >100 000 000 USD

How long is the project life?

  • Less than 1 year
  • 1 to 2 years
  • 2 - 4 years
  • 4 to 6 years
  • More than 6

How complex is the project

  • 1 component
  • 2 to 4 components
  • More than 4

How strong are your existing M&E systems/capacities?

  • Very strong
  • Strong
  • Weak
  • Very weak

Does the project has a specific focus on Gender

  • Yes
  • No
--- `12.07.2020 23:07`

See our very hot.....Flow Chart

min/embed.js" charset="utf-8"></script>


13.07.2020 08:21

4 Interview sessions by "Think Efficiency" Team

Interviewee: Mariana Branco Economist and Evaluation Specialist at The World Bank Independent Evaluation Group (WB-IEG), where she conducts evaluations and helps promote evaluation capacity development (ECD). Mariana has been supporting multiple programs such as The CLEAR Initiative, IPDET, and gLOCAL Evaluation Week. She is also the focal point of a new WBG-UNDP Global Partnership for Evaluation. She was also an independent Consultant.

What are the three main efficiency issues / concerns from your viewpoint?

  1. Conceptual theory and approach to the project design to be evaluated. Without a good Theory of Change, or good outlines and outcomes, it’s hard to build an efficiency model. This is one of the highlights of the study on Efficiency Measurement by the World Bank because when teams cannot build properly Theories of Change and conceptual frameworks, then you face difficulties in efficiency measurement.
  2. The second point is not focused on projects but institutions like the UN. There is a lack of incentives from a decision-making point of view to imbibe more efficiency analysis in the work programs of evaluation offices. There is no internal demand for those evaluations that will produce cost benefit analysis.
  3. Measurement of cost incentives are sometimes overlooked. When cost benefit analysis is applied in NGOs or smaller organizations, accounting wise, there is no sufficient information to appropriately derive cost per activity and account for them properly. On the benefit side, there is need to value intangible outcomes but this is an intellectual challenge to do that properly. We should expect to see more advances in the future on benefits monetization. There is also data constraint because to be very accurate in measuring efficiency, we need to have databases at national level as well as organize and have partnerships for benefits evaluation that are not yet in place. This will determine how further we want to go in the future in terms of monetization but there will be need to set up technical instruments at the country level.
  4. Used as a tool to approve financing. Not used as a tool for decision making or learning, it is just an investment decision.

How can those issues be addressed in an innovative way?

  1. Equip teams with theory-based impact evaluation knowledge from the scratch and be clear that in the world of efficiency analysis, there are people from various backgrounds of social accounting, development corporations, etc, because it is a multidisciplinary field. So, it is important for everybody to be in tune with the pathways, results framework, theory of change, logic framework, etc. and the models they build for their own projects are well designed before even the projects starts.
  2. On the issue of incentives, it is much bigger than us. There is need to have open conversations on why institutionally, boards of organizations are not demanding for it. This is more around a cultural study on trying to get to the root of incentive issue. One interesting solution will be to conduct a study among evaluation managers and board members of multilateral development banks, governments and international agencies to understand the issue of culture and how to address it.
  3. On cost size, when talking to smaller organizations, ensure that procedures must be in place to account cost properly and not rush to benefits measurement so much that they undervalue cost and put down whatever denominator they choose. On the valuation side is a most promising area where there can be serious partnership between governments to generate data on financial purchase. This is a large-scale study where there will be partnerships between institutions or countries to conduct studies or create systematic ways to review financial purchases.
  4. Connection between impact evaluation and efficiency analysis goes hand in hand and links to the multidisciplinary aspect where there will be need to create integrated capacity development projects that teaches impact measurement, impact evaluation, outcomes measurement, etc.

Take away

  1. Value for Money is mostly used for financing at project start but rarely used for evaluation work.
  2. There are a number of Company of Practice networks but no coordination among them. There is a Cost Benefit Analysis Association in the US and now has a branch in Europe.
  3. Replication of standards at micro levels in different countries is a major challenge.

 

Interview with Mr. Osvaldo Feinstein former Manager, Operations Evaluation Department (now IEG), World Bank professor of evaluation at the Master in Evaluation, Complutense University, Madrid and member of the Advisory Panel of the UNDP Evaluation Office

  1. What're the three main efficiency issues / concerns in evaluation from your viewpoint?
    1. Conceptual issues. focus on static efficiency, lack of concern with dynamic efficiency
    2. Data issues: deficiency of accounting systems, too limited relevant info on costs
    3. Demand issues: lack of demand for efficiency measures, even at MDB
  1. How can those issues be addressed in an innovative way?
    1. Apply dynamic evaluation (see chapter 2 of 2019 IDEAS book)
    2. Promote activity based costing
    3. Conduct brief evaluation workshops for high-level policy makers focusing on efficiency and effectiveness.

 

Interviewee: Avinesh De Silva -Senior M&E Specialist at SIMPLE project USAID Egypt

  1. Opinion on the ideation notes shared from Miro
  • The approach seems like developmental evaluation, but there are different ways on doing it.
  • Not sure how easy it is to have one approach, because it depends on the organization.
  • For example:
    • For DFID, efficiency could mean a different thing; there isn’t only the programmatic aspect but also the political one.
    • For UNDP, there is more focus on development and the policy/institution outcomes.

 The methodology

  • Using mixed methods for monitoring and looking at how systematic qualitative data can be collected and how a program is used and incorporate it into decision-making e.g. The Most Significant Change (MSC) or Outcome Harvesting (OH), etc.
  • What is the end goal? Because a single approach is not necessarily optimal.
    • Eloisa: the goal is to build something that help practitioners and donors to work within a framework (that is dynamic) to report and evaluate programs by sector.
  • So here you are creating a new resource for organizations.
  • But the donor requirements keep changing.
  • The broader message is the outputs.
  • Help organizations understand the overall mission and their donor, help build outcomes.
    • For example: how many participants in your program did something with it? (was the program WASH, health, agriculture, etc.)

 

 

Interviewee: Michael Cooper – independent Evaluation Researcher and Evaluator

  • Suggested: the solution has to be a robust cost-benefit analysis methodology suitable for all stakeholders & fits all capacities
  • Biggest barrier to measure efficiency is to how would the solution fit all types of investments or projects; huge infrastructure projects have standard guidelines for measuring efficiency, what about other sectors?
  • Smaller projects implemented by NGOs/ CSOs are the primary beneficiaries of this Hackathon
  • Practitioners developed various tools but figuring out which tool /methodology works best for you is a very common problem.
  • Too many tools in the landscape makes it overwhelming …. PITCH: How can we help you select the right tool.
  • The New Tool Parameters: select a sector, available budget, level of rigor, staff capacity, beneficiary, etc. come up with a method to measure efficiency
  • Donors are difficult to change as politics can’t be omitted …. Develop something SIMPLE
  • Decision trees can be designed with a Yes/No scenarios – Developing and APPLICATION is a good idea in terms of creating ACCESS – a great concept but APP need to provide the solution.
  • Practitioners and subject matter experts need to come together in a ‘5 years’ consultation process’ to develop the Decision Trees … this can be an APPLICATION or a STAND-ALONE PLATFORM
  • CROWD SOURCING can be used to build t populate the APP but AUTHENTICATION / VALIDATION is needed. Validation can be done by setting a voting mechanism among community of practitioners
  • CROWD SOURCING should be coupled with CURATION (5 methods of curation to choose from).

 

 

 


06.07.2020 09:25


13.07.2020 12:39


13.07.2020 12:40


13.07.2020 12:40


13.07.2020 12:40


13.07.2020 12:40


13.07.2020 12:40


13.07.2020 12:40


13.07.2020 13:16

Launched at Evaluation Hackathon by

jeremiah_ipdet zeina_fahed Adeola_Awogbemi traore_alladari hanan_kwinana alena_lappo eloisa_de-villalobos eze_hyginus

Maintainer jeremiah_ipdet

Updated 13.07.2020 13:16

  • eloisa_de-villalobos / update / 13.07.2020 13:16
  • jeremiah_ipdet / update / 13.07.2020 13:02
  • Adeola_Awogbemi / update / 13.07.2020 12:40
  • Adeola_Awogbemi / update / 13.07.2020 12:40
  • Adeola_Awogbemi / update / 13.07.2020 12:40

Measuring efficiency

Measuring efficiency in evaluations

Evaluating efficiency is weaker and not substantive enough compared to assessing effectiveness, sustainability and relevance in most project, programme and policy evaluations. What are ways of assessing efficiencies?

All attendees, sponsors, partners, volunteers and staff at our hackathon are required to agree with our Code of Conduct. Organisers will enforce this code throughout the event. We expect cooperation from all participants to ensure a safe environment for everybody.

Creative Commons LicenceThe contents of this website, unless otherwise stated, are licensed under a Creative Commons Attribution 4.0 International License.