An open source platform for evaluation data sharing for all stakeholders. Join us in the democratization of evaluations.
⛶ Full screen
Maddison Staszkiewicz, Camila Meza-Cuadra, Cristina Cibin
How do we obtain transparent, methodologically solid, rigorous evidence in an efficient way to showcase understandable, marketable impact in a competitive, changing environment with various stakeholders?
Quri: The Solution
There is “good-enough” evidence for an evaluation when:
- Beneficiaries of the program validate the findings. From the beneficiary perspective, the changes documented in the evaluation results must be truthful, tangible and accurately reflect reality. We believe this is the only way to achieve good-enough evidence for an evaluation. Participation of beneficiaries in the stages participatory evaluation varies, though there is often a focus on the data collection phase (Sette, 2018). The beneficiary participation should include participation at all stages of the evaluation, including the dissemination and validation of evaluation results.
- Evaluations of similar projects have already been done and the findings are consistent. Similar projects done in similar contexts can provide an indication of whether a project is likely to have an impact and bring desirable social and behavioral change. Even though every situation is unique, access to results of different evaluations can ensure good-enough evidence in evaluations. We propose a democratization of the evaluation process that consists in bringing together all stakeholders involved, as well as the general public, on an open-source, online platform where everyone can contribute to the transparency and the optimization of evaluations done across the world.
Quri is an open-source, online platform (website and phone application) that meets these criteria. Michael Quinn Patton states that, “Intended users are more likely to use evaluations if they understand and feel ownership of the evaluation process and findings [and that] [t]hey are more likely to understand and feel ownership if they've been actively involved” (“Utilization-Focused Evaluation”, 2020).
This is relevant in times of normal data collection, remote data collection, and times of crisis such as the COVID-19 pandemic. The ability to view data in an engaging way, share full access with beneficiaries, donors, and other stakeholders, and engage with data from other projects in similar contexts will revolutionize evaluation work. Quri also provides a way for organizations to indicate that they are in need of extra help with a project. As the trend of online volunteering continues (Millora, 2020), this is a way for evaluation experts to continue growing their portfolios while supporting projects in need of extra assistance.
The Four Main Functions:
- Impact Explorer
- Capacity Building
- Evaluation Microwave
- Community Building
The Impact Explorer allows the entire public, including all stakeholders of evaluations, to gain an in-depth look at the data that has been uploaded to Quri. Data can be explored by location, theme, funding organization, implementing organization, and indicator. Once you identify an evaluation of interest, you can view it’s individual page with dashboards of impact data that can be customized. Viewers can upload photos and comments on the page available to the public and “flag” data that is incorrect and needs to be updated.
Capacity building is a continuous theme in evaluation that affects multiple goals, sectors, and outcomes (Evaluating Capacity Development Results, 2016). As part of Quri, capacity building will include tutorials, report templates, COVID-19 resources, webinars, and more. The capacity development is focused on evaluators and organizations to improve their evaluations and on beneficiaries who are using the platform, that have historically not often been included in the result-sharing process of evaluations (UK Aid, 2015), and are gaining experience in understanding data in a new way. This is found in the Tools and Resources section.
The tool will leverage available data from other interventions to come up with projections or predictions on the sort of impact an intervention could potentially have on key indicators. It would be helpful to provide insights on the kind of impact an intervention that has not yet been evaluated could have. While the results would not be valid for scientific purposes, they would be useful for practical purposes and to help decision makers make better investment decisions with the limited amount of resources available, especially in the critical situation created by the COVID-19 pandemic.
The global community of evaluators is large and looking for ways to get involved as evidenced by the IPDET Hackathon with over 500 participants from more than 100 countries represented. Quri will provide an opportunity for evaluator involvement in specific projects as requested by the organizations in order to continually improve the evaluation community at large. There will also be a Q&A Support function for questions where in-depth support and assistance is not required.
Quri will be available in multiple languages.
- Stakeholders are literate, have a basic level of computer literacy, and are able to access Quri. If they are not, they are able to access it through the support of a proxy (friend, family member, etc.). As of 2020, there are an estimated 3.5 billion smartphone users in the world (O’Dea, 2020).
- There is time and willingness of the organizations to enter their data in Quri and share it with the public. By offering features such as easy to use and customize reports and access to exports, we believe that organizations will be interested in using Quri to analyze and showcase their work and transparency.
- Beneficiaries are comfortable confirming and/or challenging data that is shared by organizations, governments, and evaluators. They will have the opportunity to only share their personal information with Quri administrators for privacy reasons.
Level of innovation: Combining an open-source platform for evaluation results across sectors and giving beneficiaries the option to flag discrepancies in the data presented is a unique innovation. Adding the option for organizations to select if they would like to seek support from an expert evaluator adds additional innovation as it provides a way for evaluators to easily identify projects that they could work with, either voluntarily or as a consultant.
Scope of impact: Quri is an open-source platform that is intended to be accessible to all. There is no exact level of evidence required for data as it is determined by each specific project and its needs, however, the ability for transparency is crucial. Beneficiaries have often been excluded from the evaluation results sharing process and are uniquely positioned to confirm or critique the showcased data. By offering features that organizations and evaluators are interested in, there is motivation for all stakeholders to participate.
Viability: After taking into account the various limitations, Quri is determined to be a viable solution. Feedback has been gathered from stakeholders through interviews throughout the process.
Social value: This provides significant social value to beneficiaries, who are often disadvantaged and are not given as much of a voice in evaluations. In addition to sharing all of the data about the project with the public and beneficiaries, beneficiaries are able to critique and request updates of any data through Quri.
Collaboration and commitment: In an effort of collaboration and commitment, we have made our documents available to the public. Please find our project notes in the source link below and add comments on this page with any feedback or additional ideas for consideration.
Quri: The Team & Work Process
The Milestones, Achievements and Prototype
We used the Design Thinking process to guide our work throughout the Hackathon. Following this process, our milestones were creating a stakeholder map, interviewing stakeholders, redefining the problem, our ideation session, and prototyping. The ideation session was our greatest achievement as a team as we realized that we had many different ideas that would be stronger together as one overall idea. From this ideation session, the open-source platform, Quri, for evaluation results and visualization with the ability for beneficiaries to engage with the results was born.
The diverse group of stakeholders of evaluations has varying interests and requirements in terms of the levels of evidence and transparency required. Identifying the pain point for all of the stakeholders to define a problem that we could solve was the most time consuming and challenging part of the hackathon. Once we had the idea, prototyping was also challenging to keep the scale reasonable while recognizing that Quri needs features for it to be attractive to various stakeholders in order to gain sustained participation and buy-in.
The Takeaways and Lessons Learned
Taking one week to solve a problem the evaluation community is facing is no easy feat. Collaboration with and inclusion of stakeholders throughout the entire process is required to ensure that the prototype and proposed solution will provide value. It is easy to think big picture and get excited about the possibilities of what could be, but to be actionable in times of COVID-19 where decisions must happen faster and stakeholders want to be sure programs are making an impact, we must ensure the data is available to everyone as efficiently and transparently as possible. We believe Quri meets our goals.
The Next Steps
Quri needs to be developed which requires a timeline of 6 months for the implementation and rollout of the process.
Specific needs include:
- Talent and organizational building:
- Attracting and hiring a starting team including: 1 CTO, 1 product owner, 1 data scientist, 1 data architect, 1 data engineer, 2 commercial experts, 2 relationship managers
- Defining an organizational and governance structure
- Technology and data development
- Data architecture and governance
- API guidelines
- Visualization tools
- Marketing and public relationships campaign
- Go-to-market strategy design
- Campaign implementation
- Regulation compliance
- Regulation roadmap definition
- Pilot and launch
- Pilot phase with key users
- Open access launch
Quri means gold in the local language in Peru. The Incans used gold to measure value or worth, which is the goal of this project. We want to measure the worth of a program through evidence.
Innovations for Poverty Action. (2016, February 28). Impact Measurement with the CART Principles. Retrieved July 11, 2020, from https://www.poverty-action.org/sites/default/files/publications/Goldilocks-Toolkit-Impact-Measurement-with-CART-Principles_1.pdf
UK Aid. (2015, February). Beneficiary Feedback in Evaluation. Retrieved July 12, 2020, from https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/428382/Beneficiary-Feedback-Feb15a.pdf
Contact the team
This week was an amazing experience full of collaboration and sharing ideas that we would not have come up with on our own. With only 6 hours to go before submission of the final pitches, we are finishing up the final touches and finalizing our work. Thank you to everyone that made this Hackathon possible!