- Propose P.I.E (Proposal Implementation Evaluation) Framework with criteria and indicators for evaluating the conformance and performance of Gamers-On-Chained’s funded proposals (F7 + F8).
- Apply the proposed P.I.E for evaluating funded proposals for the 100% of funded proposals in Gamers-On-Chained’s funded proposals (F7 + F8).
General Framework of P.I.E consists of
[A] Conformance-based criteria: are used for assessing the degree to which the goals, commitments, KPIs/metrics set forth in the proposal have been achieved.
- Input:
- Conformance of time: percentage of actual implementing time over the expected implementing duration
- Conformance of finance / budget usage:
- Percentage of actual total budget usage over the expected budget requested.
- Average deviation of expenditure items from commitment
- Activities: Conformance of activities taking place as commitment (if available)
- Output: Conformance of products/services released as commitment
[B] Performance-based criteria: are used for assessing the degree to which the implementation results of the proposal have been achieved in supporting the given challenge, Cardano's mission, principal pillars, and annual related goals.
- Outcome:
- Community usage after operation (numbers of people, members, views, downloaded,…)
- Community satisfaction (Likert Scale Survey)
- Impact:
- Impact to the achieving of Challenge's goals/KPIs
- Impact to the achieving of Cardano's mission, principal pillars, and anual related goals
[more detail are described in attachment]
Firstly, the number of funded proposals in each funding round is increasing significantly, in detail, the average growth rate of funded proposals from Fund 2 to Fund 7 is about 20%, and the number of funded proposals is expected to exceed >350 in Fund 8 and >450 in Fund 9. This is a challenge for IOG for evaluating the implementation results and progress of these funded proposals, especially in the future. As a result, if this situation continues, the questions of conformance and performance of Catalyst Fund may rise.
Therefore, implementing this proposal would contribute to the purpose of
- Reducing the increasing load on IOG;
- Helping to identify the implementation status, problems (if any) of the Catalyst funding process, as a whole, and in Fund 8, in particular, and, as a result, implications for solutions in the following funds can be created;
- Reports published through this proposal may serve as reliable documents for decentralized representatives (dREP) as well as other institutions in decision making.
Specifically, the implementation of this proposal will contribute to the success of this Challenge in the following KPIs:
- “Number of funded proposals audited”: since auditing is a part of our P.I.E framework, through this proposal, we aim at auditing 50% of funded proposals belong to the challenge of “Gamers-On-Chained”.
- “The number of reports, articles, and videos/podcasts on Performance, Achievements, Financials”: by implementing this proposal, we aim at:
- Writing 08 reports for evaluating funded proposals in the challenge of “Gamers-On-Chained” challenges (F7+F8) in the process of implementation [6 months, 9 months, 12 months], and after the proposal finished its implementation.
- Producing 02 videos [uploaded to the Cryptoviews’s Youtube Channel] for summarizing the Final Reports.
In the phase of "on going evaluation", the 1st, 2nd, and 3rd reports are public and realtime that community members can give feedbacks, share their views on the proposal's results and our evaluation.
In the expected final reports for each fund, besides the implementation results evaluated by using P.I.E criteria, there are spaces where all the views and comments (consensus/not consensus) from the community’s different groups [including the challenge teams, the proposer(s), and other communities members] can be noted.
There are 05 main risks we may face, including
- The barriers of connecting and getting specific information from the IOG, F7's and F8’s funded proposer(s): transaction history for financial auditing, for example. For dealing with this, we use the first month to work with all the relevant stakeholders (IOG, funded proposers,…) to ensure that proposed indicators are (i) suitable for the challenges, the funded proposals; and (ii) collectible and measurable.
- Funded proposal's KPIs/metrics and ROIs are not always clear and measurable. To deal with this, our team place a 1 month period [preparation phrase] from the beginning when we work with funded proposers, challenge team for QA, completing the P.I.E framework for related proposals. We also survey experts, from different fields related to the sector of Community Development (including: from vCA, dREP, SWARM,…).
- Conformance & Performance evaluation of plans/proposals are controversial in certain points. For getting consensus on this, the surveying, working with various stake holders in the community are very important. That's why we placed the 1st month of preparation
- The number of funded proposals in F8 is greater than expected [expected 30%]: we place a financial reserve for mitigating this risk.
- Flip page of ADA price: we place a reserve as mitigation for dealing with price volatility (30% of working items listed)