P.I.E for Miscellaneous-Challenge

Propose & apply the P.I.E framework to assess the result of Miscellaneous-Challenge’s funded proposals (with various contexts) in F7+F8
It’s very hard to evaluate the implementation of funded proposals in “F8: Miscellaneous-Challenge” without a systematic framework
Community Reviews (3)
Addresses Challenge
Does the proposal effectively addresses the challenge?
Very good proposal. it is well written and has a valid problem statement as miscellaneous challene funded proposals might be harder to audit in a coordinates way becuase there's no defined/delimited strategic goal for the challenge. At the same time, it addresses the challenge in kpis correctly indicated in "how your proposed solution will address the challenge" section.
The framework to be applied (PIE) is well explained and seems to be generic enough to work for miscellaneous proposals but not too generic that its hard to understand what is going to be made exactly. Hence, I think it can work, with minor adaptation depending on the project. it also covers the main dimensions in catalyst as assessing if it helps cardano to achieve its missions, community satisfaction with the solution and its usage besides other project management quality metrics (alignment with planned schedule, deliverables, perfomance and budget).
Its really key for this project to keep everything well documented and run lessons learned after its complete. This can be applied to other challenges and have the potential to create standard audits guidelines. Other thing that adds value to this proposal and increase its impact is that the final results will be avilable in various formats as reports and videos. This increases its audience and satify people who prefer one and the other.
Last, risk and challenges were identified in a very satisfactory way and they all have a startegy to overcome potential hurdles.
Assessment Quality Assurance
Assessment Quality Assurance is an offered role to veteran in the Cardano Project Catalyst Community. The purpose is to review PA assessments of proposals, providing a second layer of Quality Assurance.
- Assessor ID
- z_assessor_298
- Total QA Ratings
- 8
- QA Rating Outcome
The proposal starts from a true proposition: it is becoming increasingly difficult to properly audit all funded proposals. With the passage of funds, this number grows, and there are several teams, carrying out different activities, having different deadlines and objectives to meet, and if each one of them does not have its own well-defined auditability plan and, above all, that it is fulfilled with commitment and honesty. , there is no way for the Cardano community and IOHK to successfully monitor everything at the same time. Therefore, the proponents suggest auditing 50% of the proposals (in this case the main ones - I wonder, how would that be defined?) of the Miscellaneous challenge, through a created standard, which they call P.I.E (Proposal Implementation Evaluation). This standard would evaluate the proposals through pre-defined criteria, as they explain, which would be, for example, “the degree of fulfillment of the goals, commitments, KPIs/metrics established in the proposal”. In theory, this idea is great and it aligns very well with what we are looking for in this challenge, which is precisely to establish safe, faster and more practical ways to audit Catalyst-funded proposals. The problem here is that there are already other proposals from these same proponents suggesting the same thing, each one aimed at different challenges: one of them is aimed at a group they classify as “Community Development” and the other focuses specifically on the challenge of games. I think the team had a great idea to come up with a structure that will be used to analyze the proposals more directly and objectively, but it is necessary for them to explain the different approaches that would be taken between auditability in this case and in the other two cases of the proposals that they also launched here. Taking into account the specifics of each challenge, some focused more on technology (as is the case with this one), others more on the humanities (such as the challenges of Africa, East Asia and India, which focus on expanding Cardnao across these regions and helping the people there), what are the differences, even if subtle, in the way of monitoring these various challenges? Because of this key question, this doubt that permeates the reader when he realizes that there are two other proposals similar to this one (the links are: https://cardano.ideascale.com/c/idea/398309 and https://cardano.ideascale .com/c/idea/398308), it would be necessary for proponents to explain the need for 3 different proposals to execute their ideas instead of one that encompasses all challenges.
Assessment Quality Assurance
Assessment Quality Assurance is an offered role to veteran in the Cardano Project Catalyst Community. The purpose is to review PA assessments of proposals, providing a second layer of Quality Assurance.
- Assessor ID
- z_assessor_57
- Total QA Ratings
- 10
- QA Rating Outcome
This proposal team has submitted 3 similar proposals to roll out a prototype of an auditing mechanism for funded proposals, including a broader one for "PIE for COMMUNITY DEVELOPMENT" and another more narrow one "PIE for GAMEFI Sector" that is similar to the current one. I make the case in my assessment of the GameFi proposal that the community only really needs one of the three prototypes, and for me the preferred one that gets my vote is the Community Development one, as it covers the most ground. For these reasons I have lowered my Impact rating only on the other two proposals by one star each. However, I would like to note that the impact issues that I covered the Community Development assessment apply here as well: this is a critical area of attention for Catalyst that matches up well with the requirements of the Challenge Setting, so a prototype like this is valuable. My only concern is that in a limited fund it would be better to have multiple types of prototypes to choose from, as opposed to many versions of the same protoype. I encourage voters to also check the other two proposals from this team in this challenge setting, and to consider the assessments provided. And I encourage the proposal team to consider bringing at least a portion of the cohorts from each of these proposals under the umbrella of their protocols if they do not get all of these proposals funded.
Assessment Quality Assurance
Assessment Quality Assurance is an offered role to veteran in the Cardano Project Catalyst Community. The purpose is to review PA assessments of proposals, providing a second layer of Quality Assurance.
- Assessor ID
- z_assessor_1193
- Total QA Ratings
- 11
- QA Rating Outcome
Feasibility
Given experience and plan presented is likely that this proposal will be implemented successfully?
There's a clear time-framed roadmap with deliverables and acitivites in each period listed. Budget breakdown is embeded in it, with the total hour of each activity and the price of the hour, which makes it really transparent. There's also a sort of contigency line, for an unexpected increase in the number of proposals. The only thing that left a question here is when it says "Proposing P.I.E framework (…) in selected challenges". I thought the challenge was already selected (miscellaneous) and there would only one in the scope of this project. Or maybe this part was already done before this fund as it is in Q12022, but that could be indicated here. Anyway, this is not affecting my score.
Regarding the team, there are 2 committed people with enoguh description of their background and capabilities and they seem to have the competencies to successfully execute this project. there are also plenty of links to verify their credentials plus 2 other team partners who may help with necessary compelmentary skills.
Assessment Quality Assurance
Assessment Quality Assurance is an offered role to veteran in the Cardano Project Catalyst Community. The purpose is to review PA assessments of proposals, providing a second layer of Quality Assurance.
- Assessor ID
- z_assessor_298
- Total QA Ratings
- 8
- QA Rating Outcome
The proposal roadmap is very detailed, some time intervals are defined and with them we know the activities that will be developed in each period. My suggestion is that this division be done monthly in all cases (in the first three months this is not done) so that we have a more specific view of the process, so that we can understand exactly how long each thing would take. For example, the first 3 months would be for building the detailed picture of this analysis structure, P.I.E. But why would that take 3 months? What would the work routine of each professional be like during the first month, and the second, and the third? As for the rest of the time, it's explained better, we know what would happen each time, we have punctuated task topics that richly explain what would need to be done. Regarding the budget, this part is described together with the explanation of the roadmap, so that we can clearly understand how the requested money would be used. Generally, in the proposals, we do not see these two things being explained so closely, I liked this idea and I think it causes a more joint understanding than dividing the parts into blocks that do not dialogue, after all, everything is part of the same process. And finally, speaking of the team, this part of the proposal is great, it is a duo that provides a lot of information about who they are: their names, various social media (I usually suggest that at least the linkedin of those involved be attached, but the more information the better, as we can delve deeper into who these people are in different facets), it is also explained in topics about their previous professional experiences. With this, we have a very complete picture of who we are dealing with, and a lot of confidence is transmitted. The only observation I would like to make is that it would be interesting if they clarify the role of one of them in the project, specifically, what tasks would they perform here? And another thing, they say they have strong partnerships and they name the groups, but it would be interesting if they clarify what helping role these partners would play. Finally, the feasibility of the project is very good.
Assessment Quality Assurance
Assessment Quality Assurance is an offered role to veteran in the Cardano Project Catalyst Community. The purpose is to review PA assessments of proposals, providing a second layer of Quality Assurance.
- Assessor ID
- z_assessor_57
- Total QA Ratings
- 10
- QA Rating Outcome
Unlike their other two proposals, the subjects of the Misc challenge may be neither low in technical requirements or suited to a particular interest or expertise the team possesses, and as such, they may be harder to assemble accurate audits on. Of the three proposals, this is the one I have the least amount of confidence in when it comes to creating a solid auditing index and protocol. As mentioned elsewhere, the critical skills needed here are not the obvious strengths of the team and its backers (programming, development, data engineering) but rather softer skills like collaboration, communication and clear, concise report writing. The team leads have some relevant background in this area, so hopefully they are prepared for the difficulty of managing and incentivizing cooperation with the funded proposal teams. I will repeat a concern I have with an otherwise reasonable budget for these proposals: the inclusion of a "price slippage" line item makes the overall budget less competitive then it could be in the challenge.
Assessment Quality Assurance
Assessment Quality Assurance is an offered role to veteran in the Cardano Project Catalyst Community. The purpose is to review PA assessments of proposals, providing a second layer of Quality Assurance.
- Assessor ID
- z_assessor_1193
- Total QA Ratings
- 11
- QA Rating Outcome
Auditability
Does the proposal provides sufficient information to assess and audit progress and completion?
There are very good kpis, all with target and some of them time-framed. They are really SMART, in general, which is awesome. They all make sense to evaluate if project was fully and succesfully implemented/done or not.
There are also good milestones listed, in the definition of success part, that helps to follow up with project progress and check if its on track.
It's also imporant they'll collect feedback from the community on the final results. This promotes engagement, attest quality and increase awareness on auditability as a whole.
This project is transparent and seems to be really auditable;
Assessment Quality Assurance
Assessment Quality Assurance is an offered role to veteran in the Cardano Project Catalyst Community. The purpose is to review PA assessments of proposals, providing a second layer of Quality Assurance.
- Assessor ID
- z_assessor_298
- Total QA Ratings
- 8
- QA Rating Outcome
The 4 deliverable reports, delivered quarterly meet part of the demand for auditability for the project, but the team will need to develop some % complete or other deliverables in order to meet the required monthly progress reporting. I will suggest again that monthly contact with the funded proposers could both create monthly audit tickets as well as provide incentive for funded proposer participation with this audit by sharing those tickets to meet the funded proposer's own reporting requirements. Everybody wins. The creation of a video to sum up the project is also a good idea as it will allow them to submit it as not just a deliverable but as the completion report required by the IOG auditing protocol. The needed channels to onboard funded proposers can be made public so that the community will also be able to keep track of the project.
Assessment Quality Assurance
Assessment Quality Assurance is an offered role to veteran in the Cardano Project Catalyst Community. The purpose is to review PA assessments of proposals, providing a second layer of Quality Assurance.
- Assessor ID
- z_assessor_1193
- Total QA Ratings
- 11
- QA Rating Outcome
The ways of monitoring the project that are suggested by the proponents are certainly good, but they can be deepened. Let's separate it into two parts: process and final results. Regarding the process, they are already willing to do what I usually suggest, a report that summarizes the activities that have been done so far, and even more would have a video that summarizes this report, making the information described easier to access. . However, they suggest that this document be prepared quarterly, when I believe that this time interval is too long, many things would be added to be reported, both small achievements, as well as the progress of the work routine, possible problems faced on the way and etc. The report would likely be long and tiring to read, so people would jump to the video. I think it would be more appropriate if this report were made available monthly, for example. This would make its size smaller and it would even be easier for the bidder pair themselves to organize it. One thing that was a little in doubt was how this information would be forwarded to the Cardano community. Would the report be emailed, posted on some Discord channel? Would the video be available on youtube and would its link be available in the report itself? It is necessary to think about how this bridge will be established. Now speaking of final results, the main objective here would be to audit 50% of the funded proposals for the challenge in question, but how would the quantity be verified? In the final report, it would be necessary for the bidders to include the link of the bids they audited, and for them to be organized in a numerical list, right? It would also be important, in addition to knowing how many people had access to the final report and its video, to understand how the adopted P.I.E structure benefited the auditability of the proposals. How can we see this difference in practice? Could the teams of some of the funded proposals be part of this video, reporting what has changed? How are they being evaluated, and how has this impacted your work? Anyway, I believe that yes, good forms of auditability are suggested, but that this should be done even more specificity and variability of approaches.
Assessment Quality Assurance
Assessment Quality Assurance is an offered role to veteran in the Cardano Project Catalyst Community. The purpose is to review PA assessments of proposals, providing a second layer of Quality Assurance.
- Assessor ID
- z_assessor_57
- Total QA Ratings
- 10
- QA Rating Outcome
No comments yet…