According to the proposers, in order to grow auditability, there is need to evaluate the implementation result of Catalyst’s funded proposals with systematic framework. this definitely addresses the issue of auditability an da clear framework that is workable to the proposers will be of great help to improve auditability

This proposal has ambitious plans for auditing (ballpark: 200) a larger number of proposals but limits this to Fund-7 and Fund-8 F7 and F8 top 30% of funding in a selection that is called "Community Development" including 15 challenge settings for those 2 funds. Instead of using the existing communities growing around Town Hall, Swarm, Circle, popular Telegram channels for Proposers, CAs etc., it includes a lot of "branding" for the auditors, namely a DAO in 2 years, and several styles like P.I.E. under the umbrella of an existing multi-chain crypto website. This then requires a lot of resources to be expended on liaison with other efforts across the two communities. While on the surface this has the potential to achieve several of the goals set in the challenge, it seems to be a poor use of funding and should have been proposed with a more selective scope to produce a strong proof-of-concept. Consuming about a sixth of the total challenge budget and assessing over 200 proposals over a period spanning much longer than those proposals will be in their implementation phase will result in relatively less "bang for our buck (or Ada)". Therefore, although relevant to the challenge setting, it may not have a good return in terms of impact on the investment.

One of the most important areas of passing self-governance from IOG to Project Catalyst, mostly because it is such a pressing issue, is in the auditing of funded proposals. This Proposal correctly identifies the issue: newly funded proposals are increasing rapidly, while previously funded proposals are slow to reach completion and be removed from the auditing workflow. The proposer is submitting here a prototype that can operate alongside the existing auditing protocol, and if it proves itself, would be able to take a portion of the auditing load onto itself in any given fund, if not all of the auditing. This is a highly relevant area of focus for this challenge, and prototypes like this are a necessary first step before such a critical step of the funding mechanism can be handed over. If successful, the project is scalable with the addition of auditors, and the only step left would be the integration of the audit results with the Treasury, in order to continue to release funds until projects reach completion. The technique being used here (labeled as PIE) uses well known conventions around the ideas of Conformance evaluation and Performance evaluation that delivers a project grade in several areas that is compared to a reference index that the proposal team builds out in relation to the challenge setting metrics and goals. Exploration and execution of these reference indexes is as important a part of the project as any, since they determine the quality and success of the auditing process.

This proposal can be summarized as “proposing the PIE (Proposal Implementation Assessment) framework with criteria and indicators to assess the compliance and performance of proposals financed by Fund 8”, where in this structure “they are used to assess the degree of compliance with the goals, commitments, KPIs/metrics established in the proposal”. Theoretically, this proposal fits perfectly with the specific objectives of the challenge, from the moment it proposes to create this new method of auditing proposals, based on the aforementioned indicators. Proponents argue that as funds go by, the number of proposals being funded grows too high and it is difficult for IOG to fully audit them. I agree with this preposition, because that is precisely why this challenge was launched, and I think that coming up with this P.I.E structure is a good way to solve the problem. However, I think the bidders could have done better to explain how the audit would happen in practice, how this structure would be used to evaluate the bids. In my view, this could become clearer if they, for example, took some proposal and applied the audit through this structure. Thus, we would see how this would happen in everyday life, and we would better understand the strategies used. Therefore, my conclusion is: this proposal addresses very well what we are looking for in the challenge, as it suggests a more standardized and probably more practical way of auditing the growing number of proposals financed by each fund, but from the explanation provided in the detail of the proposal, I cannot solidly visualize how this audit would be done, even though they brought up many issues that would be evaluated in the “specifically, the implementation of this proposal will contribute to the success of this Challenge in the following KPIs” section. With that, the impact is a little affected, I still believe it's a good idea and that it will work, but because I don't understand exactly how, I don't feel comfortable giving full marks to this topic of analysis. I would also like to know how the auditing of the proposals that they encompass as “community development” would be different from the auditability they would do, for example, in the game challenge. What are the specifics here?

This proposal would be implemented over more than a year and has a lot of ambitious goals how to create a framework and even ultimately evolve into a DAO, but does not present good auditability metrics which is unfortunate in this category. After over a year after funding, only 2 videos will be created and 8 reports issues. There is no breakdown of the milestones, targets or metrics prior to the end of the assessment phase, and the proposal itself will be very difficult to audit over check-ins and its own audit for the long-term disbursal of funding. Social media views of the final report are optimistic and should not be the main objective of an audit. No discussion of possible qualitative goals except a survey are provided, and it is limited to Fund-8 proposers only and excludes other stakeholders.

The project is focused on growing auditability by evaluating the implementation result of Catalyst’s funded proposals with systematic framework. The work is to be done on a clear a clearly outlined framework

How does one audit the auditer? For starters, the deliverables are 4 quarterly reports, starting with a first report that writes up all the indexes and protocols used, as well as the makeup of the first cohort of funded proposals. So every 3rd progress report should have a deliverable associated with it. Presumably this same method and protocol could be used to issue the monthly KPI reports as well, in the form of % completes towards the quarterly reports and number of funded proposals onboarded and tracked. The final deliverable for the project, a video report summarizing the results of the prototype and lessons learned, including the disposition at that time of the audited proposals that collaborated (which may have interesting results compared to their peers), also will able to be used as the final completion requirement of IOGs current auditing protocol, so that is one less thing the team will have to work on to reach completion status. The backing teams for the proposal are well known in the community, and their support will no doubt enable the leads to effectively stay in touch and stay on track. In addition, due to the nature of needing to network with proposers, they will probably be required to maintain a public communications channel which could be another means of maintaining transparency through this process.

The auditability of the proposal is good, but I don't think it's ideal. The team is concerned with providing various KPIs and statistical data that can serve as material to analyze for measuring progress. However, how will this data reach the community? It is not feasible that such things are defined and that they do not have a direct path to reach the members of the Cardano community. Therefore, I think it would be necessary to think of ways to make such information always available and updated, in addition to being visible to us. The team can create some online tool, such as Miro, Notion, Google Docs, etc., that serve to host such metrics and make them reachable for members of the Cardano community. Updates could be made monthly or weekly, and these directly posted there and forwarded to the community through Discord, Telegram or other communication channels we already have. Regarding success and final results, I believe that more or less the same thing happens: we know the ideal scenario to be reached, but we don't know how we could visualize it. In this case, as the information is more extensive and requires a further level of detail, I think it would be necessary for the team itself to contact the community in order to summarize everything that has been achieved. This can be done through synchronous online meetings, participation in Town Hall, for example, or even videos recorded and forwarded in the community media that I have already mentioned. In this way, I believe that auditability would be better explored and ensured, and that we would have more control over project monitoring.