[IMPACT] Please describe your proposed solution.
In the first proposal of Koios made under round of Fund 6, we were not fully sure about the adoption - especially looking at an ecosystem filled with well-established entities, but we still wanted to showcase and enforce the need of not just using open source, but going beyond and inheriting the idea of decentralization by fully utilizing open source frameworks.
To our pleasant surprise, we received a lot of participation (we have already awesome community members who have independently contributed SDKs, CLIs, and integrations) and appreciation from the development community, in return - this makes us go an extra few miles to ensure we live up to their expectations resulting in thorough coordination and preparation for base query layer (gRest) to be able to serve the absolute maximum of information there is from the chain, resulting in serving billions of requests over past months and a steady increase in adoption curve without putting as much effort into visibility and marketing.
But with the adoption, it also meant we need to up the professionalism a step further, and start unlocking value additions to the developers. This brings us to the problems we’re solving, these come with large benefits to users of the API. Before jumping into the details for a solution, we would like to first expand on the problem itself.
Problem 1 - Common data transformation tasks performed on query results
Blockchains usually have complex data standardization requirements that often need transformation after being extracted. This often ends up in developers building on Cardano to perform data transformation tasks post data consumption. While large-scale projects usually would opt to use one of the awesome libraries from the community - some prefer to be able to keep their code concise and free from dependencies. When they’re already consuming data from API (locally or remotely), it makes more sense to be able to have the freedom to provide, adopt and add value additions to the data that may not be available from the chain or from Cardano components - rather than repeating the task on every consumer downstream.
Problem 2 - End-user accessibility of data returned from Cardano blockchain:
While consuming data from a REST API is quite straightforward for developers, it may not be as accessible to an average end-user. At the time of writing, there is no significantly “easy” way for the average end-user (non-technical) to query this raw data from blockchain provided via APIs. Leveraging existing explorers usually gives them a good UI, but it is often formatted to provide information in a specific template. If users need to deviate from the template, they are reliant on querying API.
Problem 3 - Improvements/continuity for ongoing work from Koios Core members
Maintaining a decentralized query layer has its challenges like baseline knowledge for those who run the instances, setting alert/monitoring to react to outages on their own instance, catering for new requirements (e.g. changing all endpoints to accept bulk inputs instead of having developers send multiple queries), and going through updates across breaking changes while retaining decentralized framework in a minimal way that’s least disruptive.
To be able to solve the above-mentioned problems, we have come up with our solutions as below:
Solution 1 - Utilities Microservice:
To address Problem #1, we came up with a proposition to create a custom “utilities microservice” component that could be run on nodes within Koios clusters. This microservice will be primarily responsible for providing features/transformations that cannot be leveraged natively from chains which will be served from Koios nodes’ cluster independently. Some examples of these utilities would be verification of hashes, encoding/decoding between different formats, fetching ADA price from exchanges or slightly lower level common tasks like extracting staking keys and addresses, serving data from analytics proposals, etc.
These utilities will be integrated into Koios nodes as additional endpoints. This will be created in a way that we would be able to plug more endpoints into this microservice depending on requests/demands from the community.
Solution 2 - Koios Query bots:
To address Problem #2, we’d be creating a query bot that integrates with each and every query that’s available on <https://api.koios.rest>. But instead of having a single bot that runs on some server, we’d be providing the source code for the bot in addition to having an instance running. While doing so, we also aim to either contribute to existing libraries or if needed create one from scratch which leverages the complete functionality of API.
The goal of this query bot will be to allow average end-users to interact with data queried via Koios using simple commands. Telegram is one of the widely used instant messaging services and hosts many Cardano communication groups and channels, and already has a wide array of bot functionalities. Bot integrations could range from simple individual user data queries (e.g. information about a given account) to automated procedures in groups (part of this integration will also extend to Cardano’s Charlie bot that helps with moderation activities across the groups). This initial example would be open for integration into other platforms in future (eg: Discord).
Solution 3 - Commitment to work for Koios gRest updates
For Problem #3, it is more of a bucket of an array of jobs that we’d be doing for incoming months, including maintaining updates to queries (for instance every new version of protocol/db-sync brings changes that will have to be updated in a timely manner and to keep updated versions ready for the fork). In addition, there are always learnings from users about changes to requests made against the API interface - that often need triage and implementation.
Additionally, Koios nodes undergo health checks to prevent incorrect/stale data from being returned to consumers. But for an instance provider, there isn't an alert/notification mechanism on this node itself, and one is expected to work via monitoring instances run by the core team OR use something like Grafana/Prometheus on their end. This could be an overkill for some (other than analytics) when all they really need is details from health checks already being done. Part of the work here would also be creating an alert/notification bot that would be sending push alerts to users in case of trouble - from their own nodes.
Lastly, at times - there are cases when we might need to up the scaling of infrastructure from the core team itself (predominantly during upgrades, but also to increase geographical distribution or add Koios CDNs without participating in existing centralized institutions like Cloudflare). There is already some work being done to extend a self-served (using instance providers) CDN availability across two continents where more than 70% of data is requested from. Such scaling would often require an investment.
[IMPACT] Please describe how your proposed solution will address the Challenge that you have submitted it in.
Tools that improve developer productivity are extremely important for the ecosystem. By implementing a microservices branch, we aim to start reducing the need for consumers of Koios API to manage common post-processing tasks. When applied across the ecosystem, this results in a massive productivity boost and time saved overall. Moreover, the additional microservices will aim to extend the query results to fetch data for off-chain/third-party sources.
The query bot we plan to create is primarily aimed at end-users but has major benefits for developers as well. It provides a ready-to-use bot library that developers can get creative with for their use cases. Examples would range from boosting the functionalities of their communication channels to building standalone apps leveraging bot/Koios integrations.
A very important feature of Koios (gRest) is the ability to run a private cluster of nodes aimed at heavy consumers (large app teams). The popular wallet Etrnl.io is one example of that kind of usage. In order to ensure the benefits of Koios for builders are stable at both public and private levels, we want to improve the built-in monitoring systems that come with Koios. This is where the work for the alerting bot and improvements to the existing node health checks comes in.
[IMPACT] What are the main risks that could prevent you from delivering the project successfully and please explain how you will mitigate each risk?
We cannot foresee any significant risks concerning the development work required. The development and integration options for all of the components have been thoroughly considered and are completely feasible. Having said that, some unforeseeable circumstances might arise that would limit the time available for some team members. However, since we have multiple members with shared skillsets this would still be easily compensated.
[FEASIBILITY] Please provide a detailed plan, including timeline and key milestones for delivering your proposal.
SUCCESS TIMELINE
12 months:
- Keep up with updates from Cardano blockchain to gRest API
- Tend to new requirements from the community for gRest API
- Work on extending CDN to countries where instance providers have higher participation
- Support/maintenance/community engagement for the next year
3 months:
- Development for the template of utilities microservice
- Start integration of features into utilities microservices
- Prepare query bot library
- Extend library into Telegram bot and integrate with Charlie
The Koios team is also submitting another proposal for incentivising the Koios query layer and building our first Koios DApp - <https://cardano.ideascale.com/c/idea/419448>. We analysed our team’s time commitments and skills for each of the projects, and are confident that we have the ability to adhere to both timelines simultaneously.
[FEASIBILITY] Please provide a detailed budget breakdown.
BUDGET
The budget for this proposal is separated into three categories:
- Utilities Microservice - 15,000 USD
- Development/Testing of the microservice template - 10,000 USD
- Integration/Support - 5,000 USD
- Query bots - 10,000 USD
- Development/testing of bot library - 5,000 USD
- Maintenance/Integration of each endpoint - 2,000 USD
- Support - 3,000 USD
- Koios gRest updates - 35,000 USD
- Development/Testing work across bucket - 20,000 USD
- Support/Integration - 5,000 USD
- Infrastructure Scaling - 10,000 USD
[FEASIBILITY] Please provide details of the people who will work on the project.
The group behind Guild-operators (<https://github.com/cardano-community/guild-operators/graphs/contributors>) is a collection of long-term Cardano enthusiasts consisting of multiple OGs/current/ex-ambassadors that have continually been making immense contributions to the ecosystem without any additional marketing or financial cost requests. We are creators of some of the many popular tools and resources in Cardano - including Koios, CNTools, gLiveView, GuildNet (separate network similar to Testnet, but with small 1 hour epochs - useful for testing) and topologyUpdater along with comprehensive Cardano documentation (<https://cardano-community.github.io/guild-operators/>, <https://cardano-community.github.io/support-faq>).
Through its contributions, we have developed a deep understanding of the Cardano blockchain and its components. Apart from our work on Cardano, Guild members have extensive experience in the technology field, with multiple (8+) years of experience across system design, DevOps, system administration, networking, and software development.
[FEASIBILITY] If you are funded, will you return to Catalyst in a later round for further funding? Please explain why / why not.
We define the set of features in the proposal and do not expect to return to Catalyst for those (given the nature of what we're building, if there is a value addition that the community likes and would like to propose/build further, there might be future consideration). Besides, if our second proposal (incentives and lightweight Koios nodes) is approved - we expect Koios to become self-sufficient for such extensions.
[AUDITABILITY] Please describe what you will measure to track your project's progress, and how will you measure these?
Contributions, activity and continued development will be tracked through the public GitHub repositories - <https://github.com/cardano-community/guild-operators> and <https://github.com/cardano-community/koios-artifacts>.
We will keep track of our deliverables, which are as follows:
- Utilities microservice. Develop a new set of Koios endpoints with the features listed above and integrate them into the existing Koios network.
- Telegram light query bot. Development of a comprehensive light-query bot for Telegram designed for average end-users.
- Telegram alert bot. Development of a push notification bot for Telegram designed for network monitoring.
- Infrastructure. Keeping the core servers infrastructure in line with enough grunt to run by itself (to handle both heavy and light modes), when the load isn't being distributed (especially during upgrade cycles).
- Maintenance/Updates to gRest API. Continuous maintenance and improvements to Koios infrastructure and query layers.
We will also keep track of encouraging community engagement through open meetings. As the Cardano community represents the users of Koios, we want to hear and cater to its needs. Hence, 60-minute community meetings are held on a bi-weekly basis and are open to anyone interested in the project or this proposal specifically. These meetings provide a space for the core development team to report on current progress and issues and open the floor to the community to give feedback, make requests, and get involved in discussions about Koios. For more info on this, please visit <https://api.koios.rest/#overview–faq>.
[AUDITABILITY] What does success for this project look like?
First and foremost, the success of this project means that all the deliverables listed above have been completed. We expect the new additions to Koios to be readily used by many developers, both those who already leverage Koios and ones that are yet to discover it. The microservices branch will exist and will publicly serve over 1000 requests daily. The Koios query bot will exist and be leveraged in Cardano Telegram groups with over 1000 members.
[AUDITABILITY] Please provide information on whether this proposal is a continuation of a previously funded project in Catalyst or an entirely new one.
Yes, this proposal is a continuation of the initial Koios proposal which was successfully completed along with receiving great community feedback and contributions, already serving ~3.21 billion query requests in the past month, and is core to many community projects that value decentralization and avoid dependencies on centralised entities.