Please describe your proposed solution.
In the first proposal of Koios made under round of Fund 6, we were not fully sure about the adoption - especially looking at an ecosystem filled with well-established entities, but we still wanted to showcase and enforce the need of not just using open source, but going beyond and inheriting the idea of decentralization by fully utilizing open source frameworks.
To our pleasant surprise, we received a lot of participation (we have already awesome community members who have independently contributed SDKs, CLIs, and integrations) and appreciation from the development community, in return - this makes us go an extra few miles to ensure we live up to their expectations resulting in thorough coordination and preparation for base query layer (gRest) to be able to serve the absolute maximum of information there is from the chain, resulting in serving billions of requests over past months and a steady increase in adoption curve without putting as much effort into visibility and marketing.
But with the adoption, it also meant we need to up the professionalism a step further, and start unlocking value additions to the developers. This brings us to the problems we’re solving, these come with large benefits to users of the API. Before jumping into the details for a solution, we would like to first expand on the problem itself.
Problem 1 - Common data transformation tasks performed on query results
Blockchains usually have complex data standardization requirements that often need transformation after being extracted. This often ends up in developers building on Cardano to perform data transformation tasks post data consumption. While large-scale projects usually would opt to use one of the awesome libraries from the community - some prefer to be able to keep their code concise and free from dependencies. When they’re already consuming data from API (locally or remotely), it makes more sense to be able to have the freedom to provide, adopt and add value additions to the data that may not be available from the chain or from Cardano components - rather than repeating the task on every consumer downstream.
Problem 2 - End-user accessibility of data returned from Cardano blockchain:
While consuming data from a REST API is quite straightforward for developers, it may not be as accessible to an average end-user. At the time of writing, there is no significantly “easy” way for the average end-user (non-technical) to query this raw data from blockchain provided via APIs. Leveraging existing explorers usually gives them a good UI, but it is often formatted to provide information in a specific template. If users need to deviate from the template, they are reliant on querying API.
Problem 3 - Improvements/continuity for ongoing work from Koios Core members
Maintaining a decentralized query layer has its challenges like baseline knowledge for those who run the instances, setting alert/monitoring to react to outages on their own instance, catering for new requirements (e.g. changing all endpoints to accept bulk inputs instead of having developers send multiple queries), and going through updates across breaking changes while retaining decentralized framework in a minimal way that’s least disruptive.
To be able to solve the above-mentioned problems, we have come up with our solutions as below:
Solution 1 - Utilities Microservice:
To address Problem #1, we came up with a proposition to create a custom “utilities microservice” component that could be run on nodes within Koios clusters. This microservice will be primarily responsible for providing features/transformations that cannot be leveraged natively from chains which will be served from Koios nodes’ cluster independently. Some examples of these utilities would be verification of hashes, encoding/decoding between different formats, fetching ADA price from exchanges or slightly lower level common tasks like extracting staking keys and addresses, serving data from analytics proposals, etc.
These utilities will be integrated into Koios nodes as additional endpoints. This will be created in a way that we would be able to plug more endpoints into this microservice depending on requests/demands from the community.
Solution 2 - Koios Query bots:
To address Problem #2, we’d be creating a query bot that integrates with each and every query that’s available on <https://api.koios.rest>. But instead of having a single bot that runs on some server, we’d be providing the source code for the bot in addition to having an instance running. While doing so, we also aim to either contribute to existing libraries or if needed create one from scratch which leverages the complete functionality of API.
The goal of this query bot will be to allow average end-users to interact with data queried via Koios using simple commands. Telegram is one of the widely used instant messaging services and hosts many Cardano communication groups and channels, and already has a wide array of bot functionalities. Bot integrations could range from simple individual user data queries (e.g. information about a given account) to automated procedures in groups (part of this integration will also extend to Cardano’s Charlie bot that helps with moderation activities across the groups). This initial example would be open for integration into other platforms in future (eg: Discord).
Solution 3 - Commitment to work for Koios gRest updates
For Problem #3, it is more of a bucket of an array of jobs that we’d be doing for incoming months, including maintaining updates to queries (for instance every new version of protocol/db-sync brings changes that will have to be updated in a timely manner and to keep updated versions ready for the fork). In addition, there are always learnings from users about changes to requests made against the API interface - that often need triage and implementation.
Additionally, Koios nodes undergo health checks to prevent incorrect/stale data from being returned to consumers. But for an instance provider, there isn't an alert/notification mechanism on this node itself, and one is expected to work via monitoring instances run by the core team OR use something like Grafana/Prometheus on their end. This could be an overkill for some (other than analytics) when all they really need is details from health checks already being done. Part of the work here would also be creating an alert/notification bot that would be sending push alerts to users in case of trouble - from their own nodes.
Lastly, at times - there are cases when we might need to up the scaling of infrastructure from the core team itself (predominantly during upgrades, but also to increase geographical distribution or add Koios CDNs without participating in existing centralized institutions like Cloudflare). There is already some work being done to extend a self-served (using instance providers) CDN availability across two continents where more than 70% of data is requested from. Such scaling would often require an investment.
Please describe how your proposed solution will address the Challenge that you have submitted it in.
Tools that improve developer productivity are extremely important for the ecosystem. By implementing a microservices branch, we aim to start reducing the need for consumers of Koios API to manage common post-processing tasks. When applied across the ecosystem, this results in a massive productivity boost and time saved overall. Moreover, the additional microservices will aim to extend the query results to fetch data for off-chain/third-party sources.
The query bot we plan to create is primarily aimed at end-users but has major benefits for developers as well. It provides a ready-to-use bot library that developers can get creative with for their use cases. Examples would range from boosting the functionalities of their communication channels to building standalone apps leveraging bot/Koios integrations.
A very important feature of Koios (gRest) is the ability to run a private cluster of nodes aimed at heavy consumers (large app teams). The popular wallet Etrnl.io is one example of that kind of usage. In order to ensure the benefits of Koios for builders are stable at both public and private levels, we want to improve the built-in monitoring systems that come with Koios. This is where the work for the alerting bot and improvements to the existing node health checks comes in.
What are the main risks that could prevent you from delivering the project successfully and please explain how you will mitigate each risk?
We cannot foresee any significant risks concerning the development work required. The development and integration options for all of the components have been thoroughly considered and are completely feasible. Having said that, some unforeseeable circumstances might arise that would limit the time available for some team members. However, since we have multiple members with shared skillsets this would still be easily compensated.