Trusted Pods grant application

[bojidar-bg]

Project Abstract

Trusted Pods is a decentralized compute marketplace where developers can run container pods securely and confidentially through small and medium cloud providers. It is tailored towards providing a one-stop solution for finding infrastructure to run pods in a cost efficient, serverless manner—though, of course, developing it all the way to production is outside the scope of this grant application.

Within the Polkadot ecosystem, Trusted Pods would be an application/dApp which makes use of ink! smart contracts and IPFS to allow for application publishers to deploy pods to compute service providers and pay for the compute services used. What makes Trusted Pods "trusted" is the use of Trusted Execution Environment (TEE) enclaves, a feature of modern CPUs that enables for the execution of software in a way that does not allow for tampering by the host operating system or administrator.

Grant level

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $30,000, 3 approvals
• Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

• The application template has been copied and aptly renamed (trusted-pods.md).
• I have read the application guidelines.
• Payment details have been provided (bank details via email or BTC, Ethereum (USDC/DAI) or Polkadot/Kusama (USDT) address in the application).
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
• I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[bojidar-bg]

I have read and hereby sign the Contributor License Agreement.

[semuelle]

Thank you for the application, @bojidar-bg.

Could you elaborate what you mean by this project being "Apocryph-inspired"? As you state yourself, my main concern right now would be getting a critical mass of users so that the platform is worthwhile for both sides. Many of the projects involving TEEs which we have supported are struggling with this. Also, could you expand on the libp2p protocols? The interfaces for AWS EC2 are already rather complex, doing something similar with an undefined number of providers (each with their own policies and payment details) in a decentralised manner seems... ambitious.
Lastly, you say that Trusted Pods would be "layered on top [...] either as a parachain, or as part of one or more parachains." Other than the two smart contracts you listed in M4, there are no other (Polkadot) node dependencies, correct? And if you mean the contracts to be deployed on multiple parachains, what would that look like to the user and provider? Sounds like the contracts would have to be "in sync".

[bojidar-bg]

All good questions, @semuelle! I will get around and discuss them with the team, but some preliminary answers going off what I know:

The project is "Apocryph-inspired" in the sense that we took inspiration for the idea from Apocryph, a past project we worked on but never quite managed to push to production—basically, said project needed to have a runtime component executing all the user-submitted code, and Trusted Pods is, in a way, an evolution of the idea for that runtime. (:

For libp2p protocols, I'm currently envisioning something relatively simple, such as.. the provider listing their libp2p address in the registry, and then the publisher opening a connection with that address, forwarding the relevant part of the pod configuration which pertains to resource usage, then the provider ok-ing that, and finally the publisher giving the provider the address of the payment contract and the encrypted pod manifest. It is likely it would become a bit more complicated than just this as we explore the caveats of what's needed, but I do think that would be the general gist of it.

The interfaces for EC2 are not the model we were intent on following for pod manifests (hoping I got that part of your question correctly). Instead, we were thinking of the Kuberenetes PodSpec (with inspirations from the RunPod recipe configuration or I guess the Docker Compose service configuration), so, something more akin to setting up a container image, ports/connections required, and resource quotas, and not akin to setting up a virtual machine.

You are correct about there being nothing blockchain node -dependent apart from those two smart contracts. I would imagine running them on a single parachain, but since the two smartcontracts are both self-contained and do not communicate with each other, I wouldn't see a problem if, one day, we have a registry contract on one parachain and a payment contract on another. However, for the purposes of the grant as currently described, both contracts would likely be on a single parachain, possibly even just on a testnet parachain to get things off the ground. (:

I will have to discuss the critical mass issue further with the team. Feel free to elaborate on what you generally want to see in regards to solving that issue—I'm guessing just mentioning that the issue exists is not cutting it, 😂, but then the FAQ says that marketing and outreach are generally not the kind of activities you fund, so... I'm guessing some kind of more developed plan for reaching critical mass, or...?

[branimirangelov]

Spot on question @semuelle! We have been analyzing the TEE adoption aspect quite a lot and we believe that the landscape is now much more favorable for such projects. Today, there is a lot more technology marketing done by the big cloud providers on the confidential computing frontier, which has increased the overall awareness in the tech world. Also we have done a couple of product / architecture decisions to place Trusted Pods in the right position in the ecosystem to accelerate adoption. Our thinking, so far can be summarized in the following way:

1. TEE-adoption by providers (in Trusted Pods terminology).
   1.1. TEE is becoming the norm in the current (latest) generation of server CPUs, so effectively evey new server hardware equipment is TEE capable.
   1.2. Major public cloud providers (both virtualized and bare metal) offer TEE-based resources and this could be also attactive option for Trusted Pod providers, because we don't require any overhead hardware (mining) - essentially, every provider node in Trusted Pods is billable. In addition, as Trusted Pods is offering platform level service (serverless) that by standard has a higher price tag compared to infrastructure level service, so from economy perspective we make the hardware rental option viable for the providers.
   1.3. Mature and widely popular and adopted TEE-based open source projects like Confidential Containers and Constellation do the "heavylifting" of TEE-based implementations and also streamline the operations activities for running TEE-based technology stacks.

2. Crypto payments for pod execution by the publishers (in Trusted Pods terminology).
   2.1. In our partner's community we have received at first hand inquiries from publishers for crypto payments. This better aligns with the web3 nature of publishers' business, minimizing the overhead of currency exchange.
   2.2. We have seen movements in the public cloud arena about this, specifically Google + Coinbase partnership for offering crypto payments for cloud services, which is a more objective evidence for the increased demand of such offerings.

[semuelle]

Thank you for the follow-ups, @bojidar-bg and @branimirangelov, and sorry for the late reply. We have a bit of a backlog currently.

The interfaces for EC2 are not the model we were intent on following for pod manifests

Sorry, I could have worded that better. I just wanted to point out that using EC2 is already quite complex, and you are adding at least one layer of complexity.

Feel free to elaborate on what you generally want to see in regards to solving that issue

This is something that would fit well into the Future Plans section of the application. Having said that, I don't have a solution to it. I know that this is a lot to ask for an early-stage project, but often teams are too tech-minded to see past the next milestone and it helps to see that teams are aware of what pitfalls might await them and that they might have a plan of attack.

1. TEE is becoming the norm in the current (latest) generation of server CPUs, so effectively evey new server hardware equipment is TEE capable.

This is the first time I am hearing this. Do you have a source for that?

In any case, I am going to mark your application as ready for review so that other committee members can already chime in.

[branimirangelov]

Thank you @semuelle, we appreciate your time and we see the long backlog indeed, which is great for you!

When addressing the initial topic concerning EC2, it's important to clarify that our intention is not to enhance the EC2 layer, but rather to operate at an entirely different service level. EC2 operates within the Infrastructure as a Service (IaaS) paradigm, whereas our model is firmly rooted in the Platform as a Service (PaaS) layer. To draw a comparison to EC2, I would liken our model more to services like AWS Lambda [1] or Heroku [2] (which, internally, leverages AWS EC2). In contrast to EC2 [3], our approach is notably simpler, as supported by references to the AWS cost calculator.

Another crucial distinction in our proposed approach is our current lack of plans to implement load balancing or high availability across different providers, at least not as part of the grant. While this approach is undeniably easier to implement, the primary driver behind this direction is our specific target audience: small-to-medium hosting providers who rely on hyperconverged infrastructure or lease infrastructure from large, hyperscale providers. Initially, we are not optimizing for home users. Instead, our goal is to empower hosting providers with a level of flexibility that enables them to optimize their infrastructure costs (similar to managing EC2 costs) while delivering the essential platform services to developers. This level of flexibility empowers providers to innovate at the infrastructure layer, facilitating the discovery of more efficient ways to deliver their services.

In the discussion on the Trusted Execution Environment (TEE), it's crucial to reiterate that our intended provider audience does not encompass home users. When examining the TEE landscape, our focus centers on server-grade CPUs. While numerous significant challenges have arisen in the context of home and mobile users, TEE adoption takes a distinctly different and consistent trajectory in the realm of server-grade CPUs. Specifically, we are referring to processors such as Intel's Xeon within Intel's product portfolio.

Our information is primarily derived from our own analysis of product specifications from the most prominent server CPU vendors in the hyperconverged infrastructure market. It's important to note that major cloud players rely heavily on hyperscale infrastructure, which adds complexity to the market landscape. Nevertheless, Intel and AMD continue to hold significant market shares there as well.

For both Intel [4] and AMD [5], their latest generations of server-grade CPUs (Xeon and EPYC) come equipped with built-in TEE capabilities. Hyperconverged infrastructure equipment vendors, such as Dell, HP, Supermicro, and other OEMs, primarily source their CPUs from Intel and AMD. To illustrate, examining the SGX support in Xeon [6] reveals that all variants offer SGX support, albeit with varying Enclave Page Cache (EPC) sizes.

The increased adoption of TEE in server CPUs is primarily driven by the stringent security requirements set forth by national and industry-based security standards. Additionally, there is a growing preference in the enterprise world for confidential and sovereign cloud architectures. This confluence of factors has propelled the integration of TEE features into server-grade CPUs, ensuring heightened security controls and compliance with evolving security standards.

References:
[1] AWS Cost Calulator for Lambda: https://calculator.aws/#/addService/Lambda
[2] Heroku Cost Calculator: https://www.heroku.com/pricing
[3] AWS Cost Calculator for EC2: https://calculator.aws/#/addService/ec2-enhancement
[4] Xeon 4th gen product brief: https://www.intel.com/content/www/us/en/products/docs/processors/xeon-accelerated/4th-gen-xeon-scalable-processors-product-brief.html
[5] EPYC 4th gen architecture brief: https://www.amd.com/en/campaigns/epyc-9004-architecture
[6] SGX supported CPUs list: https://www.intel.com/content/www/us/en/architecture-and-technology/software-guard-extensions-processors.html

[Noc2]

Thanks for the application. I have a few comments/questions:

• Only your last milestone currently seems to be related to our ecosystem (ink! integration). Is this correct?
• Are you aware of https://acurast.com/? or Integritee? It might make sense to leverage their tech stack or even work together with them.
• Could you add the programming language to the specification of each deliverable?

[bojidar-bg]

@Noc2 Thanks for taking the time to give our application a look (:

1. Yes. Trusted Pods, as currently designed, interacts with the Substrate ecosystem only through the smart contracts that are introduced in milestone 3.
2. I hadn't seen Acurast before, but from what I can read on their website, there are two main differences between it and our project: it currently uses mobile phone CPUs as secure enclaves, while we use server-grade CPUs for the same, and it is mostly tailored towards running tasks, while we want to make something tailored for deploying long-running services.
   For Integretee, we were in fact considering using their TEE-enabled smart contracts to manage the key exchange between the provider and publisher (step 4 of the diagram), but ultimately opted for a direct libp2p connection to cut down the scope. But overall, the focus of Integretee seems to be a bit different - from what I can see, they are aiming to provide a way to remotely attest app-specific worker nodes, while we are aiming to provide a way for hiring app-neutral worker nodes. That being said, there could be a way to leverage Integretee for remote attestation, so we will keep it in mind.
3. Good call, I will have to fill that in. Until then, assume components are written in Go where possible (due to the abundance of web-native API bindings), and Rust otherwise.

[keeganquigley]

Thanks for the application @bojidar-bg it seems like a really cool idea, but I agree with my colleagues that milestone 1 isn't very specific the substrate ecosystem, and afaik we usually typically don't pay for the spec parts. Personally, I might be more apt to approve a reduced level 2 PoC of the registry and payment contracts, but I understand that you might not be able to fund the other parts on your own.

[bojidar-bg]

Hey @keeganquigley! That's a good point. Personally, I think that projects like Trusted Pods that provide infrastructure for running developer-submitted code can't quite succeed without sufficient documentation of the functions and formats exposed... but I can understand that paying for such specification out of the grant is not quite aligned with the goals of the fund.

That being said, we are going to be approaching investors as well and are committed enough to the project to self-fund parts of it, so we won't necessarily opposed to trim the scope of the grant somewhat. So, I think we could cut out the "boilerplate" first milestone (and later specification deliveries) and deliver just the TEE-related and smart-contract-related functionality as part of the grant—which is, practically speaking, the main aim of the Trusted Pods, to have a blockchain-backed marketplace where one can rent out TEE-secured compute resources. How does that sound?
(I'll edit the grant description once/if you confirm that's the roughly the way you want to go with it (: )

[keeganquigley]

Thanks for the update @bojidar-bg sure, I can't speak for others of course, but that sounds good to me. I'd be willing to support a reduced scope of just the TEE-related and smart-contract-related functionalities.

[bojidar-bg]

Alright, @keeganquigley! I have updated the application to remove the first milestone and reordered the other two. Hopefully it is more in line with what Web3 might be willing to support now (:

[keeganquigley]

Thanks for the changes @bojidar-bg this looks interesting, and also based on your previous work, I'm happy to go ahead with it.

[semuelle]

Hi @bojidar-bg, thanks again for the application. After discussion, it has become clear that the application will not find the necessary support in the committee. The main point of concern was the long-term viability of the project. I wish I had better news, but I want to point out that we are very grateful for the effort you put into the application and the friendly and thorough responses in our discussion. I hope that you will find a way to continue with the project. In any case, best of luck and feel free to apply again in the future.

[bojidar-bg]

Welp... 😅
Thanks for taking the time to look into the project! It has, again, been a pleasure to work with you, and we are thankful for all the the feedback you've given! 😊
At any rate, we shall be on our way looking around for other sources of funding and we'll keep W3F in mind for any future blockchain-related projects that might be a better fit (: