Upgrade the task scheduler to stable Tokio

[drunkirishcoder]

Background

Capsule v0.1.0 uses a customized task scheduler leveraging a preview version of Tokio and futures. We need to upgrade to the stable releases of futures and Tokio now that they are available.

Problem

Tokio lib has gone through a major project restructure. The components we leveraged in building the custom task scheduler are not accessible from outside the Tokio crate anymore. Instead we have to use the default threadpool scheduler if we want to avoid writing our own task scheduling. To move off of the preview releases, the current scheduler has to be reimplemented.

Proposal

The current threading model Capsule uses is a run-to-completion model, where each thread is responsible for reading a batch of packets from the RX queue, feed the batch through the combinator pipeline for processing, send them out through the TX queue, rinse and repeat.

Instead, we can move to a threadpool-based model where a single receiving thread is used for reading batches of packets from the RX queue, and queues up each batch as a task that the threads in a threadpool can then pick up and process.

This will be a major breaking change. And we will need to measure and compare the performance of the two threading models.

[rohitjoshi]

May be switch to smol

[drunkirishcoder]

@rohitjoshi interesting suggestion. I really like the simplicity of smol a lot after briefly looking it over. Going to read through the blog posts to get a better understanding of it. Thank you for bring this up. We will definitely add it to our consideration.

[ratnadeepb]

Have you guys figured out a way you would like to move forward with this? I have spent some time looking at your Runtime code and I was wondering if I might be able to provide some help! (Disclaimer: I am relatively new to Rust and especially async Rust)

[zeeshanlakhani]

@ratnadeepb I'll let @drunkirishcoder say more, but we do already have a new, updated run to completion model that's currently WIP. We've just been a bit busy with other projects, but plan to finish up the work soon. Once it's up for review however, we could def use some review, testing, etc...

[JakkuSakura]

I have implemented a runtime agnostic version of capsule, leveraging glommio and tokio 1.0 to better fit my requirements. It is ugly but basically worked. Hope capsule 0.2.0 give the proper api to replace the runtime more easily