diff --git a/README.md b/README.md index 2a2c01986..a4b99a857 100644 --- a/README.md +++ b/README.md @@ -1,347 +1,126 @@ -![runwasi logo light mode](./art/logo/runwasi_icon1.svg#gh-light-mode-only) -![runwasi logo dark mode](./art/logo/runwasi_icon3.svg#gh-dark-mode-only) +# Runwasi with WasmEdge Advancements -## runwasi +> ### Announcement +> This repository is not the official version of [Runwasi](https://github.com/containerd/runwasi). Its purpose is solely to showcase the functionalities of Runwasi with [WasmEdge](https://github.com/WasmEdge/WasmEdge) through demonstrations and provide users with quick access to installation package releases. If you are new to Runwasi, I would recommend reading the official [README](https://github.com/containerd/runwasi/blob/main/README.md) first. The following operations and explanations assume that you have a certain level of familiarity to proceed. -> Warning: Alpha quality software, do not use in production. +## LLAMA2 Example -This is a project to facilitate running wasm workloads managed by containerd either directly (ie. through ctr) or as directed by Kubelet via the CRI plugin. -It is intended to be a (rust) library that you can take and integrate with your wasm host. -Included in the repository is a PoC for running a plain wasi host (ie. no extra host functions except to support wasi system calls). +[![asciicast](https://asciinema.org/a/617758.svg)](https://asciinema.org/a/617758) -### Community +### Download model -- If you haven't joined the CNCF slack yet, you can do so [here](https://slack.cncf.io/). -- Come join us on our [slack channel #runwasi](https://cloud-native.slack.com/archives/C04LTPB6Z0V) -on the CNCF slack. -- Public Community Call on Tuesdays at 9:00 AM PT: [Zoom](https://zoom.us/my/containerd?pwd=bENmREpnSGRNRXdBZWV5UG8wbU1oUT09), [Meeting Notes](https://docs.google.com/document/d/1aOJ-O7fgMyRowHD0kOoA2Z_4d19NyAvvdqOkZO3Su_M/edit?usp=sharing) - -### Usage - -runwasi is intended to be consumed as a library to be linked to from your own wasm host implementation. - -There are two modes of operation supported: - -1. "Normal" mode where there is 1 shim process per container or k8s pod. -2. "Shared" mode where there is a single manager service running all shims in process. - -In either case you need to implement a trait to teach runwasi how to use your wasm host. - -There are two ways to do this: -* implementing the `sandbox::Instance` trait -* or implementing the `container::Engine` trait - -The most flexible but complex is the `sandbox::Instance` trait: - -```rust -pub trait Instance { - /// The WASI engine type - type Engine: Send + Sync + Clone; - - /// Create a new instance - fn new(id: String, cfg: Option<&InstanceConfig>) -> Self; - /// Start the instance - /// The returned value should be a unique ID (such as a PID) for the instance. - /// Nothing internally should be using this ID, but it is returned to containerd where a user may want to use it. - fn start(&self) -> Result; - /// Send a signal to the instance - fn kill(&self, signal: u32) -> Result<(), Error>; - /// Delete any reference to the instance - /// This is called after the instance has exited. - fn delete(&self) -> Result<(), Error>; - /// Wait for the instance to exit - /// The waiter is used to send the exit code and time back to the caller - /// Ideally this would just be a blocking call with a normal result, however - /// because of how this is called from a thread it causes issues with lifetimes of the trait implementer. - fn wait(&self, waiter: &Wait) -> Result<(), Error>; -} -``` - -The `container::Engine` trait provides a simplified API: - -```rust -pub trait Engine: Clone + Send + Sync + 'static { - /// The name to use for this engine - fn name() -> &'static str; - /// Run a WebAssembly container - fn run_wasi(&self, ctx: &impl RuntimeContext, stdio: Stdio) -> Result; - /// Check that the runtime can run the container. - /// This checks runs after the container creation and before the container starts. - /// By it checks that the wasi_entrypoint is either: - /// * a file with the `wasm` filetype header - /// * a parsable `wat` file. - fn can_handle(&self, ctx: &impl RuntimeContext) -> Result<()> { /* default implementation*/ } -} +```bash +curl -LO https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q5_K_M.gguf ``` -After implementing `container::Engine` you can use `container::Instance`, which implements the `sandbox::Instance` trait. +### Install wasmedge shim with ggml plugin -To use your implementation in "normal" mode, you'll need to create a binary which has a main that looks something like this: - -```rust -use containerd_shim as shim; -use containerd_shim_wasm::sandbox::{ShimCli, Instance} - -struct MyInstance { - // ... -} - -impl Instance for MyInstance { - // ... -} - -fn main() { - shim::run::>("io.containerd.myshim.v1", opts); -} +```bash +sudo ctr content fetch ghcr.io/second-state/runwasi-wasmedge-plugin:allinone.wasi_nn-ggml-preview +sudo ctr install ghcr.io/second-state/runwasi-wasmedge-plugin:allinone.wasi_nn-ggml-preview -l -r ``` -or when using the `container::Engine` trait, like this: - -```rust -use containerd_shim as shim; -use containerd_shim_wasm::{sandbox::ShimCli, container::{Instance, Engine}} +### Login github with your PAT first -struct MyEngine { - // ... -} - -impl Engine for MyEngine { - // ... -} - -fn main() { - shim::run::>>("io.containerd.myshim.v1", opts); -} +```bash +docker login ghcr.io -u ``` -Note you can implement your own ShimCli if you like and customize your wasm engine and other things. -I encourage you to checkout how that is implemented. - -The shim binary just needs to be installed into `$PATH` (as seen by the containerd process) with a binary name like `containerd-shim-myshim-v1`. - -For the shared mode: - -```rust -use containerd_shim_wasm::sandbox::{Local, ManagerService, Instance}; -use containerd_shim_wasm::services::sandbox_ttrpc::{create_manager, Manager}; -use std::sync::Arc; -use ttrpc::{self, Server}; -/// ... +### Run llama2 with docker + wasm -struct MyInstance { - /// ... -} - -impl Instance for MyInstance { - // ... -} - -fn main() { - let s: ManagerService> = - ManagerService::new(Engine::new(Config::new().interruptable(true)).unwrap()); - let s = Arc::new(Box::new(s) as Box); - let service = create_manager(s); - - let mut server = Server::new() - .bind("unix:///run/io.containerd.myshim.v1/manager.sock") - .unwrap() - .register_service(service); - - server.start().unwrap(); - let (_tx, rx) = std::sync::mpsc::channel::<()>(); - rx.recv().unwrap(); -} +```bash +docker run --rm --runtime=io.containerd.wasmedge.v1 --platform wasi/wasm \ + -v /opt/containerd/lib:/opt/containerd/lib \ + -v $PWD:/resource \ + --env WASMEDGE_PLUGIN_PATH=/opt/containerd/lib \ + --env WASMEDGE_WASINN_PRELOAD=default:GGML:CPU:/resource/llama-2-7b.Q5_K_M.gguf \ + --env LLAMA_N_CTX=4096 --env LLAMA_N_PREDICT=128 \ + ghcr.io/second-state/runwasi-demo:llama-simple \ + default 'Robert Oppenheimer most important achievement is ' ``` -This will be the host daemon that you startup and manage on your own. -You can use the provided `containerd-shim-myshim-v1` binary as the shim to specify in containerd. - -Shared mode requires precise control over real threads and as such should not be used with an async runtime. - -### Examples - -#### Components - -- **containerd-shim-[ wasmedge | wasmtime | wasmer ]-v1** - -This is a containerd shim which runs wasm workloads in [WasmEdge](https://github.com/WasmEdge/WasmEdge) or [Wasmtime](https://github.com/bytecodealliance/wasmtime) or [Wasmer](https://github.com/wasmerio/wasmer). -You can use it with containerd's `ctr` by specifying `--runtime=io.containerd.[ wasmedge | wasmtime | wasmer ].v1` when creating the container. -And make sure the shim binary must be in $PATH (that is the $PATH that containerd sees). Usually you just run `make install` after `make build`. -> build shim with wasmedge we need install library first - -This shim runs one per pod. - -- **containerd-shim-[ wasmedge | wasmtime | wasmer ]d-v1** +### Output -A cli used to connect containerd to the `containerd-[ wasmedge | wasmtime | wasmer ]d` sandbox daemon. -When containerd requests for a container to be created, it fires up this shim binary which will connect to the `containerd-[ wasmedge | wasmtime | wasmer ]d` service running on the host. -The service will return a path to a unix socket which this shim binary will write back to containerd which containerd will use to connect to for shim requests. -This binary does not serve requests, it is only responsible for sending requests to the `contianerd-[ wasmedge | wasmtime | wasmer ]d` daemon to create or destroy sandboxes. - -- **containerd-[ wasmedge | wasmtime | wasmer ]d** - -This is a sandbox manager that enables running 1 wasm host for the entire node instead of one per pod (or container). -When a container is created, a request is sent to this service to create a sandbox. -The "sandbox" is a containerd task service that runs in a new thread on its own unix socket, which we return back to containerd to connect to. - -The Wasmedge / Wasmtime / Wasmer engine is shared between all sandboxes in the service. - -To use this shim, specify `io.containerd.[ wasmedge | wasmtime | wasmer ]d.v1` as the runtime to use. -You will need to make sure the `containerd-[ wasmedge | wasmtime | wasmer ]d` daemon has already been started. - -### Contributing - -#### Building - -1. Install dependencies - -If on ubuntu/debian you can use the following script. Refer to [youki's](https://github.com/containers/youki#dependencies) documentation for other systems. - -``` -./scripts/setup-linux.sh +```bash +Robert Oppenheimer most important achievement is +1945, when he was the director of the Manhattan Project, which was the development of the atomic bomb. surely, this is the most important achievement of his life. +Robert Oppenheimer was born in New York City on April 22, 1904. He was the son of Julius Oppenheimer, a wealthy textile merchant, and Ella Friedman Oppenheimer. He was the youngest of three children. His father was Jewish and his mother was Protestant. +Oppenheimer was a brilliant student and graduated from the Ethical Culture School in New York City in 1920. He then attended Harvard University, where he studied physics and mathematics. He graduated from Harvard in 1925 with a degree in physics. +After graduating from Harvard, Oppenheimer went to Europe to study physics. He attended the University of Göttingen in Germany and the University of Cambridge in England. He then returned to the United States and began working at the University of California, Berkeley. +In 1932, Oppenheimer was appointed to the faculty of the University of California, Berkeley. He was a professor of physics and astronomy. He was also the director of the Radiation Laboratory at the University of California, Berkeley. +In 1939, Oppenheimer was appointed to the faculty of the Institute for Advanced Study in Princeton, New Jersey. He was a professor of physics and astronomy. He was also the director of the Institute for Advanced Study. +In 1943, Oppenheimer was appointed to the faculty of the University of California, Los Angeles. He was a professor of physics and astronomy. He was also the director of the Institute for Advanced Study. +In 1945, Oppenheimer was appointed to the faculty of the University of California, Berkeley. He was a professor of physics and astronomy. He was also the director of the Institute for Advanced Study. +In 1947, Oppenheimer was appointed to the faculty of the University of California, Los Angeles. He was a professor of physics and astronomy. He was also the director of the Institute for Advanced Study. +In 1952, Oppenheimer was appointed to the faculty of the University of California, Berkeley. He was a professor of ``` -If on Windows use (use [git BASH](https://gitforwindows.org/) terminal which has shell emulator) +## Introduce +The current installation process for the latest release is based on **[containerd managing opt](https://github.com/containerd/containerd/blob/main/docs/managed-opt.md)**. We maintain installable images in several `ghcr.io` repositories. Here, we'll briefly explain how to access them. -``` -./scripts/setup-windows.sh -``` +### [runwasi-wasmedge](https://github.com/second-state/runwasi/pkgs/container/runwasi-wasmedge/versions?filters%5Bversion_type%5D=tagged) -2. Build +Maintain an image with only the shim and essential libraries, currently providing a `latest` tag image for regular updates and a `preview` image for pre-release demo, specifically for showcasing the functionality required by `llama2`. -``` -make build -make check -``` +- latest +- preview -### Running unit tests +### [runwasi-wasmedge-plugin](https://github.com/second-state/runwasi/pkgs/container/runwasi-wasmedge-plugin/versions?filters%5Bversion_type%5D=tagged) -```terminal -make test -``` +This is the new plugin functionality we currently support. The installation process for plugins is similar to the one mentioned earlier (shim only), which will be explained in more detail below. The difference lies in the use of tags to differentiate functionality. In general, the naming convention is as follows: -You should see some output like: -```terminal -running 3 tests -test instance::tests::test_maybe_open_stdio ... ok -test instance::wasitest::test_delete_after_create ... ok -test instance::wasitest::test_wasi ... ok ``` - -### Building the test image - -This builds a [wasm application](crates/wasi-demo-app/) and packages it in an OCI format: - -``` -make test-image +. ``` -### Running integration tests with k3s +The only exception to this is the addition of the `-preview` postfix, which is used specifically for llama2 demos. -``` -make test/k3s-[ wasmedge | wasmer ] -``` +- \ + - lib : Library only, including plugin and its dependencies. + - allinone : In addition to the library, it also includes the Wasmedge Shim binary. +- \ : suppoted plugin + - wasi_nn-pytorch + - wasi_nn-ggml-preview + - wasm_bpf (TBV) + - wasmedge_image (TBV) + - wasmedge_tensorflow (TBV) + - wasmedge_tensorflowlite (TBV) + - wasi_crypto (TBV) + - wasi_nn-tensorflowlite (TBV) + - wasi_nn-openvino (TBV) -### Running integration tests with kind -``` -make test/k8s -``` - -## Demo +> ### Warning +> 1. **TBV** means to be verified. It has been pre-packaged, but complete integration tests have not been added yet, so there may be issues during execution. +> 2. This feature is still in progress, and the naming conventions may change at later. -### Installing the shims for use with Containerd +## Instructions -Make sure you have [installed dependencies](#Building) and install the shims: +- To install the above packages, you can simply execute the following commands. -```terminal -make build -sudo make install +```bash +sudo ctr content fetch : +sudo ctr install : -l -r ``` -Build the test image and load it into contianerd: +- To remove it, you will need to manually delete the files under the specified path (if they exist). +```bash +rm /opt/containerd/lib/* +rm /opt/containerd/bin/* ``` -make test-image -make load -``` - -#### Demo 1 using wasmedge -Run it with `sudo ctr run --rm --runtime=io.containerd.[ wasmedge | wasmtime ].v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest testwasm /wasi-demo-app.wasm echo 'hello'`. You should see some output repeated like: +> ### Warning +> Installation from the `ctr` command will overwrite the default runtime (shim) search path. So, if you want to run the shim build from source, you should first execute the previous command to remove them and then restart `containerd` using the command below. -```terminal -sudo ctr run --rm --runtime=io.containerd.wasmedge.v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest testwasm /wasi-demo-app.wasm echo 'hello' - -hello -exiting +```bash +sudo systemctl daemon-reload && sudo systemctl restart containerd ``` -#### Demo 2 using wasmtime - -Run it with `sudo ctr run --rm --runtime=io.containerd.[ wasmedge | wasmtime ].v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest testwasm`. -You should see some output repeated like: - -```terminal -sudo ctr run --rm --runtime=io.containerd.wasmtime.v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest testwasm +## More Examples -This is a song that never ends. -Yes, it goes on and on my friends. -Some people started singing it not knowing what it was, -So they'll continue singing it forever just because... +We have prepared three daily run demo actions, including [ctr](https://github.com/second-state/runwasi/blob/feature-plugin/.github/workflows/full-testing.yml), [docker](https://github.com/second-state/runwasi/blob/feature-plugin/.github/workflows/docker-demo.yml), and [llama2](https://github.com/second-state/runwasi/blob/feature-plugin/.github/workflows/llama2.yml) (preview). You can check their usage and daily action [status](https://github.com/second-state/runwasi/actions). -This is a song that never ends. -Yes, it goes on and on my friends. -Some people started singing it not knowing what it was, -So they'll continue singing it forever just because... - -(...) -``` +## Prebuilt demo image -#### Demo 3 using wasmer - -Run it with `sudo ctr run --rm --runtime=io.containerd.[ wasmedge | wasmtime | wasmer ].v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest testwasm`. -You should see some output repeated like: - -```terminal -sudo ctr run --rm --runtime=io.containerd.wasmer.v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest testwasm - -This is a song that never ends. -Yes, it goes on and on my friends. -Some people started singing it not knowing what it was, -So they'll continue singing it forever just because... - -This is a song that never ends. -Yes, it goes on and on my friends. -Some people started singing it not knowing what it was, -So they'll continue singing it forever just because... - -(...) -``` - -To kill the process from demo 2, you can run in other session: `sudo ctr task kill -s SIGKILL testwasm`. - -The test binary supports commands for different type of functionality, check [crates/wasi-demo-app/src/main.rs](crates/wasi-demo-app/src/main.rs) to try it out. - -## Demo 3 using OCI Images with custom WASM layers - -The previous demos run with an OCI Container image containing the wasm module in the file system. Another option is to provide a cross-platform OCI Image that that will not have the wasm module or components in the file system of the container that wraps the wasmtime/wasmedge process. This OCI Image with custom WASM layers can be run across any platform and provides for de-duplication in the Containerd content store among other benefits. - -To learn more about this approach checkout the [design document](https://docs.google.com/document/d/11shgC3l6gplBjWF1VJCWvN_9do51otscAm0hBDGSSAc/edit). - -> **Note**: This requires containerd 1.7.7+ and 1.6.25+ (not yet released). If you do not have these patches for both `containerd` and `ctr` you will end up with an error message such as `mismatched image rootfs and manifest layers` at the import and run steps. - -Build and import the OCI image with WASM layers image: - -``` -make test-image/oci -make load/oci -``` - -Run the image with `sudo ctr run --rm --runtime=io.containerd.[ wasmedge | wasmtime | wasmer ].v1 ghcr.io/containerd/runwasi/wasi-demo-oci:latest testwasmoci` - -``` -sudo ctr run --rm --runtime=io.containerd.wasmtime.v1 ghcr.io/containerd/runwasi/wasi-demo-oci:latest testwasmoci wasi-demo-oci.wasm echo 'hello' -hello -exiting -``` +[Here](https://github.com/second-state/runwasi/pkgs/container/runwasi-demo/versions?filters%5Bversion_type%5D=tagged), we have prebuilt demo images available. You can obtain these same images by simply running `make load_demo` and `make load` (official test case) in the root folder. They are generated based on the test cases in this [directory](https://github.com/second-state/runwasi/tree/preview/demo).