add a doc for prefetchfiles-nri-plugin

docs/prefetchfiles-nri-plugin.md

@@ -0,0 +1,156 @@
+# User self-defined nydus image files prefetch
+To improve the flexibility of nydus image files prefetch, for the k8s scenario, we can specify a prefetch files list when create a nydus daemon. The prefetch files list is user self-defined. Nydus-snapshotter has implemented a containerd NRI plugin to transmit the path of prefetch files list to nydus-snapshotter. The prefetch plugin requires NRI 2.0, which is available in containerd (>=v1.7.0). The prefetch plugin subscribes pod creation event, fetches the path of prefetch files list and forwards it to nydus-snapshotter. Then when `nydusd` starts, it will pull the files defined in the prefetch files list through lazy loading. This allows the pull of the prefetch files to be done during container creation rather than image convert, improving the flexibility of file prefetching.
+
+## Requirements
+
+- [NRI 2.0](https://github.com/containerd/nri): Which has been integrated into containerd since [v1.7.0](https://github.com/containerd/containerd/tree/v1.7.0-beta.1).
+
+## Workflow
+
+1. Add information such as image reference and prefetch list path to annotations in pod configuration file.
+2. Run the prefetch plugin to monitoring RunPodSandbox events.
+3. The prefetch plugin fetches image reference and prefetch files path and forwards them to nydus-snapshotter.
+4. Nydus-snapshotter specifies the prefetch list when starting nydus daemon.
+5. Nydusd completes the mounting of the nydus image.
+<!-- 4. Run the prefetch plugin to fetch those information and forward them to nydus-snapshotter when the RunPodSandbox event occurs.  -->
+
+
+
+
+
+## Modify configuration file 
+
+Modify containerd's toml configuration file to enable NRI.
+```console
+sudo tee -a /etc/containerd/config.toml <<- EOF
+[plugins."io.containerd.nri.v1.nri"]
+    config_file = "/etc/nri/nri.conf"
+    disable = false
+    plugin_path = "/opt/nri/plugins"
+    socket_path = "/var/run/nri.sock"
+EOF
+```
+Containerd will load all NRI plugins in the `plugin_path` directory on startup. If you want to start an NRI plugin manually, please add the following configuration to allow other NRI plugins to connect via `socket_path`.
+
+```console
+sudo tee /etc/nri/nri.conf <<- EOF
+disableConnections: false
+EOF
+```
+
+Restart the containerd service.
+
+```console
+sudo systemctl restart containerd
+```
+Add a prefetch configuration file to specify the correct socket address. Note that this configuration file is designed for starting the NRI plugin in `pre-connection` mode. In this mode, the NRI plugin and containerd will start simultaneously.
+```console
+sudo tee /etc/nydus/prefetchConfig.toml <<- EOF
+[file_prefetch]
+socket_address = "/run/containerd-nydus/system.sock"
+EOF
+```
+Also, when manually starting the prefetch NRI plugin, the socket address can also be modified through the command line parameter `socket-addr`. Note that the priority of configuration file parameter passing is higher than that of command line parameter passing. 
+
+
+
+After start the prefetch plugin, it will monitor pod creation events. Note that NRI plugin can only be called from containerd/CRI. So creation a pod using crictl as below.
+```console
+sudo tee pod.yaml <<- EOF
+kind: pod
+metadata:
+  name: wordpress-sandbox
+  namespace: default
+  attempt: 1
+  uid: hdishd83djaidwnduwk28bcsb
+log_directory: /tmp
+annotations:
+ containerd.io/nydus-prefetch: |
+    [
+    {"image": "dockerhub.kubekey.local/dfns/wordpress:nydus_latest", "prefetch": "/path/to/wordpress_prefetch.txt"}
+    ]
+
+linux: {}
+
+EOF
+
+crictl runp pod.yaml
+```
+The files that need to be prefetched are written in a txt file, and the path and specific content of the txt file can be customized by the user. The following is an example of a prefetched file:
+```console
+sudo tee wordpress_prefetch.txt <<- EOF
+/usr/bin/env
+/lib/x86_64-linux-gnu/ld-2.31.so
+/etc/ld.so.cache
+/lib/x86_64-linux-gnu/libc-2.31.so
+/bin/bash
+/lib/x86_64-linux-gnu/libtinfo.so.6.2
+/lib/x86_64-linux-gnu/libdl-2.31.so
+/etc/nsswitch.conf
+EOF
+```
+
+The prefetching NRI plugin also supports the case where the pod configuration file contains multiple containers. In this case, the definition of pod-mutlicontainers.yaml is as follows:
+
+
+```console
+sudo tee pod-mutlicontainers.yaml <<- EOF
+apiVersion: v1
+kind: Pod
+metadata:
+  name: multi-containers-pod
+  namespace: default
+  attempt: 1
+  uid: hdishd83djaidwnduwk28bcsb
+log_directory: /tmp
+annotations:
+  containerd.io/nydus-prefetch: |
+    [
+    {"image": "dockerhub.kubekey.local/dfns/busybox:nydus_latest", "prefetch": "/path/to/busybox_prefetch.txt"},
+    {"image": "dockerhub.kubekey.local/dfns/ubuntu:nydus_latest", "prefetch": "/path/to/ubuntu_prefetch.txt"},
+    {"image": "dockerhub.kubekey.local/dfns/wordpress:nydus_latest", "prefetch": "/path/to/wordpress_prefetch.txt"}
+    ]
+
+linux: {}
+
+spec:
+  containers:
+    - name: container-1
+      image: dockerhub.kubekey.local/dfns/busybox:nydus_latest
+      command: ["ls"]
+    - name: container-2
+      image: dockerhub.kubekey.local/dfns/ubuntu:nydus_latest
+      command: ["ls"]
+    - name: container-3
+      image: dockerhub.kubekey.local/dfns/wordpress:nydus_latest
+      command: ["ls"]
+EOF
+```
+
+
+
+Note that the naming of keys in annotations is fixed, and the values in annotations is user self-defined.
+After create a pod, image reference and path of prefetch list will received by nydus-snapshotter.
+
+## Nydus-snapshotter starts nydusd
+Nydusd daemon will start when the container is created. We start a container like below.
+```console
+sudo tee wordpress.yaml <<- EOF
+metadata:
+  name: wordpress
+image:
+  image: dockerhub.kubekey.local/dfns/wordpress:nydus_latest
+log_path: wordpress.0.log
+linux: {}
+EOF
+
+crictl pull dockerhub.kubekey.local/dfns/wordpress:nydus_latest
+crictl create <pod_id> wordpress.yaml pod.yaml
+```
+
+Then nydus-snapshotter will start a daemon by command as follows.
+```editorconfig
+ /usr/local/bin/nydusd fuse --thread-num 4 --config /var/lib/containerd-nydus/config/cjmnum3c3al8js5p3740/config.json --bootstrap /var/lib/containerd-nydus/snapshots/22/fs/image/image.boot --mountpoint /var/lib/containerd-nydus/snapshots/22/mnt --apisock /var/lib/containerd-nydus/socket/cjmnum3c3al8js5p3740/api.sock --log-level info --log-rotation-size 100 --prefetch-files /home/runner/prefetchfiles/wordpress_prefetch.txt
+ ```
+
+According the parameter `--prefetch-files`, we have implemented file prefetching through lazy loading.