Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add keadm offline installation #639

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 60 additions & 1 deletion docs/setup/install-with-keadm.md
Original file line number Diff line number Diff line change
Expand Up @@ -548,4 +548,63 @@ It provides a flag for users to specify kubeconfig path, the default path is `/r

### Node

`keadm reset` or `keadm deprecated reset` will stop `edgecore` and it doesn't uninstall/remove any of the pre-requisites.
`keadm reset` or `keadm deprecated reset` will stop `edgecore` and it doesn't uninstall/remove any of the pre-requisites.

## offline installation

In some case, user need to install KubeEdge components in an isolated environment without internet access. Keadm does not have a command line parameter like `offline-mode`, but as long as the matching images are preloaded in the environment, offline installation can also be achieved.

### Cloud Side Offline Installation

`keadm init` would make network requests in the following cases:
1. When no version information is specified, or incorrect version information is specified, it will access [lastversion](https://kubeedge.io/latestversion) to get the latest KubeEdge version.
2. Charts installed by Helm, depending on the configuration settings, will rely on [kubeedge/cloudcore](https://hub.docker.com/r/kubeedge/cloudcore), [kubeedge/iptables-manager](https://hub.docker.com/r/kubeedge/iptables-manager) and [kubeedge/controller-manager](https://hub.docker.com/r/kubeedge/controller-manager) images. By default, only the cloudcore image is required.

**Note**: Helm Chart is packaged into keadm binary using go embed, so no extra network request is needed.

if the version information is correctly set, the request to lastversion will not happen, for example

```shell
keadm init --advertise-address="THE-EXPOSED-IP" --kubeedge-version=1.18.0 --kube-config=/root/.kube/config
```
In the host with internet access, pull and save the images with tools like `docker`, `crictl`, `skopeo`, etc. Here is an example using `docker`:

```shell
docker pull kubeedge/cloudcore:v1.18.0
docker save -o cloudcore.tar kubeedge/cloudcore:v1.18.0
```
The image can be imported directly into the working node in the offline environment, or imported into the private image center in the offline environment. The former requires the developer to ensure that the node where the deployment service is located is the node where the image is imported, and the latter needs to replace the image address of pod in the form of a configuration file or command line parameter when `keadm init` is executed.

if the cloudcore image is in the private image center, the address is `self-registry.io/kubeedge/cloudcore:v1.18.0`, replace it during initialization through the `--set` command line parameter, as shown below:

```shell
keadm init --advertise-address="THE-EXPOSED-IP" --kubeedge-version=1.18.0 --set cloudCore.image.repository=self-registry.io/kubeedge/cloudcore --kube-config=/root/.kube/config
```

### Edge Core Offline Installation

As mentioned earlier, `keadm join` will download [kubeedge/installation-package](https://hub.docker.com/r/kubeedge/installation-package) and extract the `edgecore` binary file from it. In an offline environment, users need to import the installation-package image into the edge node to be added to the cluster. According to the container runtime of the edge node, choose the appropriate tool for import, for example, docker uses the `docker` command, containerd uses the `ctr` command, etc. The following is an operation using `docker` as an example:

```shell
# run in a networked environment
docker pull kubeedge/installation-package:v1.18.0
docker save -o installation-package.tar kubeedge/installation-package:v1.18.0
# copy installation-package.tar to the edge node, then import it into the node
docker load -i installation-package.tar
```

If use nerdctl as the command line tool, it is worth noting that nerdctl supports the concept of namespace, so when importing images, you need to specify the `k8s.io` namespace, for example:

```shell
nerdctl --namespace k8s.io load -i installation-package.tar
```

In addition to the `installation-package` image, by default, the [eclipse-mosquitto](https://hub.docker.com/_/eclipse-mosquitto) image needs to be imported into the edge node. The reason is that the Helm Chart installed by `keadm init` contains a Mosquitto DaemonSet deployment, which is used to run the MQTT proxy service on the edge node. The following is an example of docker operation:

```shell
# run in a networked environment
docker pull eclipse-mosquitto:1.6.15
docker save -o eclipse-mosquitto.tar eclipse-mosquitto:1.6.15
# copy eclipse-mosquitto.tar to the edge node, then import it into the node
docker load -i eclipse-mosquitto.tar
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember if it needed a kubeedge/pause image, and I found out via the keadm config images list that it needed a kubeedge/admission image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Piwriw

After reviewing the code, I found that kubeedge/admission is a KubeEdge image, but it is not installed by keadm.

the kubeedge/admission can only be installed manually by https://github.com/kubeedge/kubeedge/tree/master/build/admission

I add a issue for the problem kubeedge/kubeedge#5904

the doc is about the keadm function. At this moment, kubeedge/admission, which cannot be installed by keadm , can be omitted from the documentation.However, I will also modify the document to explain the situation with kubeedge/admission.

As for the issue with kubeedge/pause, I will respond after completing my research.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Piwriw the puase image for join operation isn't what must do, based on kubeedge/kubeedge#5309.

Only when the user modifies the configuration parameters of containerd, the keadm join require a pause image.

Original file line number Diff line number Diff line change
Expand Up @@ -456,3 +456,68 @@ KubeEdge edgecore is running, For logs visit: /var/log/kubeedge/edgecore.log
### 节点

`keadm reset` 将停止 `edgecore` ,并且不会卸载/删除任何先决条件。


## 离线安装

在一些应用场景中,用户需要在隔离互联网的环境中进行 KubeEdge 的组件安装。Keadm 在命令行参数中并没有类似于 offline-mode 这样的参数,但只要在环境中预先加载匹配的镜像,也能实现离线安装的目的。

### 云侧离线安装

`keadm init` 在安装过程中会在以下情况下对网络资源进行请求;

1. 未指定版本信息,或者错误指定版本信息时, 会访问 [lastversion](https://kubeedge.io/latestversion) 获取最新 KubeEdge 版本
2. 经由 Helm 安装的 Chart ,根据配置设置会依赖 [kubeedge/cloudcore](https://hub.docker.com/r/kubeedge/cloudcore)、[kubeedge/iptables-manager](https://hub.docker.com/r/kubeedge/iptables-manager)和 [kubeedge/controller-manager](https://hub.docker.com/r/kubeedge/controller-manager) 镜像,默认配置下仅依赖 cloudcore 镜像。

**注意**: Helm Chart 被以 go embed 方式打包到 keadm 二进制文件中,所以不需要额外的网络请求。

只要正确地设置了 KubeEdge 的版本信息,对于 lastversion 的请求就不会发生,例如

```shell
keadm init --advertise-address="THE-EXPOSED-IP" --kubeedge-version=1.18.0 --kube-config=/root/.kube/config
```

在可联网主机,通过工具将镜像拉取和导出,具体的工具根据您自身的情况选择,例如 `docker`,`crictl`,`skopeo` 等。

以下是以 `docker` 为工具,`cloudcore` 镜像为目标进行操作,其他镜像类似:

```shell
docker pull kubeedge/cloudcore:v1.18.0
docker save -o cloudcore.tar kubeedge/cloudcore:v1.18.0
```

离线环境下镜像的导入要不然直接导入到工作节点上,要不然导入是离线环境中私有镜像中心,前者需要开发者保证部署服务所在节点是导入镜像的节点,后者则需要在 `keadm init` 时通过配置文件或者命令行参数的形式替换服务的镜像地址。

假设 cloudcore 镜像在私有镜像中心的地址为 `self-registry.io/kubeedge/cloudcore:v1.18.0`,通过 `--set` 命令行参数在初始化进行替换,如下示例:

```shell
keadm init --advertise-address="THE-EXPOSED-IP" --kubeedge-version=1.18.0 --set cloudCore.image.repository=self-registry.io/kubeedge/cloudcore --kube-config=/root/.kube/config
```

### 边缘侧离线安装

如前所言,`keadm join` 执行过程中会下载 [kubeedge/installation-package](https://hub.docker.com/r/kubeedge/installation-package),并从中提取 `edgecore` 二进制文件。在离线环境中,用户需要将 installation-package 镜像导入到待加入集群的边缘节点上,根据边缘节点的容器运行时选择合适的工具进行导入,例如 docker 使用 `docker` 命令,containerd 使用 `ctr` 命令等,以下是以 `docker` 为例的操作:

```shell
# 在有网络的环境中执行
docker pull kubeedge/installation-package:v1.18.0
docker save -o installation-package.tar kubeedge/installation-package:v1.18.0
# 将 installation-package.tar 拷贝到边缘节点,然后导入节点中
docker load -i installation-package.tar
```

如果使用 nerdctl 作为命令行工具的话,需要注意 nerdctl 支持 namespace 的概念,所以在导入镜像时需要指定 `k8s.io` namespace,例如:

```shell
nerdctl --namespace k8s.io load -i installation-package.tar
```

除了 `installation-package` 镜像外,默认情况下,在边缘节点还需要导入 [eclipse-mosquitto](https://hub.docker.com/_/eclipse-mosquitto) 镜像,原因是 `keadm init` 安装的 Helm Chart 中包含了一个 Mosquitto 的 DaemonSet 部署,用于在边缘节点上运行 MQTT 代理服务,以下为 docker 例子的操作:

```shell
# 在有网络的环境中执行
docker pull eclipse-mosquitto:1.6.15
docker save -o eclipse-mosquitto.tar eclipse-mosquitto:1.6.15
# 将 eclipse-mosquitto.tar 拷贝到边缘节点,然后导入节点中
docker load -i eclipse-mosquitto.tar
```