Skip to content

Commit

Permalink
docs: changes to singularity documentation and document routing (#90)
Browse files Browse the repository at this point in the history
Signed-off-by: zhaoxinxin <[email protected]>
  • Loading branch information
Liam-Zhao authored Apr 9, 2024
1 parent 26463b5 commit d841bdc
Show file tree
Hide file tree
Showing 34 changed files with 88 additions and 76 deletions.
3 changes: 2 additions & 1 deletion docs/getting-started/installation/binaries.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: binaries
title: Binaries
slug: /getting-started/installation/binaries/
---

This guide shows how to install the Dragonfly. Dragonfly can be installed either from source, or from pre-built binary releases.
Expand Down Expand Up @@ -246,7 +247,7 @@ telnet 127.0.0.1 65002

Configure Dfdaemon yaml file, The default path in Linux is `/etc/dragonfly/dfget.yaml` in linux,
The default path in Darwin is `$HOME/.dragonfly/config/dfget.yaml`,
refer to [Dfdaemon](../../reference/configuration/dfdaemon.md)
refer to [Dfdaemon](../../reference/configuration/dfdaemon.md).

Set the `scheduler.manager.netAddrs.addr` address in the configuration file to your actual address.
Configuration content is as follows:
Expand Down
1 change: 1 addition & 0 deletions docs/getting-started/installation/helm-charts.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: helm-charts
title: Helm Charts
slug: /getting-started/installation/helm-charts/
---

Documentation for deploying Dragonfly on kubernetes using helm.
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/quick-start.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: quick-start
title: Quick Start
slug: /getting-started/quick-start/
slug: /getting-started/quick-start
---

Table of contents:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: containerd
title: containerd
slug: /setup/runtime/containerd
slug: /operations/integrations/container-runtime/containerd/
---

Documentation for setting Dragonfly's container runtime to containerd.
Expand Down
2 changes: 1 addition & 1 deletion docs/operations/integrations/container-runtime/cri-o.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: cri-o
title: CRI-O
slug: /setup/runtime/cri-o
slug: /operations/integrations/container-runtime/cri-o/
---

Use dfget daemon as registry mirror for CRI-O
Expand Down
2 changes: 1 addition & 1 deletion docs/operations/integrations/container-runtime/docker.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: docker
title: Docker
slug: /setup/runtime/docker
slug: /operations/integrations/container-runtime/docker/
---

Use dfget daemon as HTTP proxy for docker daemon
Expand Down
1 change: 1 addition & 0 deletions docs/operations/integrations/container-runtime/nydus.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: nydus
title: Nydus
slug: /operations/integrations/container-runtime/nydus/
---

This document will help you experience how to use dragonfly with nydus.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
---
id: singularity
title: Singularity/Apptainer
slug: /setup/runtime/singularity
---

This documentation will help you to integrate Dragonfly into
Singularity/Apptainer container runtime.

Table of contents:

- [Oras Resource Client Mode](./singularity/oras_resource_client.md)
- [Oras Resource Client Mode](./singularity/oras-resource-client.md)
- [Proxy Mode](./singularity/proxy.md)
Original file line number Diff line number Diff line change
@@ -1,22 +1,20 @@
---
id: singularity-oras-resource-client
title: Oras Resource Client Mode
slug: /setup/runtime/singularity/oras-resource-client
title: ORAS Resource Client
slug: /operations/integrations/container-runtime/singularity/oras-resource-client/
---

## Oras Resource Client {#oras-resource-client}

We can use oras resource client to pull image using Dragonfly, the key is the scheme used for the resource client.
Documentation for setting Dragonfly's container runtime to ORAS Resource Client.

This method of image pull through Dragonfly is more efficient when compared to proxy method as
it avoids TLS termination, reduces CPU time and download time as it creates hardlink(insead of copy)
for subsequent file download after downloading the image from source for first time.

## Quick Start {#quick-start}

### Step 1: Configure dfget daemon {#step-1-configure-dfget-daemon}
## Configure dfget daemon {#configure-dfget-daemon}

To use oras resource client to pull image ensure below configuraion in `/etc/dragonfly/dfget.yaml:`.
Configure Dfdaemon yaml file, The default path in Linux is `/etc/dragonfly/dfget.yaml` in linux,
The default path in Darwin is `$HOME/.dragonfly/config/dfget.yaml`,
refer to [Dfdaemon](../../../../reference/configuration/dfdaemon.md).

```yaml
# Peer task storage option.
Expand All @@ -33,7 +31,7 @@ storage:
# Set to ture for reusing underlying storage for same task id.
multiplex: true

# The singularity oras resources, most of it is same with https scheme
# The singularity ORAS resources, most of it is same with https scheme.
oras:
proxy:
dialTimeout: 30s
Expand All @@ -46,26 +44,23 @@ oras:
insecureSkipVerify: true
```
### Step 2: Pull images through oras resource client {#step-2-pull-images-through-oras-resource-client}
Through the above steps, we can start to validate if Dragonfly works as expected.
## ORAS Resource Client downloads images through Dragonfly {#oras-resource-client-downloads-images-through-Dragonfly}
And you can pull the image through oras resource client as below:
Pull the image through ORAS Resource Client:
```shell
dfget -u "oras://hostname/path/image:tag" -O /path/to/output
```

### Step 3: Validate Dragonfly {#step-3-validate-dragonfly}
### Verify {#verify}

You can execute the following command to
check if the image is distributed via Dragonfly.
You can execute the following command to check if the image is distributed via Dragonfly.

```shell
grep "peer task done" /var/log/dragonfly/daemon/core.log
```

If the output of command above has content like
The expected output is as follows:

```shell
{
Expand Down
36 changes: 18 additions & 18 deletions docs/operations/integrations/container-runtime/singularity/proxy.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
id: singularity-proxy
title: Proxy Mode
slug: /setup/runtime/singularity/proxy
title: HTTP Proxy
slug: /operations/integrations/container-runtime/singularity/proxy/
---

Uses dfget daemon as proxy for Singularity/Apptainer.
Documentation for setting up Dragonfly's container runtime as an HTTP Proxy for Singularity/Apptainer.

## Step 1: Generate CA certificate {#step-1-generate-ca-certificate}
## Generate CA certificate {#generate-ca-certificate}

Generate an RSA private key.

Expand Down Expand Up @@ -57,18 +57,21 @@ openssl x509 -req -days 36500 -extfile openssl.conf \
-extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt
```

## Step 2: Configure dfget daemon {#step-2-configure-dfget-daemon}
## Configure dfget daemon {#configure-dfget-daemon}

To use dfget daemon as proxy, first you need to append a proxy rule in
`/etc/dragonfly/dfget.yaml`, This will proxy
`your.private.registry's` requests for image layers:
Configure Dfdaemon yaml file, The default path in Linux is `/etc/dragonfly/dfget.yaml` in linux,
The default path in Darwin is `$HOME/.dragonfly/config/dfget.yaml`,
refer to [Dfdaemon](../../../../reference/configuration/dfdaemon.md).

Set the `registryMirror.url` and `hijackHTTPS.hosts.regx` addresses under the configuration
file to your actual addresses. Configuration content is as follows:

```yaml
registryMirror:
# When enable, using header "X-Dragonfly-Registry" for remote instead of url.
dynamic: true
# URL for the registry mirror.
url: <your.private.registry>
url: your_registry_mirror_url
# Whether to ignore https certificate errors.
insecure: false
# Optional certificates if the remote server uses self-signed certificates.
Expand All @@ -87,29 +90,26 @@ hijackHTTPS:
cert: ca.crt
key: ca.key
hosts:
- regx: <your.private.registry>
- regx: your_hijack_https_hosts
```
## Step 3: Pull images with proxy {#step-4-pull-images-with-proxy}
Through the above steps, we can start to validate if Dragonfly works as expected.
## HTTP Proxy downloads images through Dragonfly {#http-proxy-downloads-images-through-Dragonfly}
And you can pull the image through proxy as below:
Pull images through HTTP Proxy:
```bash
no_proxy='' NO_PROXY='' HTTPS_PROXY=127.0.0.1:65001 singularity pull oras://hostname/path/image:tag
```

## Step 4: Validate Dragonfly {#step-4-validate-dragonfly}
### Verify {#verify}

You can execute the following command to
check if the image is distributed via Dragonfly.
You can execute the following command to check if the image is distributed via Dragonfly.

```shell
grep "peer task done" /var/log/dragonfly/daemon/core.log
```

If the output of command above has content like
The expected output is as follows:

```shell
{
Expand Down
1 change: 1 addition & 0 deletions docs/operations/integrations/container-runtime/stargz.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: stargz
title: eStargz
slug: /operations/integrations/container-runtime/stargz/
---

This document will help you experience how to use dragonfly with eStargz.
Expand Down
1 change: 1 addition & 0 deletions docs/operations/integrations/git-lfs.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: git-lfs
title: Git LFS
slug: /operations/integrations/git-lfs/
---

## What is Git LFS?
Expand Down
1 change: 1 addition & 0 deletions docs/operations/integrations/harbor.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: harbor
title: Harbor
slug: /operations/integrations/harbor/
---

This document will help you experience how to use dragonfly with harbor.
Expand Down
1 change: 1 addition & 0 deletions docs/operations/integrations/hugging-face.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: hugging-face
title: Hugging Face
slug: /operations/integrations/hugging-face/
---

This document will help you experience how to use dragonfly with hugging face.
Expand Down
1 change: 1 addition & 0 deletions docs/operations/integrations/torchserve.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: torchserve
title: TorchServe
slug: /operations/integrations/torchserve/
---

This document will help you experience how to use dragonfly with [TorchServe](https://github.com/pytorch/serve).
Expand Down
1 change: 1 addition & 0 deletions docs/operations/integrations/triton-server.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: triton-server
title: Triton Server
slug: /operations/integrations/triton-server/
---

This document will help you experience how to use dragonfly with [TritonServe](https://github.com/pytorch/serve).
Expand Down
2 changes: 1 addition & 1 deletion docs/operations/integrations/upgrade.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: upgrade
title: Upgrade
slug: /setup/upgrade
slug: /operations/integrations/upgrade/
---

## Upgrade the cluster deployed by Helm {#upgrade-the-cluster-deployed-by-helm}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: binaries
title: 使用二进制文件安装
slug: /getting-started/installation/binaries/
---

文档的目标是帮助您快速开始使用源码或构建的二进制版本部署 Dragonfly。
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: helm-charts
title: Helm Charts
slug: /getting-started/installation/helm-charts/
---

文档的目标是帮助您快速开始使用 Helm Charts 部署 Dragonfly。
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: containerd
title: containerd
slug: /setup/runtime/containerd
slug: /operations/integrations/container-runtime/containerd/
---

文档的目标是帮助您将 Dragonfly 的容器运行时设置为 containerd。
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: cri-o
title: CRI-O
slug: /setup/runtime/cri-o
slug: /operations/integrations/container-runtime/cri-o/
---

使用 dfdaemon 支持 CRI-O
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: docker
title: Docker
slug: /setup/runtime/docker
slug: /operations/integrations/container-runtime/docker/
---

使用 dfget daemon 作为 docker daemon 的 http 代理
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: nydus
title: Nydus
slug: /operations/integrations/container-runtime/nydus/
---

本文档将帮助您将 Dragonfly 与 Nydus 一起使用。
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
---
id: singularity
title: Singularity/Apptainer
slug: /setup/runtime/singularity
---

该文档将帮助你集成 Dragonfly 到 Singularity/Apptainer。

集成方式:

- [Oras 资源客户端模式](./singularity/oras_resource_client.md)
- [Oras 资源客户端模式](./singularity/oras-resource-client.md)
- [HTTP Proxy 模式](./singularity/proxy.md)
Loading

0 comments on commit d841bdc

Please sign in to comment.