Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: simplify kbench metrics to focus on key performance indicators #15

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
93 changes: 93 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
name: build
on:
push:
branches:
- master
- v*
tags:
- v*
pull_request:
workflow_dispatch:
jobs:
build:
name: Build binaries
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Build binaries
run: make

- uses: codecov/codecov-action@v4
with:
files: ./coverage.out
flags: unittests
token: ${{ secrets.CODECOV_TOKEN }}

- name: Upload binaries
uses: actions/upload-artifact@v4
with:
name: binaries_artifact
path: ./bin/*
Comment on lines +12 to +32
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Consider refining the build job for better specificity and efficiency.

The build job covers essential steps, but there are opportunities for improvement:

  1. The make command is used without a specific target. Consider specifying the exact target(s) needed for the build to ensure consistency and avoid unnecessary work.

  2. The artifact upload step includes all files in ./bin/. To optimize storage and download times, consider being more selective about which files are uploaded.

Here's a suggested improvement for the 'Build binaries' step:

- name: Build binaries
  run: make build-specific-target  # Replace with your specific make target

And for the 'Upload binaries' step:

- name: Upload binaries
  uses: actions/upload-artifact@v4
  with:
    name: binaries_artifact
    path: |
      ./bin/kbench
      ./bin/other-important-binary  # Add specific binary names

This will ensure only the necessary binaries are uploaded as artifacts.


build_push_image:
name: Build and push image
runs-on: ubuntu-latest
needs: build
if: ${{ startsWith(github.ref, 'refs/heads/') || startsWith(github.ref, 'refs/tags/') }}
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Download binaries
uses: actions/download-artifact@v4
with:
name: binaries_artifact
path: ./bin/

- name: Add executable permission
run: |
chmod +x ./bin/*

- name: Copy bin folder to package
run: |
cp -r ./bin ./package/

# For multi-platform support
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Declare branch
run: |
echo "branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> "$GITHUB_ENV"

- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}

# longhornio/kbench image
- name: docker-publish
if: ${{ startsWith(github.ref, 'refs/heads/') }}
uses: docker/build-push-action@v5
with:
context: ./
push: true
platforms: linux/amd64,linux/arm64
tags: longhornio/kbench:${{ env.branch }}-head
file: package/Dockerfile

Comment on lines +73 to +83
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Consider enhancing image tag for better traceability.

The Docker image build and push step for branches is well-configured with multi-platform support. However, the current tagging strategy can be improved.

Consider adding a timestamp or short commit hash to the image tag for better traceability. This can help identify exactly which commit the image was built from. Here's a suggested improvement:

- name: Set image tag
  run: |
    echo "IMAGE_TAG=${{ env.branch }}-$(date +%Y%m%d%H%M%S)-${GITHUB_SHA::8}" >> $GITHUB_ENV

- name: Build and push Docker image
  uses: docker/build-push-action@v5
  with:
    context: ./
    push: true
    platforms: linux/amd64,linux/arm64
    tags: longhornio/kbench:${{ env.IMAGE_TAG }}
    file: package/Dockerfile

This will create a tag like master-20231101123045-a1b2c3d4, providing more detailed information about when and from which commit the image was built.

- name: docker-publish-with-tag
if: ${{ startsWith(github.ref, 'refs/tags/') }}
uses: docker/build-push-action@v5
with:
context: ./
push: true
platforms: linux/amd64,linux/arm64
tags: longhornio/kbench:${{ github.ref_name }}
file: package/Dockerfile

Comment on lines +84 to +93
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Enhance tagging strategy for release versions.

The Docker image build and push step for tags is well-configured, but the tagging strategy can be improved to provide more flexibility for users.

Consider the following enhancements:

  1. Add a 'latest' tag for the most recent release.
  2. Include major and minor version tags.

Here's a suggested improvement:

- name: Generate Docker tags
  if: startsWith(github.ref, 'refs/tags/')
  run: |
    VERSION=${GITHUB_REF#refs/tags/v}
    MAJOR=${VERSION%%.*}
    MINOR=${VERSION%.*}
    echo "DOCKER_TAGS=longhornio/kbench:latest,longhornio/kbench:${VERSION},longhornio/kbench:${MINOR},longhornio/kbench:${MAJOR}" >> $GITHUB_ENV

- name: Build and push Docker image for tags
  if: startsWith(github.ref, 'refs/tags/')
  uses: docker/build-push-action@v5
  with:
    context: ./
    push: true
    platforms: linux/amd64,linux/arm64
    tags: ${{ env.DOCKER_TAGS }}
    file: package/Dockerfile

This will create tags like latest, 1.2.3, 1.2, and 1 for a release tag v1.2.3, providing users with more options for version selection.

4 changes: 2 additions & 2 deletions Dockerfile.dapper
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
FROM golang:1.17-alpine
FROM golang:1.23-alpine

RUN apk -U add bash git gcc musl-dev docker vim less file curl wget ca-certificates

ENV DAPPER_ENV REPO TAG
ENV DAPPER_SOURCE /go/src/github.com/yasker/kbench/
ENV DAPPER_SOURCE /go/src/github.com/longhorn/kbench/
ENV DAPPER_OUTPUT ./bin
ENV DAPPER_DOCKER_SOCKET true
ENV HOME ${DAPPER_SOURCE}
Expand Down
20 changes: 7 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,14 @@ QUICK MODE: DISABLED
=====================
IOPS (Read/Write)
Random: 98368 / 89200
Sequential: 108513 / 107636
CPU Idleness: 68%

Bandwidth in KiB/sec (Read/Write)
Random: 542447 / 514487
Sequential: 552052 / 521330
CPU Idleness: 99%

Latency in ns (Read/Write)
Random: 97222 / 44548
Sequential: 40483 / 44690
CPU Idleness: 72%
```

Expand All @@ -39,17 +36,14 @@ QUICK MODE: DISABLED
850-pro-raw vs 850-pro-ext4 : Change
IOPS (Read/Write)
Random: 96,735 / 89,565 vs 98,135 / 88,990 : 1.45% / -0.64%
Sequential: 107,729 / 107,352 vs 110,843 / 107,805 : 2.89% / 0.42%
CPU Idleness: 68% vs 66% : -2%

Bandwidth in KiB/sec (Read/Write)
Random: 549,521 / 519,141 vs 549,610 / 511,755 : 0.02% / -1.42%
Sequential: 551,825 / 522,936 vs 552,337 / 520,364 : 0.09% / -0.49%
CPU Idleness: 98% vs 98% : 0%

Latency in ns (Read/Write)
Random: 82,899 / 44,437 vs 114,331 / 45,104 : 37.92% / 1.50%
Sequential: 40,335 / 44,767 vs 41,741 / 45,271 : 3.49% / 1.13%
CPU Idleness: 72% vs 73% : 1%
```

Expand Down Expand Up @@ -109,26 +103,26 @@ By default:
Step to deploy:
1. One line to start benchmarking your default storage class:
```
kubectl apply -f https://raw.githubusercontent.com/yasker/kbench/main/deploy/fio.yaml
kubectl apply -f https://raw.githubusercontent.com/longhorn/kbench/main/deploy/fio.yaml
```
1. Observe the Result:
```
kubectl logs -l kbench=fio -f
```
1. Cleanup:
```
kubectl delete -f https://raw.githubusercontent.com/yasker/kbench/main/deploy/fio.yaml
kubectl delete -f https://raw.githubusercontent.com/longhorn/kbench/main/deploy/fio.yaml
```

Note: a single benchmark for FIO will take about 6 minutes to finish.

See [./deploy/fio.yaml](https://github.com/yasker/kbench/blob/main/deploy/fio.yaml) for available options.
See [./deploy/fio.yaml](https://github.com/longhorn/kbench/blob/main/deploy/fio.yaml) for available options.

#### Deploy Comparison Benchmark in Kubernetes cluster

1. Get a local copy of `fio-cmp.yaml`
```
wget https://raw.githubusercontent.com/yasker/kbench/main/deploy/fio-cmp.yaml
wget https://raw.githubusercontent.com/longhorn/kbench/main/deploy/fio-cmp.yaml
```
1. Set the storage class for each volume you want to compare.
* By default, it's `local-path` vs `longhorn`.
Expand All @@ -153,16 +147,16 @@ See [./deploy/fio.yaml](https://github.com/yasker/kbench/blob/main/deploy/fio.ya

Note: a comparison benchmark for FIO will take about 12 minutes to finish.

See [./deploy/fio-cmp.yaml](https://github.com/yasker/kbench/blob/main/deploy/fio-cmp.yaml) for available options.
See [./deploy/fio-cmp.yaml](https://github.com/longhorn/kbench/blob/main/deploy/fio-cmp.yaml) for available options.

#### Run Single Volume Benchmark as Container Locally

```
docker run -v /volume yasker/kbench:latest /volume/test.img
docker run -v /volume longhornio/kbench:latest /volume/test.img
```
e.g.
```
docker run -e "SIZE=100M" -v /volume yasker/kbench:latest /volume/test.img
docker run -e "SIZE=100M" -v /volume longhornio/kbench:latest /volume/test.img
```

#### Run Single Volume Benchmark as a Binary Locally
Expand Down
14 changes: 1 addition & 13 deletions deploy/fio-cmp.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,35 +40,23 @@ spec:
spec:
containers:
- name: kbench
image: yasker/kbench:latest
image: longhornio/kbench:latest
imagePullPolicy: Always
env:
# - name: MODE
# value: "quick" # for debugging
# - name: MODE
# value: "random-read-iops"
# - name: MODE
# value: "sequential-read-iops"
# - name: MODE
# value: "random-read-bandwidth"
# - name: MODE
# value: "sequential-read-bandwidth"
# - name: MODE
# value: "random-read-latency"
# - name: MODE
# value: "sequential-read-latency"
# - name: MODE
# value: "random-write-iops"
# - name: MODE
# value: "sequential-write-iops"
# - name: MODE
# value: "random-write-bandwidth"
# - name: MODE
# value: "sequential-write-bandwidth"
# - name: MODE
# value: "random-write-latency"
# - name: MODE
# value: "sequential-write-latency"
- name: MODE
value: "full" # run all tests
- name: FIRST_VOL_NAME
Expand Down
14 changes: 1 addition & 13 deletions deploy/fio.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,35 +25,23 @@ spec:
spec:
containers:
- name: kbench
image: yasker/kbench:latest
image: longhornio/kbench:latest
imagePullPolicy: Always
env:
# - name: MODE
# value: "quick" # for debugging
# - name: MODE
# value: "random-read-iops"
# - name: MODE
# value: "sequential-read-iops"
# - name: MODE
# value: "random-read-bandwidth"
# - name: MODE
# value: "sequential-read-bandwidth"
# - name: MODE
# value: "random-read-latency"
# - name: MODE
# value: "sequential-read-latency"
# - name: MODE
# value: "random-write-iops"
# - name: MODE
# value: "sequential-write-iops"
# - name: MODE
# value: "random-write-bandwidth"
# - name: MODE
# value: "sequential-write-bandwidth"
# - name: MODE
# value: "random-write-latency"
# - name: MODE
# value: "sequential-write-latency"
- name: MODE
value: "full" # run all tests
- name: FILE_NAME
Expand Down
3 changes: 3 additions & 0 deletions fio/bandwidth-include.fio
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
bs=128K
iodepth=16
numjobs=4
18 changes: 4 additions & 14 deletions fio/bandwidth-quick.fio
Original file line number Diff line number Diff line change
@@ -1,19 +1,9 @@
[rand-read-bw]
readwrite=randread
include bw-include.fio
include quick-include.fio

[rand-write-bw]
readwrite=randwrite
include bw-include.fio
include quick-include.fio

[seq-read-bw]
[seq-read-bandwidth]
readwrite=read
include bw-include.fio
include bandwidth-include.fio
include quick-include.fio

[seq-write-bw]
[seq-write-bandwidth]
readwrite=write
include bw-include.fio
include bandwidth-include.fio
include quick-include.fio
6 changes: 3 additions & 3 deletions fio/bandwidth-random-read.fio
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[rand-read-bw]
[rand-read-bandwidth]
readwrite=randread
include bw-include.fio
include common-include.fio
include bandwidth-include.fio
include common-include.fio
4 changes: 2 additions & 2 deletions fio/bandwidth-random-write.fio
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[rand-write-bw]
[rand-write-bandwidth]
readwrite=randwrite
include bw-include.fio
include bandwidth-include.fio
include common-include.fio
6 changes: 3 additions & 3 deletions fio/bandwidth-sequential-read.fio
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[seq-read-bw]
[seq-read-bandwidth]
readwrite=read
include bw-include.fio
include common-include.fio
include bandwidth-include.fio
include common-include.fio
4 changes: 2 additions & 2 deletions fio/bandwidth-sequential-write.fio
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[seq-write-bw]
[seq-write-bandwidth]
readwrite=write
include bw-include.fio
include bandwidth-include.fio
include common-include.fio
19 changes: 5 additions & 14 deletions fio/bandwidth.fio
Original file line number Diff line number Diff line change
@@ -1,19 +1,10 @@
[rand-read-bw]
readwrite=randread
include bw-include.fio
include common-include.fio

[rand-write-bw]
readwrite=randwrite
include bw-include.fio
[seq-write-bandwidth]
readwrite=write
include bandwidth-include.fio
include common-include.fio

[seq-read-bw]
[seq-read-bandwidth]
readwrite=read
include bw-include.fio
include bandwidth-include.fio
include common-include.fio

[seq-write-bw]
readwrite=write
include bw-include.fio
include common-include.fio
2 changes: 0 additions & 2 deletions fio/bw-include.fio

This file was deleted.

Loading
Loading