Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Cloudflare KV Storage #1298

Merged
merged 70 commits into from
Apr 16, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
c49806a
feat: add cloudflare kv
Geun-Oh Mar 21, 2024
79a9ae2
test: add test
Geun-Oh Mar 21, 2024
adad63a
chore: add config
Geun-Oh Mar 21, 2024
8b6245b
chore: initialize mod
Geun-Oh Mar 21, 2024
5817eaf
chore: add contribute configuration
Geun-Oh Mar 21, 2024
c424c66
fix: rollback lint
Geun-Oh Mar 21, 2024
85133f6
fix: change kv => cloudflarekv
Geun-Oh Mar 21, 2024
8028341
chore: remove godotenv package
Geun-Oh Mar 21, 2024
d6e0360
fix: remove local env testing
Geun-Oh Mar 21, 2024
7410373
add: add error return
Geun-Oh Mar 21, 2024
115d290
Update .github/workflows/release-drafter-cloudflarekv.yml
Geun-Oh Mar 21, 2024
551373a
fix: remove v2 path
Geun-Oh Mar 21, 2024
cc2ea9a
fix: remove line
Geun-Oh Mar 21, 2024
7a7807b
feat: preallocate key with make
Geun-Oh Mar 21, 2024
5f53fa0
feat: add benchmark tests
Geun-Oh Mar 21, 2024
c83d7f8
docs: kv => Cloudflare KV
Geun-Oh Mar 21, 2024
e9bd7e9
add: set default value
Geun-Oh Mar 21, 2024
398c183
fix: remove stale version
Geun-Oh Mar 22, 2024
19be481
feat: add DB Conn & its testing
Geun-Oh Mar 22, 2024
796363e
docs: add ref on root
Geun-Oh Mar 22, 2024
01d6a94
fix: update go version
Geun-Oh Mar 22, 2024
1c65d38
test: change Conn test with NotNil
Geun-Oh Mar 22, 2024
65db3ca
docs: add Conn() ref
Geun-Oh Mar 22, 2024
6a5b6eb
test: add test settings
Geun-Oh Mar 28, 2024
f022a14
Merge branch 'gofiber:main' into main
Geun-Oh Mar 28, 2024
4798a58
test: add testing shell yaml file
Geun-Oh Mar 28, 2024
154e150
fix: clarify go version for testing
Geun-Oh Mar 28, 2024
b8d752b
fix: fix testing jobs
Geun-Oh Mar 28, 2024
fdda850
fix: run wrangler dev server at background
Geun-Oh Mar 28, 2024
12eb0f1
chore: remove unused library
Geun-Oh Mar 28, 2024
8ef7cd7
Merge branch 'main' into main
Geun-Oh Mar 28, 2024
5e03e38
Update test-cloudflarekv.yml
gaby Mar 28, 2024
033ebf7
Update test-cloudflarekv.yml
gaby Mar 28, 2024
f0076b5
Update cloudflarekv/README.md
gaby Mar 30, 2024
5b26a0b
Merge branch 'gofiber:main' into main
Geun-Oh Mar 30, 2024
510d583
feat: add cloudflare worker init step at benchmark
Geun-Oh Mar 30, 2024
a5bee00
Update cloudflarekv/README.md
ReneWerner87 Apr 2, 2024
5ef27d1
fix: remove logging in benchmark test
Geun-Oh Apr 2, 2024
9d3cdce
Merge branch 'main' into main
Geun-Oh Apr 2, 2024
26e9d83
Merge branch 'main' into main
Geun-Oh Apr 8, 2024
d8f61c2
Update benchmark.yml
ReneWerner87 Apr 8, 2024
f4f69d7
Update benchmark.yml
ReneWerner87 Apr 8, 2024
f0533e6
fix: add git core.fileMode true
Geun-Oh Apr 8, 2024
bc9b6c3
Merge branch 'main' of https://github.com/Geun-Oh/storage
Geun-Oh Apr 8, 2024
554fa73
fix: remove chmod
Geun-Oh Apr 8, 2024
2c4232d
Merge branch 'main' into main
Geun-Oh Apr 9, 2024
64725d9
Update cloudflarekv/cloudflarekv.go
ReneWerner87 Apr 9, 2024
b72e733
Update cloudflarekv/cloudflarekv.go
ReneWerner87 Apr 9, 2024
235e47e
fix: add chmod
Geun-Oh Apr 9, 2024
126df74
Update cloudflarekv.go
ReneWerner87 Apr 9, 2024
ad8df22
Add Cloudflare KV Storage
ReneWerner87 Apr 9, 2024
6d5d57d
Add Cloudflare KV Storage
ReneWerner87 Apr 9, 2024
7e321c1
fix: check in github permission change
Geun-Oh Apr 9, 2024
73bb4f1
Add Cloudflare KV Storage
ReneWerner87 Apr 9, 2024
a5b09ad
Merge remote-tracking branch 'origin/main'
ReneWerner87 Apr 9, 2024
1ae7fa0
Add Cloudflare KV Storage
ReneWerner87 Apr 9, 2024
1527f88
fix: fix dev env with 8787 port
Geun-Oh Apr 9, 2024
64b08d4
Add Cloudflare KV Storage
ReneWerner87 Apr 9, 2024
9590f1a
Merge branch 'main' into main
Geun-Oh Apr 10, 2024
64ba9a2
fix: set cloudflarekv schedule interval daily
Geun-Oh Apr 11, 2024
5cd7843
fix: set setup-node version 4
Geun-Oh Apr 11, 2024
29245c6
fix: init storage per test & remove default storage
Geun-Oh Apr 11, 2024
ffeab66
fix: set key test
Geun-Oh Apr 11, 2024
8bbaa80
fix: add cloudflare testing validation
Geun-Oh Apr 11, 2024
b397ca3
fix: valid get with list-up method
Geun-Oh Apr 11, 2024
0f8625f
fix: test get event with listing kv pairs
Geun-Oh Apr 11, 2024
da47a1f
Merge branch 'main' into main
Geun-Oh Apr 12, 2024
d4b1c83
fix: make get works
Geun-Oh Apr 12, 2024
8e3d3f9
fix: revalidate cache of getter
Geun-Oh Apr 12, 2024
71bbf1e
Merge branch 'main' into main
Geun-Oh Apr 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@ updates:
- "🤖 Dependencies"
schedule:
interval: "daily"
- package-ecosystem: "gomod"
directory: "/cloudflarekv/" # Location of package manifests
labels:
- "🤖 Dependencies"
schedule:
interval: "weekly"
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved
- package-ecosystem: "gomod"
directory: "/couchbase/" # Location of package manifests
labels:
Expand Down
44 changes: 44 additions & 0 deletions .github/release-drafter-cloudflarekv.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name-template: "CloudflareKV - v$RESOLVED_VERSION"
tag-template: "cloudflarekv/v$RESOLVED_VERSION"
tag-prefix: cloudflarekv/v
include-paths:
- cloudflarekv
categories:
- title: "🚀 New"
labels:
- "✏️ Feature"
- title: "🧹 Updates"
labels:
- "🧹 Updates"
- "🤖 Dependencies"
- title: "🐛 Fixes"
labels:
- "☢️ Bug"
- title: "📚 Documentation"
labels:
- "📒 Documentation"
change-template: "- $TITLE (#$NUMBER)"
change-title-escapes: '\<*_&' # You can add # and @ to disable mentions, and add ` to disable code blocks.
exclude-contributors:
- dependabot
- dependabot[bot]
version-resolver:
major:
labels:
- "major"
minor:
labels:
- "minor"
- "✏️ Feature"
patch:
labels:
- "patch"
- "📒 Documentation"
- "☢️ Bug"
- "🤖 Dependencies"
- "🧹 Updates"
default: patch
template: |
$CHANGES
**Full Changelog**: https://github.com/$OWNER/$REPOSITORY/compare/$PREVIOUS_TAG...cloudflarekv/v$RESOLVED_VERSION
Thank you $CONTRIBUTORS for making this update possible.
19 changes: 19 additions & 0 deletions .github/workflows/release-drafter-cloudflarekv.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: Release Drafter Cloudflare KV
on:
push:
# branches to consider in the event; optional, defaults to all
branches:
- master
- main
paths:
- "cloudflarekv/**"
jobs:
draft_release_dynamodb:
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: release-drafter/release-drafter@v6
with:
config-name: release-drafter-cloudflarekv.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
29 changes: 29 additions & 0 deletions .github/workflows/test-cloudflarekv.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
on:
push:
branches:
- master
- main
paths:
- "cloudflarekv/**"
pull_request:
paths:
- "cloudflarekv/**"
name: "Tests Cloudflare KV"
jobs:
Tests:
runs-on: ubuntu-latest
strategy:
matrix:
go-version:
- 1.19.x
- 1.20.x
- 1.21.x
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved
steps:
- name: Fetch Repository
uses: actions/checkout@v4
- name: Install Go
uses: actions/setup-go@v5
with:
go-version: "${{ matrix.go-version }}"
- name: Run Test
gaby marked this conversation as resolved.
Show resolved Hide resolved
run: cd ./cloudflarekv && go test ./... -v -race
119 changes: 119 additions & 0 deletions cloudflarekv/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
---
id: cloudflarekv
title: Cloudflare KV
---

![Release](https://img.shields.io/github/v/tag/gofiber/storage?filter=cloudflarekv*)
[![Discord](https://img.shields.io/discord/704680098577514527?style=flat&label=%F0%9F%92%AC%20discord&color=00ACD7)](https://gofiber.io/discord)
![Test](https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-cloudflarekv.yml?label=Tests)
![Security](https://img.shields.io/github/actions/workflow/status/gofiber/storage/gosec.yml?label=Security)
![Linter](https://img.shields.io/github/actions/workflow/status/gofiber/storage/linter.yml?label=Linter)

A kv storage driver using [cloudflare/cloudflare-go](https://github.com/cloudflare/cloudflare-go).
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved

**Note: Requires Go 1.19 and above**
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved

### Table of Contents

- [Signatures](#signatures)
- [Installation](#installation)
- [Examples](#examples)
- [Config](#config)
- [Default Config](#default-config)

### Signatures

```go
func New(config ...Config) Storage
func (s *Storage) Get(key string) ([]byte, error)
func (s *Storage) Set(key string, val []byte, exp time.Duration) error
func (s *Storage) Delete(key string) error
func (s *Storage) Reset() error
func (s *Storage) Close() error
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved
```

### Installation

Cloudflare KV is tested on the 2 last [Go versions](https://golang.org/dl/) with support for modules. So make sure to initialize one first if you didn't do that yet:
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved

```bash
go mod init github.com/<user>/<repo>
```

And then install the Cloudflare KV implementation:

```bash
go get github.com/gofiber/storage/cloudflarekv/v2
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved
```

### Examples

Import the storage package.

```go
import "github.com/gofiber/storage/cloudflarekv/v2"
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved
```

You can use the following possibilities to create a storage.

Key must be an API Token generated with at least `Account.Workers KV Storage` permission.

Check [Create API Token](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/) docs to generate.
gaby marked this conversation as resolved.
Show resolved Hide resolved

```go
// Initialize default config
store := cloudflarekv.New()

store := cloudflarekv.New(cloudflarekv.Config{
Key: "",
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
Email: "",
AccountID: "fiber",
NamespaceID: "fiber",
Reset: false,
})

```

### Config

```go
type Config struct {

// Cloudflare Auth Token
//
// Optional. Default is ""
Key string

// Cloudflare Email
//
// Optional. Default is ""
Email string

// Account id
//
// Optional. Default is "fiber"
AccountID string

// Namespace id
//
// Optional. Default is "fiber"
NamespaceID string

// Reset clears any existing keys in existing Table
//
// Optional. Default is false
Reset bool
}
```

### Default Config

```go
var ConfigDefault = Config{
Key: "",
Email: "",
AccountID: "fiber",
NamespaceID: "fiber",
Reset: false,
}
```
124 changes: 124 additions & 0 deletions cloudflarekv/cloudflarekv.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
package cloudflarekv

import (
"context"
"log"
"time"

"github.com/cloudflare/cloudflare-go"
)

type Storage struct {
api *cloudflare.API
email string
accountID string
namespaceID string
}

func New(config ...Config) *Storage {

Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved
cfg := configDefault(config...)

api, err := cloudflare.NewWithAPIToken(cfg.Key)

if err != nil {
log.Println("Error with cloudflare api initialization")
}
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved

storage := &Storage{
api: api,
email: cfg.Email,
accountID: cfg.AccountID,
namespaceID: cfg.NamespaceID,
}

return storage
}

func (s *Storage) Get(key string) ([]byte, error) {
resp, err := s.api.GetWorkersKV(context.Background(), cloudflare.AccountIdentifier(s.accountID), cloudflare.GetWorkersKVParams{NamespaceID: s.namespaceID, Key: key})

if err != nil {
log.Println("Error occur in GetWorkersKV")
return nil, err
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
}

return resp, nil
}

func (s *Storage) Set(key string, val []byte, exp time.Duration) error {

_, err := s.api.WriteWorkersKVEntry(context.Background(), cloudflare.AccountIdentifier(s.accountID), cloudflare.WriteWorkersKVEntryParams{
NamespaceID: s.namespaceID,
Key: key,
Value: val,
})

if err != nil {
log.Println("Error occur in WriteWorkersKVEntry")
return err
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
}

return nil
}

func (s *Storage) Delete(key string) error {
_, err := s.api.DeleteWorkersKVEntry(context.Background(), cloudflare.AccountIdentifier(s.accountID), cloudflare.DeleteWorkersKVEntryParams{
NamespaceID: s.namespaceID,
Key: key,
})

if err != nil {
log.Println("Error occur in WriteWorkersKVEntry")
return err
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
}

return nil
}

func (s *Storage) Reset() error {

var cursor string

for {
resp, err := s.api.ListWorkersKVKeys(context.Background(), cloudflare.AccountIdentifier(s.accountID), cloudflare.ListWorkersKVsParams{
NamespaceID: s.namespaceID,
Cursor: cursor,
})

if err != nil {
log.Println("Error occur in ListWorkersKVKeys")
return err
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
}

keys := []string{}
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved

for _, element := range resp.Result {
name := element.Name
keys = append(keys, name)
}

_, err = s.api.DeleteWorkersKVEntries(context.Background(), cloudflare.AccountIdentifier(s.accountID), cloudflare.DeleteWorkersKVEntriesParams{
NamespaceID: s.namespaceID,
Keys: keys,
})

if err != nil {
log.Println("Error occur in DeleteWorker")
return err
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
ReneWerner87 marked this conversation as resolved.
Show resolved Hide resolved
}

if len(resp.Cursor) == 0 {
log.Println("No keys left in the namespace")
break
}

cursor = resp.Cursor
}

return nil
}

func (s *Storage) Close() error {
return nil
}
Geun-Oh marked this conversation as resolved.
Show resolved Hide resolved
Loading