Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature:support hot reloading through a multi-version InvertedDB. #72

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

KANIOYH
Copy link

@KANIOYH KANIOYH commented Nov 9, 2024

feature

issue:#51
Support hot reloading by multi-version InvertedDB.
Specifically, each request to InvertedDB logs a version number, and each time a new index is loaded, a new version is created.
The data from outdated versions is then released asynchronously.

// Manager -> versionSet:
// 	                   current ↓
// | version-1,ref:1 | version-2,ref:0 |

type Version struct {
	versionId int64 // version id, unique
	dbTable   map[string]*InvertedDB // InvertedDB  held by this version is mapped by file
	oldFiles  []string               // Files that are obsolete for the next version
	dbs       []*InvertedDB
	ref       atomic.Int64
}

Exposefunc (m *InvertedDBManager) Ref() ([]*InvertedDB, int64)
func (m *InvertedDBManager) Unref(versionId int64)
func (m *InvertedDBManager) UpdateFromRedis(ctx context.Context)

To reduce interference on the critical write path for online requests, we build versions not-in-place, minimizing the time spent on write operations.

usage

dbs, versionId := storage.GlobalInvertedDB.Ref()
defer storage.GlobalInvertedDB.Unref(versionId)
for _, inverted := range dbs {
// you code here...
}

test

mock redis chan in repository/redis/inverted_index_test.go

…cifically, each request to InvertedDB logs a version number, and each time a new index is loaded, a new version is created. The data from outdated versions is then released asynchronously. To reduce interference on the critical write path for online requests, we build versions not-in-place, minimizing the time spent on write operations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant