indexer: add persistence plugin #88
Conversation
|
|
||
| describe("Sqlite Persistence", () => { | ||
| let db: Database<sqlite3.Database, sqlite3.Statement>; | ||
| let store: SqlitePersistence; |
There was a problem hiding this comment.
Should be called persistence :)
| }); | ||
|
|
||
| indexer.hooks.hook("connect:before", async ({ request }) => { | ||
| const store = new SqlitePersistence(db); |
There was a problem hiding this comment.
I think it's possible to instantiate the persistence once and share the instance between the hooks (like you do with db).
|
|
||
| indexer.hooks.hook("sink:flush", async ({ endCursor }) => { | ||
| const store = new SqlitePersistence(db); | ||
| if (endCursor) store.put(endCursor); |
There was a problem hiding this comment.
| if (endCursor) store.put(endCursor); | |
| if (endCursor) { | |
| store.put(endCursor); | |
| } |
| import { afterAll, beforeAll, describe, expect, it } from "vitest"; | ||
| import { SqlitePersistence } from "./persistence"; | ||
|
|
||
| describe("Sqlite Persistence", () => { |
There was a problem hiding this comment.
One issue I have with these tests is that they're very dependent on the order in which they're run.
I would change them to have only 2 cases:
Test 1: assert there's no data, then insert value, check that value was stored, update value, check that value was updated
Test 2: assert there's no data, insert value, check value was stored, delete value, check there's no data.
For each test, you should create a db/persistence instance specific to that test. That way if in the future we run tests in parallel or if vitest changes how they order tests, test won't break.
|
|
||
| type SqliteArgs = ISqlite.Config; | ||
|
|
||
| export function sqlitePersistence<TFilter, TBlock, TRet>(args: SqliteArgs) { |
There was a problem hiding this comment.
BTW now that we have a mock stream, we can test plugins end to end.
Add a tests where you setup a mock indexer with the sqlitePersistence plugin, run it over a sequence of messages and then assert that the stored cursor is the last endCursor in your stream.
| id TEXT NOT NULL PRIMARY KEY, | ||
| order_key INTEGER NOT NULL, | ||
| unique_key TEXT, | ||
| UNIQUE(order_key, unique_key) |
There was a problem hiding this comment.
I don't think UNIQUE is needed. We can safely remove it.
There was a problem hiding this comment.
Not needed because we have id in this table.
jaipaljadeja
force-pushed
the
feat/persistence-plugin
branch
from
27d3bde to
e56cb30
Compare
jaipaljadeja
force-pushed
the
feat/persistence-plugin
branch
from
e56cb30 to
9c93e23
Compare
add persistence plugin to persist the indexer’s state between restarts.