invoices: migrate KV invoices to native SQL for users of KV SQL backends #8831

bhandras · 2024-06-12T10:35:27Z

Change Description

This pull request adds the migration of old key-value (KV) invoices to the new native SQL schema when the --db.use-native-sql flag is set, unless the --db.skip-sql-invoice-migration flag is also specified.

Please note that since we currently do not support running on mixed database backends for users of bbolt or etcd, an additional step is required to migrate their KV database to SQL first. For more context, please see lightninglabs/lndinit#21.

coderabbitai · 2024-06-12T10:35:32Z

Important

Review skipped

Auto reviews are limited to specific labels.

🏷️ Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

invoices/sql_migration.go

bhandras · 2024-11-29T16:20:24Z

Please hold off with the next round of reviews as I'm still investigating some performance issues with larger databases.

bhandras · 2024-12-02T19:36:42Z

Thank you for your patience. Tested the PR with large KV invoice datasets and I believe migration performance is adequate. There's no slowdown and memory use remains constant given batch size. PTAL.

This commit adds the migration_tracker table which we'll use to track if a custom migration has already been done.

This commit introduces support for custom, in-code migrations, allowing a specific Go function to be executed at a designated database version during sqlc migrations. If the current database version surpasses the specified version, the migration will be skipped.

This commit separates the execution of SQL and in-code migrations from their construction. This change is necessary because, currently, the SQL schema is migrated during the construction phase in the lncfg package. However, migrations are typically executed when individual stores are constructed within the configuration builder.

Previously we intentially did not set settled_at and settle_index when inserting a new invoice as those fields are set when we settle an invoice through the usual invoice update. As migration requires that we set these nullable fields, we can safely add them.

Certain invoices may not have a deterministic payment hash. For such invoices we still store the payment hashes in our KV database, but we do not have a sufficient index to retrieve them. This PR adds such index to the SQL database that will be used during migration to retrieve payment hashes.

…a hash The current sqlc GetInvoice query experiences incremental slowdowns during the migration of large invoice databases, primarily due to its complex predicate set. For this specific use case, a streamlined GetInvoiceByHash function provides a more efficient solution, maintaining near-constant lookup times even with extensive table sizes.

This commit runs the invoice migration if the user has a KV SQL backend configured.

ellemouton

Thanks for the updates! Just one or two questions about the strategy here before I do a final, detail orientated review round. I think perhaps the one question re leaning towards duplication rather than adding migration queries to the interface is open for discussion and so very happy to give in there if others disagree with me!

ellemouton · 2024-12-03T11:03:03Z

sqldb/sqlc/migrations/000005_migration_tracker.up.sql

+        -- migration_id is the id of the migration.
+        migration_id TEXT PRIMARY KEY,


still getting to that part of the PR but assuming that the order of migrations is kept track of at a code level then if this is text based?

ellemouton · 2024-12-03T11:06:30Z

sqldb/migrations.go

+	// Version is the schema version at which the migration is applied.
+	Version int


"schema version" as in up file number yeah? if so, what about if we have 2 code-level migrations in a row that depend on each-other/where ordering is important?

hmm ok I see the order is gleaned implicitly from the order in which it is passed to ApplyMigrations

ellemouton · 2024-12-03T11:11:41Z

sqldb/migrations.go

+	// Sort migrations by version to ensure they are applied in order.
+	sort.SliceStable(migrations, func(i, j int) bool {
+		return migrations[i].Version < migrations[j].Version
+	})


slightly scary to me cause I think this doesnt account for the case where the versions are equal. I think maybe we should have an explicit order for these code-level migrations

basically i think it would be cool if there was an overall, explicit version for each migration as one day these can diverge quite a bit but there will always be 1 single absolute DB version that we are talking about then. Thinking like a 1:1 map from: Overall Version to migration:

map[OverallVersionNum] -> Migration

where Migrations has fields: type = sql/code and then a versionNum where that versionNum is the sql level version number or code level version number. We can persist this overall version and use it to know where to start from.

ellemouton · 2024-12-03T11:28:27Z

sqldb/migrations_test.go

+		},
+		{
+			// We use this special case to test that a migration
+			// will never be aplied in case the current version is


s/aplied/applied

ellemouton · 2024-12-03T11:28:53Z

sqldb/migrations_test.go

+	// Some migrations to use for both the failure and success tests. Note
+	// that the migrations are not in order to test that they are executed
+	// in the correct order.
+	migrations := []MigrationConfig{


think we should cover the case of having 2 code level migrations applied directly after eachother on same sql level version

ellemouton · 2024-12-03T11:41:31Z

sqldb/sqlc/migrations/000006_invoice_migration.up.sql

+-- invoice_payment_hashes table contains the hash of the invoices. This table
+-- is used during KV to SQL invoice migration as in our KV representation we
+-- don't have a mapping from hash to add index.
+CREATE TABLE IF NOT EXISTS invoice_payment_hashes (


if we go with duplicating DB state (sql files) and other codecs etc per migration (like we do for our channeldb migrations today) then we would be able to do this no? it would just mean having some duplication. but it might be worth it so that we dont have to have migration methods on the interface and so that we actually can drop these DBs and keep this "live" version clean.

ellemouton · 2024-12-03T11:46:01Z

sqldb/sqlc/queries/invoices.sql

+-- name: GetInvoicePaymentHashByAddIndex :one
+SELECT hash
+FROM invoice_payment_hashes


just confirming understanding: we might have invoices with no add index right? but that is only the case for invoices that defs have preimages and so we would never need to actually call this method for those?

ellemouton · 2024-12-03T11:48:32Z

invoices/sql_migration.go

+		// Clean up the hash index as it's no longer needed.
+		err = tx.ClearInvoiceHashIndex(ctx)
+		if err != nil {
+			return fmt.Errorf("unable to clear invoice hash "+
+				"index: %w", err)
+		}


we can't drop the table right now as there are queries depending on it and the query files are not versioned like the migration files.

but we can technically have copied query files per migration like we do today for channeldb migrations yeah? ie, introduce some duplication in order to keep the live version of the interface clean?

ellemouton · 2024-12-03T11:50:19Z

invoices/sql_store.go

+
+	ClearInvoiceHashIndex(ctx context.Context) error
+
+	GetMigration(ctx context.Context, migrationID string) (
+		sqlc.MigrationTracker, error)
+
+	UpdateMigration(ctx context.Context,
+		arg sqlc.UpdateMigrationParams) error


I think we might just keep them here and remove in the next version we we also remove the temp table.

I think im maybe struggling to picture this move - can you maybe just explain a bit more what we will do in the next version?

bhandras self-assigned this Jun 12, 2024

bhandras added database Related to the database/storage of LND migration labels Jun 12, 2024

bhandras added this to the 0.19.0 milestone Jun 12, 2024

bhandras force-pushed the sql-invoice-migration branch 3 times, most recently from 6682b50 to 338e1f0 Compare June 12, 2024 15:30

bhandras force-pushed the sql-invoice-migration branch 3 times, most recently from d2a329f to 6379a8b Compare June 21, 2024 17:37

bhandras mentioned this pull request Jul 31, 2024

mod: bump kvdb to v1.4.10 #8959

Merged

bhandras force-pushed the sql-invoice-migration branch 2 times, most recently from 5fe92e2 to a7bf598 Compare August 14, 2024 09:38

aakselrod reviewed Sep 5, 2024

View reviewed changes

invoices/sql_migration.go Show resolved Hide resolved

bhandras force-pushed the sql-invoice-migration branch 5 times, most recently from b6f0ac8 to b983851 Compare September 17, 2024 14:42

bhandras changed the title ~~[wip] invoices: migrate KV invoices to native SQL~~ invoices: migrate KV invoices to native SQL for users of KV SQL backends Sep 17, 2024

bhandras force-pushed the sql-invoice-migration branch from b983851 to 706b444 Compare September 17, 2024 14:51

bhandras marked this pull request as ready for review September 17, 2024 14:51

bhandras force-pushed the sql-invoice-migration branch 7 times, most recently from 96f0cbe to bfe4ad5 Compare September 19, 2024 15:08

bhandras force-pushed the sql-invoice-migration branch 3 times, most recently from 0c5dd72 to a124788 Compare December 2, 2024 19:35

bhandras requested review from ellemouton, ziggie1984, Roasbeef and aakselrod December 2, 2024 19:37

bhandras force-pushed the sql-invoice-migration branch 2 times, most recently from f9842ec to 1c0b28a Compare December 2, 2024 20:18

bhandras added 17 commits December 2, 2024 21:42

mod: temporarily replace sqldb with local version

ffb1937

sqldb: add table to track custom SQL migrations

debe842

This commit adds the migration_tracker table which we'll use to track if a custom migration has already been done.

multi: add call to directly insert an AMP sub-invoice

de0ea74

sqldb: add a temporary index to store KV invoice hash to ID mapping

3d60eda

sqldb: remove unused preimage query parameter

aa4ce9c

invoices: extract method to create invoice insertion params

9a9bec1

invoices: add migration code for a single invoice

0a3a2c0

invoices: add migration code that runs a full invoice DB SQL migration

2e2c66a

lnd: run invoice migration on startup

2d4aa8e

This commit runs the invoice migration if the user has a KV SQL backend configured.

itest: add integration test for invoice migration

7d1a558

itest: remove obsolete itest

30c8e8f

docs: update release notes for 0.19.0

f3ae54b

bhandras force-pushed the sql-invoice-migration branch from 1c0b28a to f3ae54b Compare December 2, 2024 20:42

ellemouton reviewed Dec 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

invoices: migrate KV invoices to native SQL for users of KV SQL backends #8831

invoices: migrate KV invoices to native SQL for users of KV SQL backends #8831

bhandras commented Jun 12, 2024 •

edited

Loading

coderabbitai bot commented Jun 12, 2024 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

bhandras commented Nov 29, 2024

bhandras commented Dec 2, 2024

ellemouton left a comment

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

ellemouton Dec 3, 2024

		-- migration_id is the id of the migration.
		migration_id TEXT PRIMARY KEY,

		// Version is the schema version at which the migration is applied.
		Version int

invoices: migrate KV invoices to native SQL for users of KV SQL backends #8831

Are you sure you want to change the base?

invoices: migrate KV invoices to native SQL for users of KV SQL backends #8831

Conversation

bhandras commented Jun 12, 2024 • edited Loading

Change Description

coderabbitai bot commented Jun 12, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

bhandras commented Nov 29, 2024

bhandras commented Dec 2, 2024

ellemouton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bhandras commented Jun 12, 2024 •

edited

Loading

coderabbitai bot commented Jun 12, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)