Implement SQLite Store using gorm and relational approach

[surik]

Describe your changes

This approach involves loading associated tables into slices and then converting them into maps. This technique is introducing additional fields prefixed with 'G' (shows Gorm relation), allowing for granular resource management without many changes to existing structures. While this approach introduces complexities, it serves the goal of achieving a functional implementation while minimizing disruptions to the current architecture.

One significant advantage is the elimination of the complexity associated with the lookup table used in the document-oriented approach. This simplification not only enhances query performance but also contributes to improved write speed, especially as the number of accounts grows.

Another key advantage introduced by the new relational approach is the ability to operate on separated data structures, such as accounts, users, and peers, independently. This level of separation empowers more granular and focused updates, ensuring that modifications to one resource do not require loading the entire account. But this may require additional changes in the Store interface and the Account Manager.

Benchmark Results:

$ go test  ./management/server -run Benchmark_ -bench=.  -benchmem
goos: darwin
goarch: amd64
pkg: github.com/netbirdio/netbird/management/server
cpu: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
BenchmarkTest_StoreWrite/FileStore_Write_100-8         	     146	  14532698 ns/op	 6130954 B/op	    8549 allocs/op
BenchmarkTest_StoreWrite/SqliteStore_Write_100-8       	     278	   4623969 ns/op	  210247 B/op	    2714 allocs/op
BenchmarkTest_StoreWrite/FileStore_Write_500-8         	      44	  25448032 ns/op	13313935 B/op	   15779 allocs/op
BenchmarkTest_StoreWrite/SqliteStore_Write_500-8       	     242	   5269740 ns/op	  210193 B/op	    2714 allocs/op
BenchmarkTest_StoreWrite/FileStore_Write_1000-8        	      25	  45887216 ns/op	26595502 B/op	   30004 allocs/op
BenchmarkTest_StoreWrite/SqliteStore_Write_1000-8      	     205	   5795530 ns/op	  210193 B/op	    2715 allocs/op
BenchmarkTest_StoreWrite/FileStore_Write_2000-8        	      12	 100217523 ns/op	50800290 B/op	   58819 allocs/op
BenchmarkTest_StoreWrite/SqliteStore_Write_2000-8      	     178	   6024594 ns/op	  210188 B/op	    2715 allocs/op
BenchmarkTest_StoreRead/FileStore_Read_100-8           	  434073	      2602 ns/op	    3192 B/op	      27 allocs/op
BenchmarkTest_StoreRead/SqliteStore_Read_100-8         	    1483	    694025 ns/op	  118992 B/op	    2113 allocs/op
BenchmarkTest_StoreRead/FileStore_Read_500-8           	  412197	      2588 ns/op	    3192 B/op	      27 allocs/op
BenchmarkTest_StoreRead/SqliteStore_Read_500-8         	    1731	    676320 ns/op	  118894 B/op	    2113 allocs/op
BenchmarkTest_StoreRead/FileStore_Read_1000-8          	  439562	      2681 ns/op	    3192 B/op	      27 allocs/op
BenchmarkTest_StoreRead/SqliteStore_Read_1000-8        	    1508	    708517 ns/op	  118790 B/op	    2113 allocs/op

We can see that the write speed of the relational approach is better compared with the result shown in #1030.
Unfortunately, the reading speed is worse but this might be due to the amount of copying I have to do when loading an account. This can be improved.

ERD diagram:

Issue ticket number and link

Checklist

• Is it a bug fix
• Is a typo/documentation fix
• Is a feature enhancement
• It is a refactor
• Created tests that fail without the change (if possible)
• Extended the README / documentation, if necessary

[surik]

I managed to make all tests pass with SQLite implementation. The testing matrix is extended to cover both file and SQLite store scenarios. The only exception is Windows as I keep getting:

testing.go:1206: TempDir RemoveAll cleanup: remove C:\Windows\TEMP\TestSqlite_GetTokenIDByHashedToken2617861885\001\store.db: The process cannot access the file because it is being used by another process.

This seems related to the cleanup procedure and needs further investigation. For now, SQLite store tests are disabled for Windows.

I also added a sub-command that can help with the migration of json storage to the SQLite store:

./management migration --help
Migrate JSON file store to SQLite store. Please make a backup of the store json file before running this command.

This command reads the content of {datadir}/store.json and migrates it to {datadir}/store.db that can be used by SQLite store driver.

Usage:
  netbird-mgmt migration [--datadir directory] [--log-file console] [flags]

By default, management uses JSON file store as before. To use SQLite there is an option in the config file StoreKind which can be set either to Sqlite or JsonFile.

[surik]

I managed to fix Sqlite store tests on Windows by not using cache=shared. Also, made a few manual tests locally and could not find any issues. Saying that this one is ready for review.

[pappz]

Because the global lock doing nothing then this line may cause problem. Here we write out the account but without account lock.

[surik]

The global lock was implemented for SQLite. I managed to eliminate the usage of one global lock-in here. We may consider addressing the rest later.

Additionally, I implemented a rollback command in case someone wants to migrate back from SQLite to the JSON store.

Also, I made a few correctness checks by migrating a test store.json to store.db and then back to store.json. This may look like the following:

# you need store.json in CWD
$ go build -o mgmt ./management
$ ./mgmt sqlite-migration up --log-file console --datadir .
$ mv store.json store.before.json
$ ./mgmt sqlite-migration down --log-file console --datadir .

Then you have both store.before.json and store.json. Just run diff on them or use any other tool to compare two JSONs. There may be some differences mostly related to the way how JSON is serialized. You may see things like this:

12c12
<                     "AccountID": "",
---
>                     "AccountID": "b000jehgmce4vh1360cc",
...
312c312
<                     "Peers": null
---
>                     "Peers": []
...
3332,3334c3334,3338
<             "Routes": null,
<             "NameServerGroups": null,
<             "DNSSettings": null,
---
>             "Routes": {},
>             "NameServerGroups": {},
>             "DNSSettings": {
>                 "DisabledManagementGroups": null
>             },

Which mostly are equivalents in the current file store implementation.

Management should behave the same way with store.json after up and down migration.