[Bug] Blockchain db overwritten when -wal and -shm files are present. Verified on windows with chia v2.1.1

[BrandtH22]

What happened?

When starting chia on a windows client without removing the -shm and -wal files, the blockchain db is overwritten to the state that was present when the -shm and -wal files were created. In addition the chia client will remain on the Starting Services screen when this occurs.

This effects all windows users who need to upgrade their db for version 2.1+ and have already updated their config file.

Workaround - windows users will need to manually remove the -shm and -wal files when using the blockchain db snapshot.

Steps to reproduce (confirmed bug on Windows 11, confirmed this bug is NOT present on Mac Ventura 13.4.1):

1. Start with an empty db directory
2. Start chia and let it begin syncing the full node from scratch (this creates the db file and the -shm and -wal files)
3. Stop Chia and remove the created db and move unpacked snapshot db into the directory
4. Restart chia (the created -shm and -wal files are left in the directory with the snapshot db)

• The client overwrites the snapshot db (noticeable by file size)
• The client remains on the starting services screen

Version

2.1.1

What platform are you using?

Windows

What ui mode are you using?

GUI

Relevant log output

No response

[arvidn]

my understanding is that the shm and wal files are temporary files used by sqlite. If the node is shut down gracefully, I would not expect those files to stick around. But leaving them in place, migrating the DB, I can see that sqlite may be attempting to apply the pending changes stored in the wal onto the new database file.

I'm not sure what we can do about this. Perhaps we can include empty shm and wal files in our torrent file, to ensure those get cleared out as well.

[BrandtH22]

my understanding is that the shm and wal files are temporary files used by sqlite. If the node is shut down gracefully, I would not expect those files to stick around. But leaving them in place, migrating the DB, I can see that sqlite may be attempting to apply the pending changes stored in the wal onto the new database file.

I'm not sure what we can do about this. Perhaps we can include empty shm and wal files in our torrent file, to ensure those get cleared out as well.

I know in the past (likely prior to 1.8.2 if memory serves) the shm and wal files were being deleted during a graceful shutdown as intended; however, this seems to no longer be the case (I have verified this on windows and mac that the shm and wal files are still present even after a graceful shutdown on v2.1.1).

Including empty files in the snapshot might work but I was thinking that this likely identifies an issue with how those files are being handled during shutdown and startup; however, I am not proficient enough with sqlite to know where to start digging.

[wjblanke]

Arvid also suggests including a 0 length shm and wal file in the interim. We'll need time to address the fix. Brandt can u see if there was a regression in the Mac behavior and when?

[BrandtH22]

Arvid also suggests including a 0 length shm and wal file in the interim. We'll need time to address the fix. Brandt can u see if there was a regression in the Mac behavior and when?

Hey, I have been testing but have yet to reproduce this issue on Mac (unable to test on linux for another week or so).
Currently it seems this is only an issue on windows.

To help users with this, I have submitted a pr to our docs faq that adds a note about the wal and shm files (should have that merged within the hour):
Chia-Network/chia-docs#405

[github-actions]

This issue has not been updated in 14 days and is now flagged as stale. If this issue is still affecting you and in need of further review, please comment on it with an update to keep it from auto closing in 7 days.

[github-actions]

This issue was automatically closed because it has been flagged as stale, and subsequently passed 7 days with no further activity from the submitter or watchers.