Race condition during initramfs image creation in 1FA mode can lead to data loss

[jaseg]

I just found a nice failure mode. I've been using this piece of software for a while on archlinux. Yesterday, I did a system upgrade, and the system hard crashed while pacman was re-building the initramfs images. The result was that the old images were deleted, but the new images were not yet written to disk.

On a normal arch setup, this is easily fixed by rebooting into a rescue image, then re-building the initramfs images. This does not work here, since the 1FA challenge is only stored inside the initramfs image, and inside the (encrypted) root partition. Thus, this system crash effectively shredded the disk encryption key.

I have a few suggestions on solutions for this.

1. Store a backup of the 1FA challenge in a plain text file on the boot partition, that is never touched or deleted afterwards.
2. Store the 1FA challenge somewhere in the LUKS header, or generate it from persistent data such as the volume UUID.
3. Ensure the old initramfs images are never ever deleted before the new images have been built. (ideally, after the new images have successfully booted at least once)

[Vincent43]

Hi, that's unfortunate situation, backups in some form (challenge, whole passkey, alternate password) are always strongly recommended for such cases I hope you had one of them.

Beside that I don't really see reasonable action on ykfde side - storing challenge on boot partition means where exactly - /boot, /boot/efi /efi? Any of them may be used for boot. This is more a tip for user action who knows their system setup rather than the project.

Touching luks header is asking for trouble - imagine hard system crash while writing to it...only header backup could save you. Persistent challenge may be some idea although changing it would be hard then.

Initramfs creation process is something to discuss in https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio . I'm surprised the old image was gone before the new was done or maybe it was extreme bad timing for crash.

[jaseg]

I think storing a backup of the challenge in /boot during enrollment would be a reasonable action on the ykfde side. You can look at the output of mount(8) to find the right place:

$ mount|grep boot|cut -d' ' -f3|sort|head -n 1
/boot
$ echo $CHALLENGE > $(mount|grep boot|cut -d' ' -f3|sort|head -n 1)/ykfde_challenge_backup_$(date +%Y-%m-%d-%H-%M-%S).txt

You could even put a little shell script next to the backup file that contains the right invocation of ykfde-open to mount the disk from a live rescue system.

Additionally, a big fat warning in the README would be in order until this has been conclusively fixed given the chance of catastrophic data loss. I know it sucks when people get upset about something you consider a hobby project, but if your hobby project is the custodian to terabytes of users' data, you have some responsibility to get things right. "I hope you have a backup" is not an appropriate policy for software with a critical role like ykfde.

I'm not going to raise an issue over at mkinitcpio for two reasons, one is that I don't have the time to take on this fix. The other is that I'm reasonably sure they would respond that you just shouldn't store the only copy of a critical key in these images, and that ensuring this can't happen is out of scope for their project.

[Vincent43]

$ mount|grep boot|cut -d' ' -f3|sort|head -n 1
/boot

On my system /boot isn't boot partition and it's encrypted so storing challenge there will be useless. Solution that sometimes will work for someone isn't something we can rely on. In general storing backup on disk on the same partition where original challenge is stored inside intramfs gives false impression it would be help when needed which may not be the case, i.e. accidental shred of the partition. To be reliable backup must be stored in different place than original (removable storage, paper, etc.)

Additionally, a big fat warning in the README would be in order until this has been conclusively fixed given the chance of catastrophic data loss. I know it sucks when people get upset about something you consider a hobby project, but if your hobby project is the custodian to terabytes of users' data, you have some responsibility to get things right. "I hope you have a backup" is not an appropriate policy for software with a critical role like ykfde.

I agree that we should additionally warn about necessary backups in README. To clarify: backups aren't specific to ykfde usage - the challenge and/or password, the yubikey, the data itself MUST be backed up otherwise you risk data loss regardless of what ykfde does. It doesn't matter if it's hobby or professional project - having backup policy is pretty fundamental everywhere.

The other is that I'm reasonably sure they would respond that you just shouldn't store the only copy of a critical key in these images, and that ensuring this can't happen is out of scope for their project.

Well, leaving system without initramfs is certainly in scope and quite critical issue for mkinitcpio. Whether you used ykfde or not is isn't relevant - you may have used encrypted keyfile stored in initramfs for unlocking. The issue you report is very serious but it's generic rather than ykfde specific.