Can incremental snapshots be restored at all?

[2fst4u]

I'm a bit confused by the restore process. I was hoping to be able to schedule backups and have snapshots be taken, allowing the CSI snapshots to be restored in the regular way of creating a PVC by referencing the snapshot.

I understand velero now removes snapshots once a backup is done so that it's not accidentally removed if the namespace is deleted, which I understand.

But is there now no way for a snapshot to be restored? Is the only way to restore data to do a full PVC restore?

My concern there is I really have no way of knowing my backups are working because the only way to restore them to test them is to destroy the PVC, to restore over it - unless you have another cluster to restore into, which I don't have.

Am I missing something?

[kaovilai]

is there now no way for a snapshot to be restored?

No. VolumeSnapshotContent remains in the cluster. It is intentionally disconnected from VolumeSnapshot so it lives past namespace deletion.

VSC will only get deleted when backup gets deleted

On restore velero create a volumesnapshot object referencing the original volumesnapshotcontent and will be able to create a PVC using VolumeSnapshot as source.

I really have no way of knowing my backups are working

Check there are volumesnapshotcontents with your backup name in its labels/annotations.

[kaovilai]

velero/pkg/exposer/csi_snapshot.go

Lines 143 to 153 in 9dcfe16

	retained, err := csi.RetainVSC(ctx, e.csiSnapshotClient, vsc)
	if err != nil {
	return errors.Wrap(err, "error to retain volume snapshot content")
	}
	curLog.WithField("vsc name", vsc.Name).WithField("retained", (retained != nil)).Info("Finished to retain VSC")
	err = csi.EnsureDeleteVS(ctx, e.csiSnapshotClient, volumeSnapshot.Name, volumeSnapshot.Namespace, csiExposeParam.OperationTimeout)
	if err != nil {
	return errors.Wrap(err, "error to delete volume snapshot")
	}

Here we first make sure VSC is retained, then VS is deleted.
Thus you can simulate disaster by deleting NS of workload without VSC deletion and restore the NS later.

[kaovilai]

If we had NOT retained VSC manually in this case, then yes, a VS deletion = VSC deletion = poof no restore

[2fst4u]

I get that the VSC is retained which is good, but then what I'm confused about is don't you need the VS to restore a PVC from? Which has now since been deleted?

[kaovilai]

There is a way to delete VS without deleting VSC.

VS itself has no data but references VSC which has the data.

[2fst4u]

I get that the VS can be deleted without the VSC but I'm just trying to wrap my head around how you get the data. Manually making a VS to reference it and then manually making a PVC to reference that doesn't seem very user friendly, I'm just trying to figure out if there is an easy way.

[kaovilai]

the only way to restore them to test them is to destroy the PVC, to restore over it

Not true.

You can create VolumeSnapshot pointing to existing VolumeSnapshotContent.
You can then create PVC with volumeSource set to the VolumeSnapshot.

[2fst4u]

Ah ok, so it's a bit of manual intervention required. Is there an easier way to do it than creating a VS manifest and applying it to the cluster? Or is this really the only way to do it?

[kaovilai]

idk.. an alias/bash func that does the same thing? The limitation is with kube. You would need to change kube to allow PVC dataSource from VSC then you can skip VS creation.

The dataSourceRef field may contain different types of objects, while the dataSource field only allows PVCs and VolumeSnapshots.

[2fst4u]

All good I just wanted to make sure I wasn't missing something. Thank you for the info, much appreciated.