feat: migrate from file list

[smrz2001]

This PR adds support for migrating IPFS blocks from an input file list. It also introduces both an offset and a limit that can be used to paginate migrations, whether from a directory or from a file list.

This is needed for being able to apply incremental ZFS snapshots without having to re-migrate the entire blockstore. The list of added blocks can be extracted from the incremental snapshot and input to the migration so that only new blocks are migrated. This will dramatically reduce the amount of time needed for migrating new blocks added after the first snapshot was taken.

Note that paginating over a directory might have unpredictable results if it is receiving live updates.

[nathanielc]

Good addition, some UX changes proposed but nothing significant

[nathanielc]

Can you document the contents of this file? Is it newline delimited paths? Are the paths relative to the input dir? Or absolute?

[nathanielc]

There are skip and take methods on streams using StreamExt. Which should simplify this logic to let lines = lines.skip(offset).take(limit); and the simply have the while loop without any bounds checks.

[AaronGoldman]

tokio::io::Lines did not have the StreamExt functions. tokio_stream::wrappers::LinesStream might let us do it that way.

[smrz2001]

I figured it out.

[smrz2001]

tokio::io::Lines did not have the StreamExt functions. tokio_stream::wrappers::LinesStream might let us do it that way.

Oh wait, lol, that's what I ended up using.

[nathanielc]

I don't think we should support limit/offset for the read dir case. From the docs:

The order in which this iterator returns entries is platform and filesystem dependent.

We cannot guarantee that limit/offset will be meaningful. And in practice the order in which blocks where traversed appeared to be random. Meaning it was not in lexicographical order on my local filesystem nor in various production filesystems we used.

I propose if limit/offset are not None and there is no input file list then we error.

[nathanielc]

https://doc.rust-lang.org/nightly/std/fs/fn.read_dir.html

[AaronGoldman]

I will likely be nearly stable on any one disk as the tree in the filesystem try's to not change much. If the user wants a guaranty that they cover the entire ipfs repo then they should use make a file list file.
I wander if we should make a CLI argument for making a sorted file list file and a CLI for subtracting one from another

or maybe in the make file

# List files in directory at migration_1 and sort them alphabetically
ls -l ipfs/blocks | sort > migration_1.txt
 
CERAMIC_ONE_INPUT_FILE_LIST_PATH=migration_1.txt \
ceramic-one migrations from-ipfs 

# List files in directory at migration_2 and sort them alphabetically
ls -l ipfs/blocks | sort > migration_2.txt

# Find files in migration_2 that are not in migration_1
comm -23  migration_2.txt migration_1.txt > delta

CERAMIC_ONE_INPUT_FILE_LIST_PATH=delta.txt \
ceramic-one migrations from-ipfs 

[nathanielc]

Do we need the feature? I'd rather not add an unused feature that has a potential to break in mysterious ways.

[smrz2001]

I took it out for now and added the check.