Feeding a Builder parts of a file at a time

[jjl]

Currently, so far as I can tell, it's not possible to stream a file's contents as it becomes available - it is necessary to read an entire file into memory at a time to add it to an archive.

It would be awesome if it were possible to build up an entry's body incrementally, such as in response to filesystem events from tokio.

[alexcrichton]

The append method takes a Read, would that suffice for this?

[jjl]

It's entirely possible I'm wrong about this, but I think that would block up the tokio event loop?

[alexcrichton]

Ah no yeah that's true, this library is not currently compatible with async I/O

[jjl]

I'd be willing to put in the effort to make it so, if you've got ideas about how it might be done.

[alexcrichton]

Hm sure! I haven't put a lot of thought into how this library would support futures and async I/O but my guess is that it'd be relatively low-level. It might be the case that we need to basically just add a new append method that attempts to write data after the header is written and returns how much was written, basically being compatible with "would block"

[jjl]

That sounds plausible for a first start. It would be nice to get recursive directory adding etc. working with async too if the user opts into it (tokio feature flag like your flate2 crate?)

[alexcrichton]

Seems plausible to me!

[jjl]

I've had a go at something slightly better (imo) than this initial solution, but I can't quite finish it because I'm slightly confused by what this code (from prepare_header) is doing:

        let mut data2 = data.chain(io::repeat(0).take(0));
        append(dst, &header2, &mut data2)?;

In particular, does chaining .take(0) not make this a noop?

[alexcrichton]

Oh dear that is odd! That can probably be removed now, it may just be something that didn't survive refactorings well

[ghost]

Is there any news on this?

[jjl]

On 18 Mar 2019, at 16:23, Boastful Squirrel ***@***.***> wrote: Is there any news on this?

I gave up on the project that needed this and I’m unlikely to look at it again until Tokio is stabilised. /j

[the8472]

Passing the data in parts would also be necessary to build sparse files.

[danieleades]

it would be pretty handy if async interop was a thing that existed