replace O_DIRECT by fadvise()

[nraynaud]

Hi all,
I am working on ZFS SR support, and I am constantly butting on O_DIRECT which is not supported by ZFS (the symptom is EINVAL on open()).

This time, it's XAPI calling vhd-tool with --direct hard-coded

• https://github.com/xapi-project/xen-api/blob/master/ocaml/xapi/vhd_tool_wrapper.ml#L160
• https://github.com/xapi-project/xen-api/blob/master/ocaml/xapi/vhd_tool_wrapper.ml#L73

And through vhd-tool it comes here. I suspect the intent of --direct is to avoid filling the disk cache in this instance. Both operations are simple in-order complete read and in-order complete write.

I think those cases could be covered by posix_fadvise() and fsync().

I am ready to create a patch, but I would like to know where you think I should make the change before I get to work.

Thanks,
Nicolas.

[lindig]

I am not familiar with file I/O enough to understand why O_DIRECT is desirable in the first place. But I want to caution that we need to be careful to discover when certain options are available only on some file systems. We had a problem of trying to use SEEK_DATA in lseek(2) which is not supported on all file system.

[djs55]

IIRC the original rationale for O_DIRECT was to avoid filling up the disk cache as you suggest. The thinking was that the data being streamed wouldn't be read more than once, so there was no point evicting other cache entries just to cache it.

[nraynaud]

Thanks for your input, after googling a bit, I understand why you dropped the ball on posix and used O_DIRECT, there is basically no reasonable solution to the problem in linux.

I think I will make my changes in vhd-tool, only on the streaming operations, and evict the cache with POSIX_FADV_DONTNEED.
My thinking is that when writing a vhd file from the network nobody had a pre-existing need for a cache on a file that was not yet there, it's mostly for importing new VMs. For the read operation they are mostly sequential and nobody really need a random cache either, they are mostly exporting snapshots to the network.

What do you think?