Unpredictable Behavior when Upperdir is also Mountpoint

[Swivelgames]

There's a use-case where the upperdir provided is also the mountpoint. In this scenario, the directory is fixed and utilized by a particular application or process, and already contains core files. Using OverlayFS, configuration files can be put in place and applied on top of this directory. However, while I'm not entirely sure if this use-case is entirely supported, it is also not forbidden. I would argue for one or the other, in order to prevent confusing or major errors.

Currently, when the Upperdir is also the mountpoint, there is quite a bit of unpredictable behavior that happens. Listing the mountpoint gives you different results depending on what is listing the directory (for instance, /bin/ls gives a certain list, and other applications give a different listing). Still files that are technically there can still be read and written, even by the applications that can't find them when listing a directory.

I'm fully aware of the reason for this instability, because the same location is technically occupied by two different trees. In the worst case scenario, where the directory tree of the Upperdir and the ephemeral Mountpoint overlap, utilities can often times reference one and the other, causing application errors and even, in the worst case scenarios that I've experienced, system instability and crashing.

Preserved from original post

I would advocate for one of two options:

1. Ideally, an unknown level of effort to fix the stability issues and support the Upperdir and Mountpoint being the same location
2. Barring this type of scenario altogether

While I would obviously prefer number 1 here, I could certainly see the argument that this type of unpredictable behavior may not be easily wrangled, so I would advocate for number 2. Had I received an error in the outset saying Cannot mount to upperdir, Upperdir cannot be the same as mountpoint, or something similar, I probably would have gone with a different solution (albeit, far less elegant, like just simply symlinking, rsync, or something else).

Additionally, there should also probably some mechanism in place to bar the mountpoint from being a lowerdir.

After perusing different places, it seems that while fuse-overlayfs may or may not openly support it, I've found a number of different places where the mountpoint is set to either a lowerdir or the upperdir, which may make barring this behavior problematic for those doing this already. This would definitely break backwards compatibility since this is technically working with some degree of stability for others.

It might make sense to dig into this a bit deeper, instead, to figure out how it might be possible to remedy some of the stability issues.

Thoughts?

All in all, thanks for the awesome work that's already been put into this project. It's been fantastic for most all other uses I've had.

[Swivelgames]

In general, actually, I don't think this really has anything to do with any of the directories being the same, but probably more specifically, the mountpoint being not empty.

I think the upperdir being the same as the mountpoint might have simply uncovered most of the odd behavior.

[winnscode]

I think maybe the application that can't find the file still hold the old fd of the upperdir/mountpoint? IMHO it should be ok for overlayfs (both overlayfs-in-kernel and fuse-overlayfs) to use the upperdir or lowdir or any other non-empty directory as the mountpoint...

Before mounting, we'll get the origin fd of the upperdir when we open the upperdir by path. And after mounting, we would follow/traverse the mount and get the root of overlayfs intead when we open the same path. They are totally different objects in kernel (file/dentry/inode/fs), even though they are opened through the same path/name...

The overlayfs will open directoris of all layers and hold their fd for further access, like writting file to upper-layer, but the user application should drop the origin fd and switch to the new upperdir, then access the newly mounted file system.

[Swivelgames]

I agree, it definitely should be a supported Use Case. If this use case was properly supported, though, it would not necessarily matter if they were holding on to the old fd, given that fuse-overlayfs should be propagating the changes in bother the upperdir and mountdir appropriately. In other words, writes to the mountdir should propagate to the upperdir, and changes to the upperdir should be adequately reconciled to the mountdir. This is a complicated mechanism, but it needs attention in this case.

Regardless, unfortunately my extensive testing has shown that the application(s) are not holding onto the old fd of the upperdir/mountpoint. This can be reproduced by mounting the overlayfs at startup before any of the applications that are using the mount point interact with the file system.

Applications opened after the fact still result in corruption of files. I'm actually still uncovering file corruption that's occurred last month on an actively used machine for various applications and purposes. That's definitely concerning and points to some instability in the way that fuse-overlayfs is currently reconciling changes between the mountdir and upperdir.

[Swivelgames]

@giuseppe Any idea what could be going on here? This is starting to turn into a blocker, and my head is hurting tracking down the source of the issue 😞

[giuseppe]

this use case is not supported.

fuse-overlayfs will be confused to lookup files on its own mountpoint, as it gets into loops when it uses paths.