error: Not a regular file or symlink: node when adding node container image to the tree

[w4tsn]

Hey there! I thought opening a bug report on this might be the best way to investigate this further.

I'm currently working on committing uncompressed container images into the OSTree in an alternative image store under /usr/containers. I've successfully added the alpine container so far.

When trying this with a more complex container like node-red, the ostree commit command fails with error: Not a regular file or symlink: node. Is this intended behavior and I just don't get it because of ostree details I don't know of? Should this work in theory?

See this issue on the rpm-ostree issue tracker for more details on the context: coreos/rpm-ostree#2675

In order to reprodue you can try this:

# just checkout /usr to a temp, empty sysroot
ostree --repo="$BUILD_REPO" checkout --verbose --subpath=/usr "$(cat "$COMMIT_FILE")" "$WK_DIR"/sysroot/usr

mkdir "$WK_DIR"/sysroot/usr/containers
podman --root "$WK_DIR"/sysroot/usr/containers pull docker.io/library/alpine
podman --root "$WK_DIR"/sysroot/usr/containers pull docker.io/nodered/node-red:1.2.9

# create an orphan commit based on the rpm-ostree compose using the sysroot dir only containing /usr including /usr/containers created previously
# specify the selinux policy necessary so follow up commits won't complain about missing selinux policies
new_commit=$(ostree --repo="$BUILD_REPO" --parent="$(cat "$COMMIT_FILE")" commit --tree=dir="$WK_DIR"/sysroot -s "$COMMIT_SUBJECT" --orphan --selinux-policy=/usr "$WK_DIR"/sysroot/usr)

# Create the commit in the actual branch using both of the previous commits layered over each other
ostree --repo="$BUILD_REPO" commit -b "$OSTREE_REF" -s "$COMMIT_SUBJECT" --tree=ref="$(cat "$COMMIT_FILE")" --tree=ref="$new_commit"

[mike-sul]

Docker/OCI image spec implies that folder/file removal is marked with a whiteout file in an image layer https://github.com/opencontainers/image-spec/blob/master/layer.md#representing-changes. The docker daemon implementation creates a whiteout file as a character device in order to fulfill an overlayfs kernel module requirement https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#whiteouts-and-opaque-directories.

Thus, committing container image layers into an ostree repo becomes not so straightforward as it was noticed, error: Not a regular file or symlink: node is produced if committing a file structure that includes non-regular files.

We tried to overcome this inconvenience by removing all non-regular files and recording them into a regular text file (file path, attributes) before invoking a commit command. Then, on a target/destination side we do an opposite operation, after issuing a checkout command, create non-regular files listed in the record file.
You can find out more details here https://github.com/foundriesio/ci-scripts/blob/master/apps/ostree_store.py#L117 and https://github.com/foundriesio/aktualizr-lite/blob/master/src/compose/composeapptree.cc#L25.

[w4tsn]

So if I understand it correctly, to avoid a custom deploy/checkout mechanism on the client I see following alternatives:

• use a systemd-service as part of the system that recovers this on boot
• commit compressed images, import them at boot (imposes difficulties with deltas I suppose)

Either way we'd have to come up with work-arounds.

I wonder if this is something desired by ostree to support and in scope and what would be necessary to support this.

[wmanley]

There's nothing in the ostree model that would prevent storing device nodes. I guess it's just not implemented. The file header includes a space for rdev even:

ostree/src/libostree/ostree-core-private.h

Lines 43 to 59 in 6d64477

	/*
	* File objects are stored as a stream, with one #GVariant header,
	* followed by content.
	*
	* The file header is of the following form:
	*
	* <BE guint32 containing variant length>
	* u - uid
	* u - gid
	* u - mode
	* u - rdev (must be 0)
	* s - symlink target
	* a(ayay) - xattrs
	*
	* Then the rest of the stream is data.
	*/
	#define _OSTREE_FILE_HEADER_GVARIANT_FORMAT G_VARIANT_TYPE ("(uuuusa(ayay))")

I'd previously attempted to add the ability to check in overlayfs layers and do the composition in ostree, but I hit some issues with xattr permissions: #1651 (comment)

[wmanley]

You'd have to put some thought in how to implement device nodes with bare-user ostree repos. Probably an empty file with the appropriate xattrs set would be a good bet.

[mangelajo]

Hi, I was about to file a new bug/rfe for this but I see this already exists, I had also a smaller reproducer:

rm -rf rootfs test
ostree --repo=./test init --mode=bare-user
mkdir rootfs
mknod -m 0 rootfs/whiteout c 0 0
ostree commit --repo=./test ./rootfs --branch test/stable/x86_64

results:

error: Not a regular file or symlink: whiteout

I'm going to open a formal enhancement request for this and link it, as well as other related issues

[cgwalters]

Also xref ostreedev/ostree-rs-ext#246