Shared cache between users, cacheonly, cachedir

[j-mracek]

Problems

• DNF has separate metadata for each user. Even availability of outdated cache decrease required downloads - zchunk metadata
• When a cachedir is set, users get a problems with permissions (cache generated by root cannot be refreshed by a user).

Questions

• May a user copy roots metadata cache?
• May a user copy and use roots solv files?
• May root use a user cached metadata? Are we able to verify files?
• May root use generated solv files by user? Are we able to verify files?
• How to prevent a re-download of the same metadata and used already downloaded copy (fedora repository)
• How to use outdated cache from another user to download only diff when I don't have my own copy (zchunk - update repository)

Solutions

1. DNF4 solution - have a fresh metadata for root and with --cacheonly use root cache

[jan-kolarik]

I am adding my thoughts to this.

So in general, it seems to me we want to have a smart logic for copying existing system cache metadata to the user's location rather than relying directly on the system cache. This seems reasonable to me since there may be situations where certain repository metadata are found only in the system cache, while others are exclusively in the user cache.

May a user copy roots metadata cache?

I suggest checking if the user cache is present and up-to-date for a given repository first. If not, then check the system cache. If both caches are empty, we can trigger a remote download if the cacheonly option allows it.

May a user copy and use roots solv files?

I'd also copy the solv files if we choose to copy from the system cache in the above scenario. Is there any potential issue with that?

May root use a user cached metadata? Are we able to verify files?
May root use generated solv files by user? Are we able to verify files?

In general, how would the root know which user metadata to check? Does this imply checking the cache for all available users? I probably wouldn't implement this additional reverse users-to-root cache checking and would wait for potential future use cases...

How to prevent a re-download of the same metadata and used already downloaded copy (fedora repository)

Maybe I don't fully understand this question. Isn't the current implementation already checking if the stored repomd is in sync with the remote?

How to use outdated cache from another user to download only diff when I don't have my own copy (zchunk - update repository)

I'd continue to reuse data in the root-to-user direction and copy outdated metadata to the user's cache if they are missing.

[j-mracek]

How to prevent a re-download of the same metadata and used already downloaded copy (fedora repository)

Use case - root has downloaded metadata for the fedora repository. Then other user request metadata for the same repository - in DNF4 we download metadata again. In DNF5 we might use those metadata.

The second use case: standard user has downloaded metadata for the fedora repository. Then root request metadata for the same repository .

I am not saying that we have to support both use cases, but we can at least consider them.

[evan-goode]

I would rather keep it simple and not share cache data among multiple users. In my day-to-day use, I rarely run DNF as a non-root user, and when I do, I don't mind if it takes some extra time to download metadata. But my opinion here is not very strong.

If DNF 5 used system cache data when run as a non-root user, it would be nice if it could use the system cache files directly rather than duplicating them into the user cache and costing space. If we duplicate the cache files anyway, there is less benefit to sharing the cache compared to re-downloading the files.

I certainly think that DNF 5 should not use any part of a non-root user's cache when running as root, since as @jan-kolarik said it wouldn't even know where to look, but more importantly for security reasons.

[jrohel]

A question to think about security:
DNF5 will download metadata/packages from some mirror repository on the internet. Someone in the middle could have changed something. But we don't mind. It is signed and so we believe.
Can root use the user's cache (metadata, packages) if they are signed with keys that root trusts? Is it more dangerous than downloading data from the internet via http? Can't the user cache be viewed as another mirror repository?

[kontura]

My vote is definitely in favor of sharing the cache. It will speed up dnf5 and lower the requirements on repository infrastructure.

Ideally in both directions but if that is too problematic I think we could at least do the case where unprivileged users uses system cache.

[j-mracek]

I think that repomd.xml contains enough information to validate other files. What we cannot validate is solvx files. In past we trusted solvx file from root for other users with --cacheonly option. But I would not recommend to do it in an opposite direction.

I think that we should optimize cache handling due to

• Download is not for free, but the bill is paid from the server/distribution side (including RedHat for RHEL)
• The current state force users to run all commands as a root (dnf repoquery) - bad pratise
• user experience
• even sharing outdated cache makes a difference for zchunk metadata download

[ppisar]

Reusing root's cache by a normal user is a good thing: Bandwidth and disk space is not for free. It will speed up DNF from user's point of view. Running DNF for read-only operations (e.g. repoquery) under a normal user is in line with best security practices. Reusing root cache is security-safe because a normal user has to trust root for various other reasons. Nevertheless, user's DNF process must be prepared for root removing that files anytime.

Reusing a normal user's cache by a root would be great, but it's dangerous: Root must revalidate origin and integrity of the data before using them. If repomd.xml is not singed, root will have to contact a mirror for the verification (either comparing a hash from a mirror manager or an ETag from an HTTP mirror). While root uses the normal user's cache, DNF will have to prevent a user from modifying it.

Overall maintaining the coherency (that the cache files are not modified while in use by a different user) can be challenging. Simple solution is copy/reflink them to own cache and validating them after that. But that might take disk space.

[jan-kolarik]

In summary, based on the responses here, it appears there is consensus on sharing only the data from the root to the user, including solv files. When the user cache is empty, it makes sense to reuse even outdated root metadata.

Copying metadata seems to be a safe and straightforward solution, as mentioned by @ppisar. Additionally, if we implement the reflink solution where supported by the underlying filesystem, it should address the space concerns raised by @evan-goode.

[jan-kolarik]

Agreed functionality to be implemented against this issue: #1000.