Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fetchToStore(): Avoid duplicate copying if the input is already a store path #10511

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

edolstra
Copy link
Member

@edolstra edolstra commented Apr 15, 2024

Motivation

This is needed for the path:// input scheme (until it returns a FSInputAccessor to the original path, but that's blocked by #10089) and the Mercurial input scheme.

Note: it would be cleaner to have fetchToStore() as an overridable method on InputAccessor (as was originally the case), but the latter got moved to libutil so we can't do that anymore.

Context

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

@github-actions github-actions bot added the fetching Networking with the outside (non-Nix) world, input locking label Apr 15, 2024
Copy link
Member

@Ericson2314 Ericson2314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need isStorePath? I don't think so

// an FSInputAccessor pointing to a store path.
if (path.accessor->isStorePath
&& path.path.isRoot()
&& method == FileIngestionMethod::Recursive
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What goes wrong if we skip this? I am trying to special-case NAR hashing less in Nix currently.

Suggested change
&& method == FileIngestionMethod::Recursive

Copy link
Member

@Ericson2314 Ericson2314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meant that to not be approval

@Ericson2314 Ericson2314 self-assigned this Apr 15, 2024
@edolstra
Copy link
Member Author

Do we need isStorePath? I don't think so

We do need that to ensure it's only used for store paths where we know their ingestion method.

As mentioned above, the architecturally cleaner way is to make fetchToStore() a method in InputAccessor (which is pretty much why that class exists in the first place), but it would require moving InputAccessor back to libfetcher. Which would be a much bigger change.

@Ericson2314
Copy link
Member

I think I rather solve this in a different way:

  1. SourceAccessor has optional content address (on the FSO level, so (FileIngestionMethod, Hash) pair)
  2. fetchToStore can use that to construct a store path and see if it is in the store
  3. only add to store if it isn't.

I am still hoping InputAccessor can go away entirely.

@Ericson2314
Copy link
Member

If you could take a stab at #10467 (review) @edolstra, I would be happy to take on this one per my plan above.

…re path

This is needed for the path:// input scheme (until it returns a
FSInputAccessor to the original path, but that's blocked by NixOS#10089)
and the Mercurial input scheme.
@edolstra edolstra force-pushed the optimize-fetchToStore branch from 893c383 to 8d08846 Compare April 19, 2024 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fetching Networking with the outside (non-Nix) world, input locking
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants