How to slim down .scalarCache?

[miku1958]

I tried running this, but the .scalarCache size just went from 24GB to 23.3GB

git maintenance run --task=incremental-repack
git maintenance run --task=pack-refs
git maintenance run --task=loose-objects
git maintenance run --task=commit-graph

And if I delete all the objects, and spend an hour running git reset --hard, it only generates about 8GB of objects.

[dscho]

if I delete all the objects, and spend an hour running git reset --hard, it only generates about 8GB of objects.

Unfortunately, there is no way (that I am aware of) to accomplish this with Git in any faster way.

A hacky way around that would be to perform a fresh partial clone from the current one (using file://<path>, so that really only the needed objects are fetched), and then move the Git objects from the fresh partial clone to the original clone (deleting the original Git objects first), trusting the authoritative remote repository to have the now-missing objects, still.

This concern about the missing objects is with merit, by the way. Imagine that you performed an interactive rebase in the worktree, and then relied on the reflog to refer to the otherwise-unreachable objects. With that "hacky way" I described, those would become missing objects and since they were never pushed, irretrievably lost ones at that.

[miku1958]

Why are the objects that were never pushed not stored in .git/objects?

[dscho]

Why are the objects that were never pushed not stored in .git/objects?

Oh, right, you want to trim the shared alternate object database, not .git/objects!

Typically, it is an unsolvable problem how to determine what objects in a share alternate object database can be deleted: you never know which repositories use this as an alternate object database.

To safely remove objects from such a shared alternate object database, it would have to be known which repositories use that alternate, and then you would have to determine all objects which correspond to checked-out files from all the worktrees of those repositories.

However, in this instance, .scalarCache/<key>/ is an alternate object database that is populated via the scheduled prefetch task. So maybe you could try moving the objects out of the way (not deleting them just yet!), then running git maintenance run prefetch in the worktree to re-populate the Scalar cache, and take it from there? A couple of Git objects might still be needed, and they would be fetched (most likely individually). In the worst case, you would end up spending those 8h again. In the best case, it would address the question how to trim .scalarCache.

[miku1958]

Yes, I am using git maintenance now, and it is very slow.

[miku1958]

The idea is that, for the most part, I only need the objects for my current checkout version, so I want to remove the unneeded objects after updating the code.

[dscho]

@miku1958 do you have a single clone with a single worktree? If so, you may still be able to get what you need by enumerating the object names (SHAs) corresponding to the checked-out files. git ls-files --sparse --stage will give you a start, but you have to exclude sparse directories (lines end in /) and submodules (lines start with 160000). The idea being that you feed these object names to git pack-objects trimmed-pack via stdin, to obtain a pack corresponding to the objects you want to retain.

This is not enough, though: you will need the object name of the current HEAD commit as well as of the involved trees that correspond to (at least partially) checked-out folders.

Here is an attempt to do that:

(
  # the blobs corresponding to the checked-out files
  git ls-files --sparse --stage |
    grep -ve '^160000' -e '/$' |
    cut -c 8-47 &&
  # the commit and root tree
  git rev-parse HEAD HEAD^{tree} &&
  # the trees corresponding to the checked-out files
  git ls-files --sparse |
    sed -n 's/\/[^/][^/]*$//p' |
    uniq |
    sort |
    uniq |
    xargs -d '\n' rev-parse
) |
git pack-objects trimmed-pack

Note: This will not be enough, as even something as simple as git show will then want to fetch objects. Here would be a call to enumerate the blob objects of the parent commit corresponding to the files modified in HEAD:

git diff --no-abbrev --raw HEAD^! |
  sed -ne '/^:160000/d' -e 's/^[^ ]* [^ ]* \([^ ]*\).*$/\1/p'

But even that would not be enough, it would forget the parent commit's object name as well as all of the involved trees. And since trees can be deep, a shell scriptlet to enumerate those would have to look something like this:

git diff --no-abbrev --raw HEAD^! |
  sed -ne '/^:160000/d' -e '/\//{
    s/^[^ ]* [^ ]* [^ ]* [^ ]* .\t\(.*\/\)\?.*$/HEAD^:\1/
    :1
    s/\/[^/]*$//
    p
    /\//b1
  }' |
  uniq |
  sort |
 uniq

At this stage, we're already on a journey to a ridiculously-involved shell script, and I am sure that I forgot something crucial that also needs to go into that trimmed packfile...

[miku1958]

I probably get it, let me try tomorrow to see if this will work. Thanks!

[dscho]

I probably get it, let me try tomorrow to see if this will work. Thanks!

To be honest, if it works, you will most likely want to use a better programming language to implement this ;-)