Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement cache size for CachingFileSystem #1377

Merged
merged 4 commits into from
Sep 27, 2023

Conversation

ianthomas23
Copy link
Collaborator

This PR implements retrieving the cache size of CachingFileSystem. The latest size is stored as a private attribute which is invalidated on any write to the cache and only recalculated if necessary when cache_size() is called. If multiple storage locations are specified it only considers the last one, i.e. the writable one.

This is a necessary precursor to providing a cache size limit (#510).

fsspec/implementations/cached.py Show resolved Hide resolved
fsspec/implementations/cached.py Outdated Show resolved Hide resolved
@ianthomas23
Copy link
Collaborator Author

I've temporarily removed one of the checks for filecache on Windows as when you delete a cached file the parent directory gets larger not smaller 🤷

The remaining test that is failing on Windows is a blockcache one that does not allow deletion of a file that it claims is still in use. I don't yet know if this is something wrong with Windows or fsspec.

@martindurant
Copy link
Member

does not allow deletion of a file that it claims is still in use. I don't yet know if this is something wrong with Windows or fsspec.

Windows does like to keep locks on files...

@ianthomas23
Copy link
Collaborator Author

If I remove the problematic asserts related to file locking for blockcache on Windows then there isn't really anything left worth testing, so I've just skipped that test on Windows. This isn't ideal, but I would rather have the functionality available on posix systems than not at all.

@martindurant
Copy link
Member

This isn't ideal, but I would rather have the functionality available on posix systems than not at all.

Agreed - but I expect it works fine in practice, and at most results in warnings when closing the interpreter. "Sparse" files are not a thing on windows anyway, so you only get part of the benefit of blockcache.

@martindurant martindurant merged commit 89626e8 into fsspec:master Sep 27, 2023
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants