Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pin generation on open for version aware file system #601

Open
emfdavid opened this issue Jan 5, 2024 · 4 comments
Open

Pin generation on open for version aware file system #601

emfdavid opened this issue Jan 5, 2024 · 4 comments

Comments

@emfdavid
Copy link

emfdavid commented Jan 5, 2024

Here is an example of the current behavior of version_aware file systems and open files in GCSFS.

I would have expected a version_aware file system to pin the generation of an object while the file is open so that the reads are consistent like they are when the open url specifies the generation.

If this is the desired behavior, I would be happy to take a look and see if I can fix it. Based on your comment in the previous issue I think the python gcs sdk download_as_bytes api should support what we need implementing etag matching similar to what exists in S3 but I haven't compared the details yet.

@martindurant
Copy link
Member

I would have expected a version_aware file system to pin the generation of an object while the file is open so that the reads are consistent like they are when the open url specifies the generation.

This is a reasonable expectation

@martindurant
Copy link
Member

FWIW, I think s3fs does this right even without version-aware by saving the etag.

@emfdavid
Copy link
Author

Is this worth asking Google about?

@martindurant
Copy link
Member

No, I'm pretty sure we can do this by ourselves with the available information

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants