AbstractFileSystem is not abstract

[TomNicholas]

I'm trying to learn more about how fsspec works (so I can better understand the IO part of the Pangeo stack), but I'm confused by the structure of fsspec/filesystem_spec.

Despite the name, it seems AbstractFileSystem is not an abstract base class (nor is AbstractBufferedFile) - I'm able to import and instantiate it directly. It contains lots of actual code that does things.

Neither is there a typing.Protocol that subclasses could conform to that could play the same role. (In fact there is no typing at all, which seems like an issue for a project whose purpose is to ensure interface compatibility. Wouldn't it help solve issues like #1411?)

Finally I can't find a written spec document either, at least not one that conforms to RFC 2119 like the Zarr Spec does for example. The fsspec documentation just points me to fsspec/spec.py.

As a new user / potential developer using fsspec, how am I supposed to know which parts of AbstractFileSystem are implementation details that can be overridden, and which are actual requirements? Is there some reason why this lack of a distinct specification doesn't cause problems in practice? Apologies if I have missed something here, I am new to this whole area.

[cisaacstern]

Thanks for raising this @TomNicholas.

I wonder if a stricter typing system would help reduce the need for workarounds like this, where we've found certain types of fsspec.open'ed files themselves have an .open method which needs to be called to get a usable open file?

[martindurant]

It's worth pointing out that fsspec and it's precursor in dask.bytes predates python typing and common use of ABCs. AbstractFileSystem is indeed abstract in the sense that it provides methods to be overwritten (mostly with NotImplemented) and does not provide functionality by itself. It is also a protocol/interface.

The trouble that we are facing, is that not all implementations need to implement all functions. For instance, some filesystem might be read-only or (like HTTP) not support general listing. Further, many things are dynamic and almost all implementations come with many many options.

[martindurant]

There is some work to incrementally add typing: #1396