feat: add fsspec as required dependency

[lobis]

Notice that this PR merges into main-fsspec not into main. The main-fsspec will exist along main for a while. I will make a PR to merge main-fsspec into main.

[agoose77]

@jpivarski and I discussed this the other day, and a concern that I had regarding making fsspec a required dependency is whether it's available on Emscripten. At one point, I didn't think that this was the case, and was therefore against this idea.

However, I've revisited the question today, and it seems to be fine. I'm not sure whether the package changed, or my memory is failing. As long as we still support an fsspec-less file-reader, then it will be possible to have some kind of integration on emscripten. Perhaps down the road we can look into support for things like multipart which I think will fail trying to use the built-in HTTP packages.

[jpivarski]

That's great to hear. fsspec should be okay on Emscripten because it's just doing pure Python dispatching things in itself; the networking functionality are all in optional dependencies. The aiohttp package (optional dependency) might not work, but that's a secondary question. It wouldn't be strictly required by Uproot the way that fsspec would.

[lobis]

@jpivarski and I discussed this the other day, and a concern that I had regarding making fsspec a required dependency is whether it's available on Emscripten. At one point, I didn't think that this was the case, and was therefore against this idea.

However, I've revisited the question today, and it seems to be fine. I'm not sure whether the package changed, or my memory is failing. As long as we still support an fsspec-less file-reader, then it will be possible to have some kind of integration on emscripten. Perhaps down the road we can look into support for things like multipart which I think will fail trying to use the built-in HTTP packages.

Yes, I think it wouldn't be a problem. I don't know much about the topic but this is how I tested it:

https://pyodide.org/en/stable/console.html

import micropip
await micropip.install("fsspec")
import fsspec
import fsspec.asyn # I thought this would make it crash but it doesn't
fsspec.asyn.get_loop() # this causes an exception as expected since it cannot start a new thread, some sources won't work (but it's expected)

Regarding the http source in the browser, we should be able to use fsspec (fsspec/filesystem_spec#1177). It wouldn't support multipart but it should work. We can always implement this feature in fsspec if we deem it necessary.

I am willing to work on this since but I may need to ask you a few questions @agoose77 since looks like installing awkward is not straightforward in pyodide (and awkward is a strict dependency of uproot).

[jpivarski]

Do you have to await micropip.install("fsspec")?

I just tried installing Awkward in this terminal (it should work), and I found that the latest awkward-cpp is just a bit old.

>>> micropip.list()
Name               | Version | Source
------------------ | ------- | -------
awkward-cpp        | 22      | pyodide

(awkward 2.3.3 was bound to awkward-cpp 22.)

[jpivarski]

I'm in favor of this and the implementation as represented by this PR. Maybe we should wait until the Awkward installation is fixed, though. Awkward 2.5.0 will be ready soon.

[jpivarski]

Oh, nevermind: this is going into the main-fsspec branch, which isn't what anyone would get from a naive installation.

So yes, please go ahead and merge into the main-fsspec branch!

[lobis]

Do you have to await micropip.install("fsspec")?

Yes, sorry, I already updated the comment.

[lobis]

I'm in favor of this and the implementation as represented by this PR. Maybe we should wait until the Awkward installation is fixed, though. Awkward 2.5.0 will be ready soon.

I don't follow. I don't think it's going to be possible to install awkward (and by extension uproot) using micropip because awkward does not provide a pure Python wheel, right? It's my understanding that this is because of the underlaying cpp code. However it's possible to install numpy (which I assume is similar to awkward in it's dependancy to cpp) so you are saying that in the next release of awkward it will be possible to install from micropip?

[agoose77]

I don't follow. I don't think it's going to be possible to install awkward (and by extension uproot) using micropip because awkward does not provide a pure Python wheel, right? It's my understanding that this is because of the underlaying cpp code. However it's possible to install numpy (which I assume is similar to awkward in it's dependancy to cpp) so you are saying that in the next release of awkward it will be possible to install from micropip?

awkward is a pure-Python wheel, and awkward-cpp is built with the pyodide distribution (although its not up-to-date: I'd thought there was a PR bot there, but I was mistaken).

It's already possible to install the latest version of Awkward in a particular version of pyodide; we do this in our documentation such that the branch previews always have the custom (branch) version of Awkward installed in pyodide.