fs.glob("/dir/**/*") is inconsitent with glob.glob

[alanhdu]

If you run the following code:

import fsspec

fs, __ = fsspec.core.url_to_fs("memory://")
fs.touch("/a/1")
fs.touch("/a/2")
fs.touch("/a/b/1")
fs.touch("/a/b/2")

print(fs.glob("/a/**/*"))

you will get different ansewrs -- on fsspec 2023.5, you get ['/a/b/1', '/a/b/2'] but in fsspec 2023.9 you get ['/a/1', '/a/2', '/a/b', '/a/b/1', '/a/b/2'].

I believe the 2023.5 behavior is more consistent with how Python's glob.glob works, but don't have a strong opinion.

I believe this separate from #1380, since this has to do with the behavior of /dir/**/*, not the behavior of **/, but happy to be corrected!

[martindurant]

Does #1382 fix the situation?

[alanhdu]

It does not -- I just checked it out 046edda locally and you get:

In [1]: import fsspec

In [2]: fs, __ = fsspec.core.url_to_fs("memory://")
   ...: fs.touch("/a/1")
   ...: fs.touch("/a/2")
   ...: fs.touch("/a/b/1")
   ...: fs.touch("/a/b/2")
   ...:
   ...: print(fs.glob("/a/**/*"))
['/a/1', '/a/2', '/a/b', '/a/b/1', '/a/b/2']

In [3]: fsspec.__version__
Out[3]: '2021.11.1+324.g046edda'

[mariosasko]

glob.glob should also return ['/a/1', '/a/2', '/a/b', '/a/b/1', '/a/b/2'] with recursive=True. By default, recursive is False, so "/a/**/*" is effectively the same as "/a/*/*". fsspec's glob treats ** as if recursive=True, which is also consistent with Unix globbing.

Switching to the non-recursive mode would be a big breaking change for fsspec. Still, I can add a flag for switching between the modes in the PR, but I don't think it will be used a lot.

[martindurant]

This is the posix behaviour, suggesting fsspec is getting this right

$ mkdir a
$ mkdir a/b
$ touch a/1
$ touch a/2
$ touch a/b/1
$ touch a/b/2
$ echo a/**/*
a/1 a/2 a/b a/b/1 a/b/2

[alanhdu]

Yeah, I think it's reasonable to say that this is expected behavior -- I was just surprised that the behavior changed from 2023.5 to 2023.9. Happy to close this as "working as expected".