Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRAFT: Fix retry #3950

Closed
wants to merge 1 commit into from
Closed

DRAFT: Fix retry #3950

wants to merge 1 commit into from

Conversation

payno
Copy link
Member

@payno payno commented Oct 16, 2023

Looks like that on recent version of h5py we can get the following error from using retry (that used to be handled on the older version - < 3.10):

period = 0.01, spectrum_cu_from_larch = <est.core.types.spectrum.Spectrum object at 0x7fa83e658b10>, tmpdir = local('/tmp/pytest-of-payno/pytest-60/test_get_data_0_01_0')

    @pytest.mark.parametrize("period", [0.01, 0.1, 0.5])
    def test_get_data(period, spectrum_cu_from_larch, tmpdir):
        filename = str(tmpdir / "data.h5")
        scan = "1.1"
        npoints = len(spectrum_cu_from_larch.energy)
        positioners = {"energy": spectrum_cu_from_larch.energy}
        detectors = {"mu": spectrum_cu_from_larch.mu}
        npoints = spectrum_cu_from_larch.energy.size
        blocksize = max(int(npoints / 10), 1)
        est_time = npoints / blocksize * period
        tmax = time.time() + est_time + 5
        print("Estimated scan time", est_time, "s")
    
        urls = [
            f"silx://{filename}::/{scan}/measurement/energy",
            f"silx://{filename}::/{scan}/measurement/mu",
        ]
        with nxwriter_process(filename, scan, positioners, detectors, blocksize, period):
            nprogress = 0
            while nprogress != npoints:
>               data = [get_data(url, retry_timeout=3) for url in urls]

src/est/tests/test_acquisition.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/est/tests/test_acquisition.py:34: in <listcomp>
    data = [get_data(url, retry_timeout=3) for url in urls]
/home/payno/.local/share/virtualenvs/est_venv/lib/python3.11/site-packages/silx/utils/retry.py:169: in wrapper
    return method(*args, **kw)
src/est/io/utils/read.py:57: in get_data
    with silx.io.h5py_utils.File(url.file_path(), "r") as h5:
/home/payno/.local/share/virtualenvs/est_venv/lib/python3.11/site-packages/silx/io/h5py_utils.py:407: in __init__
    super().__init__(filename, mode=mode, swmr=swmr, libver=libver, **kwargs)
/home/payno/.local/share/virtualenvs/est_venv/lib/python3.11/site-packages/h5py/_hl/files.py:562: in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
/home/payno/.local/share/virtualenvs/est_venv/lib/python3.11/site-packages/h5py/_hl/files.py:235: in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
    ???
h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: name = '/tmp/pytest-of-payno/pytest-60/test_get_data_0_01_0/data.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

h5py/h5f.pyx:102: FileNotFoundError

Changelog:
fix retry for h5py version >= 3.10

Related to https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/78

@payno payno reopened this Oct 16, 2023
@payno
Copy link
Member Author

payno commented Oct 16, 2023

For info on my use case the namespace is always None. Which seems unexpected

    for frame in traceback.walk_tb(e.__traceback__):
        for namespace in (frame[0].f_locals, frame[0].f_globals):
            if namespace.get("__package__", None) == "h5py":
                return True

@payno
Copy link
Member Author

payno commented Oct 16, 2023

indeed the troubles is already fixed by 16888f0

@payno payno closed this Oct 16, 2023
@woutdenolf
Copy link
Contributor

16888f0 seems unrelated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants