Provide dedicated "Cannot load NWB 1.0" Exception/Error instead of an arbitrary crash #1077

yarikoptic · 2019-09-28T16:52:56Z

I think I have ran into the now closed #1051 while trying to read an example file from exp2nwb: https://github.com/NeurodataWithoutBorders/exp2nwb/blob/master/example/Example.nwb?raw=true

$> wget 'https://github.com/NeurodataWithoutBorders/exp2nwb/blob/master/example/Example.nwb?raw=true'
--2019-09-28 12:46:29--  https://github.com/NeurodataWithoutBorders/exp2nwb/blob/master/example/Example.nwb?raw=true
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github.com/NeurodataWithoutBorders/exp2nwb/raw/master/example/Example.nwb [following]
--2019-09-28 12:46:29--  https://github.com/NeurodataWithoutBorders/exp2nwb/raw/master/example/Example.nwb
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/NeurodataWithoutBorders/exp2nwb/master/example/Example.nwb [following]
--2019-09-28 12:46:29--  https://raw.githubusercontent.com/NeurodataWithoutBorders/exp2nwb/master/example/Example.nwb
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.200.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.200.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 180819 (177K) [application/octet-stream]
Saving to: ‘Example.nwb?raw=true’

Example.nwb?raw=true                                     100%[================================================================================================================================>] 176.58K  --.-KB/s    in 0.1s    

2019-09-28 12:46:29 (1.32 MB/s) - ‘Example.nwb?raw=true’ saved [180819/180819]


$> python3 ./print-pynwb.py Example.nwb\?raw=true 
pynwb: <module 'pynwb' from '/usr/lib/python3/dist-packages/pynwb/__init__.py'> pynwb.__version__: 1.0.3
/usr/lib/python3/dist-packages/hdmf/backends/hdf5/h5tools.py:96: UserWarning: No cached namespaces found in Example.nwb?raw=true
  warnings.warn(msg)
Traceback (most recent call last):
  File "./print-pynwb.py", line 9, in <module>
    nwbfile = reader.read()
  File "/usr/lib/python3/dist-packages/hdmf/backends/hdf5/h5tools.py", line 246, in read
    return call_docval_func(super(HDF5IO, self).read, kwargs)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 325, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/backends/io.py", line 35, in read
    container = self.__manager.construct(f_builder)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/build/map.py", line 207, in construct
    result = self.__type_map.construct(builder, self, None)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/build/map.py", line 1668, in construct
    attr_map = self.get_map(builder)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/build/map.py", line 1595, in get_map
    container_cls = self.get_cls(obj)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/build/map.py", line 1522, in get_cls
    raise ValueError("No data_type found for builder %s" % builder.path)
ValueError: No data_type found for builder root

If that is the best way to determine that the file is NWB 1.0 not 2.0, please consider catching it around h5tools.py", line 246, in read and reraising a dedicated/descriptive exception. If there is a better way (I hope) then please add it that sensing even before even trying to load an incompatible .nwb file.

The text was updated successfully, but these errors were encountered:

yarikoptic · 2019-10-07T21:45:49Z

One more example where I was told that file is probably NWB 1.0

(git)smaug:/mnt/btrfs/datasets/datalad/crawl/crcns/ssc-7[master]git
$> dandi ls data/L4E_whole_cell/Exp_2015-09-05_001_0001-0162.nwb
Traceback (most recent call last):
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/bin/dandi", line 11, in <module>
    load_entry_point('dandi', 'console_scripts', 'dandi')()
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/yoh/proj/dandi/dandi-cli/dandi/cli/command.py", line 169, in ls
    rec.update(get_metadata_pyout(path))
  File "/home/yoh/proj/dandi/dandi-cli/dandi/cli/command.py", line 61, in get_metadata_pyout
    meta = get_metadata(path)
  File "/home/yoh/proj/dandi/dandi-cli/dandi/pynwb_utils.py", line 30, in get_metadata
    nwb = io.read()
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 293, in read
    return call_docval_func(super(HDF5IO, self).read, kwargs)
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 327, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 438, in func_call
    return func(self, **parsed['args'])
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/io.py", line 31, in read
    f_builder = self.read_builder()
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 438, in func_call
    return func(self, **parsed['args'])
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 308, in read_builder
    f_builder = self.__read_group(self.__file, ROOT_NAME, ignore=ignore)
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 385, in __read_group
    builder = read_method(sub_h5obj)
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 385, in __read_group
    builder = read_method(sub_h5obj)
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 385, in __read_group
    builder = read_method(sub_h5obj)
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 393, in __read_group
    ret = GroupBuilder(name, **kwargs)
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 438, in func_call
    return func(self, **parsed['args'])
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/build/builders.py", line 179, in __init__
    self.set_dataset(dataset)
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 438, in func_call
    return func(self, **parsed['args'])
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/build/builders.py", line 289, in set_dataset
    self.__set_builder(builder, GroupBuilder.__dataset)
  File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/build/builders.py", line 259, in __set_builder
    (name, self.obj_type[name], self.name, obj_type))
KeyError: "'electrode_name' already exists as attributes in membrane_potential, cannot set as datasets"

but the exception (`KeyError: "'electrode_name' already exists as attributes in membrane_potential...`) had no hint on that

yarikoptic · 2019-10-07T22:10:18Z

aha -- nwb-schema talks about /specifications/<namespace-name>/<version>/...,

and my sample NWB 2.0 file does carry a core/2.0.1 group

$> h5ls -S -r mouse1_fni16_150818_001_ch2-PnevPanResults-170808-180842.nwb/specifications/    
/core                    Group
/core/2.0.2              Group
/core/2.0.2/namespace    Dataset {SCALAR}
/core/2.0.2/nwb.base     Dataset {SCALAR}
/core/2.0.2/nwb.behavior Dataset {SCALAR}
/core/2.0.2/nwb.ecephys  Dataset {SCALAR}
/core/2.0.2/nwb.epoch    Dataset {SCALAR}
/core/2.0.2/nwb.file     Dataset {SCALAR}
/core/2.0.2/nwb.icephys  Dataset {SCALAR}
/core/2.0.2/nwb.image    Dataset {SCALAR}
/core/2.0.2/nwb.misc     Dataset {SCALAR}
/core/2.0.2/nwb.ogen     Dataset {SCALAR}
/core/2.0.2/nwb.ophys    Dataset {SCALAR}
/core/2.0.2/nwb.retinotopy Dataset {SCALAR}

while a sample file in ssc-7 dataset has none:

$> h5ls -S -r data/L4E_whole_cell/Exp_2015-09-05_001_0001-0162.nwb/specifications/                                                             
specifications/**NOT FOUND** %

from glancing over that docs/storage/source/storage_hdf5.rst it is not 100% clear to me if that is a feature of NWB:N specifically or NWB in general, and when was it introduced?

Also the https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/README.rst starts with "It provides a high-level API for efficiently working with Neurodata stored in the NWB format." - it seems not to limit to version of NWB 2.0 or above. If NWB prior 2.0 are not supposed to be supported, it should be clarified in the README.rst IMHO.

Unfortunately I am still learning NWB ecosystem so cannot really propose any definitive solution/PR here without guidance ;)

bendichter · 2019-10-07T22:17:35Z

@yarikoptic can we take this over to slack? I'm happy to help you sort these issues out and open this file in pynwb. It looks like this file is in 2.0 after all, so not relevant to this issue.

yarikoptic · 2019-10-07T23:15:13Z

@yarikoptic can we take this over to slack? I'm happy to help you sort these issues out and open this file in pynwb. It looks like this file is in 2.0 after all, so not relevant to this issue.

and we did (in jitsi), summary: me

being pointed to an obvious nwb_version field at the top level (so it could be used to decide to throw an informative exception, and @bendichter promising to propose a PR for this, while I add it to extracted metadata on dandi-cli side)
clarifying that ssc-7 dataset is coming from CRCNS portal. DataLad "copy" is available from http://datasets.datalad.org/?dir=/crcns/ssc-7 (but you would need to have crcns credentials to fetch actual data)

yarikoptic added the category: bug errors in the code or code behavior label Sep 28, 2019

rly mentioned this issue Sep 29, 2019

Throw specific exception when no data type specified for root group hdmf-dev/hdmf#157

Closed

yarikoptic mentioned this issue Oct 7, 2019

should we list found (NWB:N?) specification versions? dandi/dandi-cli#13

Closed

yarikoptic mentioned this issue Oct 8, 2019

nwb_version is neither populated, nor validation causes any error #1085

Closed

bendichter linked a pull request Oct 8, 2019 that will close this issue

Enh/detect nwb1 #1086

Open

5 tasks

NilsNyberg mentioned this issue Dec 4, 2020

"Permission denied" when using se.NwbRecordingExtractor SpikeInterface/spikeinterface#85

Closed

legouee mentioned this issue Sep 7, 2022

Error in reading NWB files NeuralEnsemble/python-neo#1131

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide dedicated "Cannot load NWB 1.0" Exception/Error instead of an arbitrary crash #1077

Provide dedicated "Cannot load NWB 1.0" Exception/Error instead of an arbitrary crash #1077

yarikoptic commented Sep 28, 2019

yarikoptic commented Oct 7, 2019

yarikoptic commented Oct 7, 2019

bendichter commented Oct 7, 2019

yarikoptic commented Oct 7, 2019

Provide dedicated "Cannot load NWB 1.0" Exception/Error instead of an arbitrary crash #1077

Provide dedicated "Cannot load NWB 1.0" Exception/Error instead of an arbitrary crash #1077

Comments

yarikoptic commented Sep 28, 2019

yarikoptic commented Oct 7, 2019

yarikoptic commented Oct 7, 2019

bendichter commented Oct 7, 2019

yarikoptic commented Oct 7, 2019