Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZeroDivisionError when FileReader.get_batch is called without block_size #12

Open
yngvem opened this issue Apr 3, 2018 · 1 comment
Labels

Comments

@yngvem
Copy link

yngvem commented Apr 3, 2018

I get a ZeroDivisionError when I try to read a batch without specifying what the block_size is. My HDF5 file is created with h5py and is a simple table.

Here is my code and the error message:

In [2]: reader = tftables.open_file('/home/yngve/table.h5', batch_size=5)

In [3]: a = reader.get_batch('/train/images', n_procs=3)
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-3-05142b89f50c> in <module>()
----> 1 a = reader.get_batch('/train/images', n_procs=3)

~/anaconda3/envs/tf/lib/python3.6/site-packages/tftables.py in get_batch(self, path, **kw_args)
    264         block_size = queue.block_size
    265         # get an example for finding data types and row sizes.
--> 266         example = self.streamer.get_remainder(path, block_size)
    267         batch_type = example.dtype
    268         inner_shape = example.shape[1:]

~/anaconda3/envs/tf/lib/python3.6/site-packages/multitables.py in get_remainder(self, path, block_size)
    459         :return: A copy of the remainder elements as a numpy array.
    460         """
--> 461         return self.__get_batch(path, length=block_size, last=True)
    462 
    463     class Queue:

~/anaconda3/envs/tf/lib/python3.6/site-packages/multitables.py in __get_batch(self, path, length, last)
    444 
    445         if last:
--> 446             example = h5_node[length*(len(h5_node)//length):].copy()
    447         else:
    448             example = h5_node[:length].copy()

ZeroDivisionError: integer division or modulo by zero

The error is resolved if I specify block_size. The code

In [2]: reader = tftables.open_file('/home/yngve/table.h5', batch_size=5)

In [3]: a = reader.get_batch('/train/images', block_size=5,  n_procs=3)

does not give any errors. I have tested this code with the FIFOQueue and it does indeed give the expected result.

System info:
OS: Ubuntu 16.04
Python version: 3.6
Tensorflow version: 1.5.0
Tftables version: 1.1.2 (latest from pip)

@ghcollin
Copy link
Owner

ghcollin commented Apr 4, 2018

This definitely seems like a bug.

Thanks for the report!

@ghcollin ghcollin added the bug label Apr 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants