Pandas complains about an already opened file, even when the file has been closed.

[spott]

import pandas as pd

class Foo():
    def __init__(self):
        self.tt = pd.HDFStore('<filename>.hdf', mode = 'a')
        self.tt.close()
        self.tt = pd.HDFStore('<filename>.hdf', mode='r')

Foo()

This throws a ValueError:

ValueError: The file '<filename>.hdf' is already opened, but in read-only mode.  Please close it before reopening in append mode.

This does not happen when not in a class initializer.

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-16-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.5.1
Cython: None
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.7.0
patsy: None
dateutil: 2.6.1
pytz: 2018.3
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[TomAugspurger]

You code works for me without throwing. Can you post the full traceback?

[TomAugspurger]

I do get a warning when closing that processes about a still-open file though

Closing remaining open files:<filename>.hdf...done

did you run that multiple times? If so, that would make sense I think since you self.tt is open at the end?

[spott]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-12-fdea65d60c59> in <module>()
----> 1 Foo()

<ipython-input-11-2fa5d70ccf4d> in __init__(self)
      3 class Foo():
      4     def __init__(self):
----> 5         self.tt = pd.HDFStore('<filename>.hdf', mode = 'a')
      6         self.tt.close()
      7         self.tt = pd.HDFStore('<filename>.hdf', mode='r')

~/.virtualenvs/idnet_conv_net--ROjEtZd/lib/python3.6/site-packages/pandas/io/pytables.py in __init__(self, path, mode, complevel, complib, fletcher32, **kwargs)
    465         self._fletcher32 = fletcher32
    466         self._filters = None
--> 467         self.open(mode=mode, **kwargs)
    468 
    469     def __fspath__(self):

~/.virtualenvs/idnet_conv_net--ROjEtZd/lib/python3.6/site-packages/pandas/io/pytables.py in open(self, mode, **kwargs)
    603                             hdf_version=tables.get_hdf5_version()))
    604 
--> 605             raise e
    606 
    607         except (Exception) as e:

~/.virtualenvs/idnet_conv_net--ROjEtZd/lib/python3.6/site-packages/pandas/io/pytables.py in open(self, mode, **kwargs)
    578 
    579         try:
--> 580             self._handle = tables.open_file(self._path, self._mode, **kwargs)
    581         except (IOError) as e:  # pragma: no cover
    582             if 'can not be written' in str(e):

~/.virtualenvs/idnet_conv_net--ROjEtZd/lib/python3.6/site-packages/tables/file.py in open_file(filename, mode, title, root_uep, filters, **kwargs)
    310                     "The file '%s' is already opened, but "
    311                     "in read-only mode.  Please close it before "
--> 312                     "reopening in append mode." % filename)
    313             # 'w' means that we want to destroy existing contents
    314             elif mode == 'w':

ValueError: The file '<filename>.hdf' is already opened, but in read-only mode.  Please close it before reopening in append mode.

[TomAugspurger]

Can you post the output of running that as a script? e.g. if the file is

# foo.py
import pandas as pd

class Foo():
    def __init__(self):
        self.tt = pd.HDFStore('<filename>.hdf', mode = 'a')
        self.tt.close()
        self.tt = pd.HDFStore('<filename>.hdf', mode='r')

Foo()

what's the output of python foo.py?

[spott]

It looks like you are right, it is attempting to reopen the same file, and running into issues there.

There isn't a way to close a file when an object is destroyed in python, is there?

[TomAugspurger]

That's up to the file object choosing to implement __del__ or not. IIRC, implementing __del__ to close file handles properly is tricky, especially when the user can just close it manually, or even better use a context manager.

[lazerrobot]

I find this error occurs upon using pd.read_hdf() AFTER having opened/closed the file using pd.HDFStore(). The HDFStore object shows the file is closed, however, once HDFStore opened a file pd.read_hdf() will no longer work.

I do see that once the python terminal is closed, the HDFStore lock is removed from the file. However, I have seen where I've had to close the entire IDE to release the lock, and even then it would not. The lock for HDFStore is disfunctioning IMO.

[saraNgolestani]

I fixed the problem by running the below code before using pd.read_hdf:

    if 'store' in locals() and isinstance(store, pd.HDFStore):
        store.close()