You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Just migrated my AiiDA installation from 1.6 to 2.0. In my old file-based repository I used gzip to mannally compress many text files, which resulted in a large save in space (600 GB -> 80GB). When I migrate to 2.0 I unzip all files and then do the migration.
After the migration I realized that the migrated storage does not actually compress the files storage, despite that the capability been there in disk-objectstore. For migration this is understandable, since compressing everything would be slow. I can get a compressed repository by creating a new Container and import all data from the migrated Container with compression on.
However, looking at the verdi storage maintain code, it seems that compression is also not turned on when packing loose files:
Should there be an option to have this turned on? Or it is some choice by design?
At the moment, I can still do pack_all_loose with compression via disk-objectstore's interface. But this do mean that a normal AiiDA user would not be benefiting from compression?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Just migrated my AiiDA installation from 1.6 to 2.0. In my old file-based repository I used
gzip
to mannally compress many text files, which resulted in a large save in space (600 GB -> 80GB). When I migrate to 2.0 I unzip all files and then do the migration.After the migration I realized that the migrated storage does not actually compress the files storage, despite that the capability been there in
disk-objectstore
. For migration this is understandable, since compressing everything would be slow. I can get a compressed repository by creating a newContainer
and import all data from the migratedContainer
with compression on.However, looking at the
verdi storage maintain
code, it seems that compression is also not turned on when packing loose files:aiida-core/aiida/repository/backend/disk_object_store.py
Lines 178 to 184 in 5c1eb3f
Sinec in
pack_all_loose
,compress
defaults toFalse
.https://github.com/aiidateam/disk-objectstore/blob/16e6ff98c904ef0ec8b03ba3ce6e1eb00204c328/disk_objectstore/container.py#L1305-L1310
Should there be an option to have this turned on? Or it is some choice by design?
At the moment, I can still do
pack_all_loose
with compression viadisk-objectstore
's interface. But this do mean that a normal AiiDA user would not be benefiting from compression?Beta Was this translation helpful? Give feedback.
All reactions