Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change temporary directory every 10.000+ active files #690

Open
2 tasks
SSoelvsten opened this issue Aug 1, 2024 · 1 comment
Open
2 tasks

Change temporary directory every 10.000+ active files #690

SSoelvsten opened this issue Aug 1, 2024 · 1 comment
Labels
🔥 bug Something isn't working ✨ feature New operation or other feature 📁 internal This is where the ✨magic✨happens ✨ optimisation It's all about speed / space

Comments

@SSoelvsten
Copy link
Owner

SSoelvsten commented Aug 1, 2024

On the Grendel cluster, we have created a head-ache for the sysadmins by creating 175.000 concurrent files in the temporary folder (presumably model checking a system of about 20.000+ transitions). We ought to change (sub)directory every 10.000 files.

Tasks

  • Each subdirectory is a mere number 0, 1, 2, ... in the given tmp_path.
  • When creating a file, keep track of whether this is file number 10.000 in the current directory with an (atomic) counter. If so, increment the subdirectory number for the next set of files and reset the counter.
@SSoelvsten SSoelvsten added 🔥 bug Something isn't working ✨ feature New operation or other feature ✨ optimisation It's all about speed / space 📁 internal This is where the ✨magic✨happens 🎓 student programmer Work, work... labels Aug 1, 2024
@SSoelvsten SSoelvsten added this to the v2.1 : Relational Product milestone Aug 1, 2024
@SSoelvsten
Copy link
Owner Author

SSoelvsten commented Aug 1, 2024

We may as well set this up to be the basis for our own file_manager (for #199 , #679 , #678 , and so on). In fact, we may want to flip the problem on its head - leave TPIE to place unnamed temporary files (sorter, priority queue, and arc-based results) in its temporary folder while we create a second temporary folder for all of Adiar's temporary BDDs and ZDDs.

Tasks

  • Allow a file to be temporary even if given path.
  • Obtain (temporary) files from a global file manager.
    • If we want to create fewer temporary directories: keep track of the number of files in each subdirectory (presumably this can be done with a single vector of integers). In the destructor of a file, decrement said counter. Now a new subdirectory is only created if truly needed.

@SSoelvsten SSoelvsten removed the 🎓 student programmer Work, work... label Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🔥 bug Something isn't working ✨ feature New operation or other feature 📁 internal This is where the ✨magic✨happens ✨ optimisation It's all about speed / space
Projects
None yet
Development

No branches or pull requests

1 participant