Home

Merger Wiki

Welcome to the merger wiki!

Feature/Update Wish List

• Make the processing of the Ecal and DQM streams independent from the rest. There is a number of options including:
  • Factor out transfers to the NFS areas at P5 to the transfer hook
• Add support for monitoring of the total number of threads spawned
• Use the thread pool for all the threads spawned, including the subsequent iterations over the input folders
• Improve the latency of the merger after the start up. Possible solutions:
  • Only look at a configurable subset of runs
  • Ignore folders with runs that are complete, e.g. by checking for the presence of EoR
  • Automate cleaning up of closed runs
• Add more info in the lock file to help find the BUs that (a) haven't written their piece yet, (b) are stuck holding a lock or waiting for it for some reason
• Add support for treating and/or logging theorized Lustre issues/hiccups
  • Measure the time required by the various Lustre I/O operations. Log them if they take too long.
  • Spawn threads doing the Lustre I/O, time them, log them, and kill them if they take too long. don't release the lock:
  • Name of the BU holding the lock
  • Date and time of the update
• Remove duplicated time stamps from the logs
• Factor out the parsing of JSON filenames and files into a separate module
• Calculate the check sum during the merge step
• Use inotify to detect directory changes for the minimerger
• Try to avoid unnecessary iterations over directories that have not changed
• Try to efficiently detect macro-merger end-of-run (now done by the merger)
• Make sure the lock-file handling is thread safe
• Set up a centralized logging servier
• pip-ify
• Use messages instead of files to notify the transfer hook about new work
• Try to use Lustre Changelogs instead of polling to discover files on Lustre