-
Notifications
You must be signed in to change notification settings - Fork 2
Home
janveverka edited this page Nov 12, 2014
·
9 revisions
Welcome to the merger wiki!
- Make the processing of the Ecal and DQM streams independent from the rest. There is a number of options including:
- Factor out transfers to the NFS areas at P5 to the transfer hook
- Add support for monitoring of the total number of threads spawned
- Use the thread pool for all the threads spawned, including the subsequent iterations over the input folders
- Improve the latency of the merger after the start up. Possible solutions:
- Only look at a configurable subset of runs
- Ignore folders with runs that are complete, e.g. by checking for the presence of EoR
- Automate cleaning up of closed runs
- Add support for treating and/or logging theorized Lustre issues/hiccups
- Measure the time the required by the various Lustre I/O operations. Log them if they take too long.
- Spawn threads doing the Lustre I/O, time them, log them, and kill them if they take too long.
- Add more info in the lock file to help find the BUs that (a) haven't written their piece yet, (b) don't release the lock:
- Name of the BU holding the lock
- Date and time of the update
- Remove duplicated time stamps from the logs
- Factor out the parsing of JSON filenames and files into a separate module