-
Notifications
You must be signed in to change notification settings - Fork 1
Add, Update and Delete Records
A routine runs everyday morning at 3am from symphony that dumps flat marc to the production indexer server. These chages are secure copied from production Symphony to the load directory (/data/symphony_data/daily) on blackcat01qa (the production indexer server) with the filename matching this pattern: daily_addupdate_YYYYMMDD.mrc (note that month and day are zero-padded). This is processed by the indexer at 4:10am. Deletes are set to daily_deletes_YYYYMMDD.txt (again, zero-padded month and day). The process of extracting data happens outside of the concerns of the BlackCat application world, BlackCat expects it though.
Flat marc file extracts on the Symphony side hourly from 7am to midnight everyday and are secure copied from production Symphony to the load directory (/data/symphony_data/hourly) on blackcat01qa with the filename matching this pattern: hourly_addupdate_YYYYMMDD.mrc (note that month and day are zero-padded). Deletes are set to houry_deletes_YYYYMMDD.txt (again, zero-padded month and day). This process is managed outside of the concerns of the BlackCat application world, BlackCat expects it though.
The first incremental hourly run at 7am will collect changes that occurred between midnight and 7am (rather than just the last hour).
This is the whenever gem schedule of processing data from Sirsi.
We have created a search in Splunk that will count the number of times an incremental task has finished. Then, with Dann Bohn's help, we setup an alert in Splunk to tell us when these tasks are not completing as expected.
The rake tasks in psulib_traject take care of indexing and deleting records programmatically via traject. These traject executions are logged with a note that it is coming from the incremental rake task in the traject log. Indexed marc files and to-delete files are deleted upon success, if there's an error it will be logged.
Timing of the imports/deletes is handled via whenever gem settings in the psulib_traject application. Our deploy application has a task to execute the whenever settings as needed.