Preparing for v1.5.0 release

CHANGES.md

@@ -1,5 +1,12 @@
 # bunyan Changelog
 
+## 1.5.0 Templates Release
+
+We now have the ability to specify templates in the log's filename. For full details see the templating section, but briefly: we can template where you put the number when rotating files (this allows you to preserve the extension so that the files open in the correct viewer), or give your log filename a timestamp in your preferred format.
+All previous features are maintained and you should be able to use your existing configuration without change.
+
+Integration testing is still rudimentary in terms of technology, but the coverage has been massively improved, checking that no logs have been re-ordered or lost.
+
 ## 1.4.0
 - New feature to allow the user to specify the order the fields of the log records are written to the log file, this can make manual browsing of the files easier if you put the timestamp and other relevant information first.
 - Some clean ups and refactorings to try to remove some flags.

README.md

@@ -1,5 +1,6 @@
-Bunyan is **a simple and fast JSON logging library** for node.js services:
-Bunyan Rotating File Stream is a rotating file stream component that has some extra features
+Bunyan Rotating File Stream is a stream component for the logging system "node bunyan" that provides rich and flexible control over your log files.
+
+[![Join the chat at https://gitter.im/Rcomian/bunyan-rotating-file-stream](https://badges.gitter.im/Rcomian/bunyan-rotating-file-stream.svg)](https://gitter.im/Rcomian/bunyan-rotating-file-stream?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
 ```js
     var log = bunyan.createLogger({
@@ -20,21 +21,13 @@ Bunyan Rotating File Stream is a rotating file stream component that has some ex
 
 # Recent changes
 
-[![Join the chat at https://gitter.im/Rcomian/bunyan-rotating-file-stream](https://badges.gitter.im/Rcomian/bunyan-rotating-file-stream.svg)](https://gitter.im/Rcomian/bunyan-rotating-file-stream?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
-
-## 1.4
-
-Allow an option to specify the order that fields can be written to the file. Purely for visual purposes.
 
+## 1.5 Templates Release
 
-## 1.3
+We now have the ability to specify templates in the log's filename. For full details see the templating section, but briefly: we can template where you put the number when rotating files (this allows you to preserve the extension so that the files open in the correct viewer), or give your log filename a timestamp in your preferred format.
+All previous features are maintained and you should be able to use your existing configuration without change.
 
-It's a very common programming error to accidentally create 2 rotating file streams against the same log file.
-By default, we detect and disallow this option, throwing an exception as it normally means that a mistake has been made.
-If, however, you really want to be able to do this, we can cache the file streams and return the original one
-each time. To do this add `shared: true` to the list of options when creating each file stream.
-
-Either way, it is now not possible to create 2 rotating file streams against the same file.
+Integration testing is still rudimentary in terms of technology, but the coverage has been massively improved, checking that no logs have been re-ordered or lost.
 
 
 # Current Status
@@ -43,8 +36,13 @@ The basics of the features are there, you should be able to use the
 rotating file logging to limit disk space usage while maximising
 the amount of logs kept.
 
-As ever, there's a lot more features to add as well as a number of
-gaurantees to make.
+There are a few extra features to add to the system, but in general it needs stabilisation, code cleanup and bug fixing.
+
+# Planned Future Features
+
+* Prevent multiple processes logging to the same file
+* Allow multiple processes to safely log to the same file
+* Allow you to say where to put the number in date formatted file names
 
 # Installation
 
@@ -54,6 +52,7 @@ npm install bunyan-rotating-file-stream
 
 # Main Features
 
+- Name log files with templates
 - Rotate to a new log file periodically (can also rotate on startup to clean old log files)
 - Rotate to a new log file once the main log file goes over a certain size
 - Keep a maximum number of archival log files
@@ -85,6 +84,9 @@ The stream supports being both a raw and normal stream modes. Raw streams can be
 under some high-load scenarios but may serialize the json differently to bunyan.
 
 ```js
+    var bunyan = require('bunyan');
+    var RotatingFileStream = require('bunyan-rotating-file-stream');
+
     var log = bunyan.createLogger({
         name: 'foo',
         streams: [{
@@ -94,8 +96,8 @@ under some high-load scenarios but may serialize the json differently to bunyan.
                 period: '1d',          // daily rotation
                 totalFiles: 10,        // keep 10 back copies
                 rotateExisting: true,  // Give ourselves a clean file when we start up, based on period
-                threshold: '10m',       // Rotate log files larger than 10 megabytes
-                totalSize: '20m',       // Don't keep more than 20mb of archived log files
+                threshold: '10m',      // Rotate log files larger than 10 megabytes
+                totalSize: '20m',      // Don't keep more than 20mb of archived log files
                 gzip: true             // Compress the archive log files to save space
             })
         }]
@@ -110,9 +112,6 @@ This will rotate '/var/log/foo.log' every day (at midnight) to:
 /var/log/foo.log.3     # 2 days ago
 ```
 
-*Currently*, there is no support for providing a template for the rotated
-file names.
-
 <table>
 <tr>
 <th>Field</th>
@@ -196,8 +195,120 @@ works. It may break for a time on new versions of node if the internals of seria
 In that case, the replacement code will likely be even slower.</p>
 </td>
 </tr>
+<tr>
+<td>startNewFile</td>
+<td>No</td>
+<td>false</td>
+<td>
+<p>By default the file stream will open the most recent log file it can find and append to it. This flag will
+force the stream to create a new file instead.</p>
+</td>
+</tr>
 </table>
 
+# Templating
+
+## Behaviour without templating
+
+By default, if you just give a normal filename for your log, it will be rotated by appending a number to the end of the file.
+
+For example, if you log to a file `webapi.log`, you'll have the following in your log directory:
+
+```
+    webapi.log  // log file having logs written to it.
+```
+
+When the file needs to be rotated, we'll rename the existing file to `webapi.log.1` and create a new empty file `webapi.log`. Giving you:
+
+```
+    webapi.log   // new log file, logs will be written here.
+    webapi.log.1 // old log file, for archival only.
+```
+
+When we rotate again, we rename the `.1` file to `.2`, the log file to `.1` and create another file, giving us:
+
+```
+    webapi.log   // new log file, logs will be written here.
+    webapi.log.1 // archive of the previously active log file.
+    webapi.log.2 // oldest log file containing oldest entries.
+```
+
+As you can see, the extension of the log files is effectively changed to a number, making it lose its association with any tool that you use to open log files and look at them.
+
+## Rotation number templating [%N]
+
+We can tell the system where to insert the rotation number in the filename. To do this, use the `%N` template parameter (uppercase N). This parameter can only appear in the filename, not in the directories or the extension.
+
+If you use the parameter to make `webapi.%N.log`, after the 2 rotations as above, we end up with:
+
+```
+    webapi.log   // the log file receiving new logs.
+    webapi.1.log // archive of the previously active log file.
+    webapi.2.log // oldest log file containing oldest entries.
+```
+
+Notice that the %N has been stripped out of the name for the active log file - we have no number so we have nothing to put there. If the preceding character is `'.', '_' or '-'`, then that will be stripped out too.
+
+This preserves both the numbering system of the archived files and their extension, so we can open them with the correct tool when we click on them.
+
+## Datetime templating [%Y %m %d %H %M %S]
+
+We can also insert the current time into the log file name. This is the time that the log file was created and should contain logs from that time onwards (notice: this is a hint, not a guarantee: there may be some stragglers from the previous file).
+
+When we do this, we no-longer do any renaming when we rotate to a new log file - we simply create a new file with a new timestamp and use that.
+
+The filename is formatted using <https://www.npmjs.com/package/strftime> and so supports any of the format strings they allow.
+The template parameters must be in the main body of the filename and cannot be in the extension (after the last dot in the name) or in the directories leading up to it.
+
+As an example, we can use the a log name for our system like: `webapi.%d-%b-%y.log`. If we then rotated twice, on separate days, we'd end up with the following files:
+
+```
+    webapi.28-Feb-16.log // Oldest file containing the oldest logs
+    webapi.29-Feb-16.log
+    webapi.01-Mar-16.log // Current file receiving logs
+```
+
+### Filename clashes
+
+The file stream makes no requirement that your filename be particularly unique or sort sensibly in any way. If in the previous example, we rotated twice on the same day, we'd differentiate the files by adding a number like this:
+
+```
+    webapi.28-Feb-16.log   // Original file containing the oldest logs
+    webapi.28-Feb-16.1.log
+    webapi.28-Feb-16.2.log // Current file receiving logs
+```
+
+As with rotating number templating, you can specify where this differentiating number goes using %N.
+
+### Deleting old files
+
+When deleting files based on how many archive files we want to keep, or how much space we want to give to log archives, we delete the oldest files based on the modified time of that file.
+
+When we look for files to delete, we look at all the files in the directory that:
+
+* Match up to the first % sign in the filename template
+* Have the same extension (excluding the .gz if you are compressing old files)
+
+Any files matching those criteria are considered part of the logging system and will be deleted based on the normal deletion rules.
+
+# Creating new files on startup
+
+When we startup we look for any existing log files to append to. If we find a viable file, we simply open it and start appending logs to it. This is the behaviour for all files, whether templated or not.
+
+## Rotating old files [{rotateExisting: true}]
+If an old file should have been rotated but your process wasn't running at the time (maybe you choose to have a new file every day, but the process wasn't running at midnight), we will still append to it. Force a rotation on old files using the option: `{rotateExisting: true}`. This will only rotate files that would have been rotated had we stayed running. This option works for templated or non-templated files.
+
+
+**Note 1: Timestamp in filename**: If your log files have a timestamp in the filename, the timestamp will match when the file would have last rotated, not the current time.
+
+**Note 2: Size thresholds**: If a file would be rotated because of a change in threshold size between runs, then that rotation will happen on the first log write as normal regardless of any flags that are set.
+
+## Force new files [{startNewFile: true}]
+Instead of appending logs to an existing file, you can force a new file to be created using the option: `{startNewFile: true}`.
+If the file date stamp clashes with an existing file, the dotted number notation will be used as normal.
+
+
+# Versioning
 
 The scheme I follow is most succinctly described by the bootstrap guys
 [here](https://github.com/twitter/bootstrap#versioning).

package.json

@@ -1,6 +1,6 @@
 {
   "name": "bunyan-rotating-file-stream",
-  "version": "1.4.0",
+  "version": "1.5.0",
   "description": "a rotating file stream for the bunyan logging system",
   "author": "Jim Tupper <[email protected]> (http://github.com/rcomian)",
   "main": "./index.js",