PIE-67 Support user customizable indexes via macros (#12)

* prep for rebase

* remove autogen readme

* adds support for variable index use by sourcetype

* pie_67_macros Use macros to allow custom index

Before this update all the searches were configured assuming the main
index was the only option.

With this update, the searches will respect user set indexes for each
sourcetype or a single global puppet data index. See the advanced
configuration option in the Readme for more details on configuring it.

This also starts the 1.5.1 release process.

* remove metadata/local.meta from commit

README.md

@@ -29,8 +29,38 @@ For detailed report generation, a feature for Puppet Enterprise Users, there are
 
 ![Report Builder](https://raw.githubusercontent.com/puppetlabs/TA-puppet-report-viewer/master/README/img/report_builder.png)
 
-More information
+Advanced Configuration
 ----------------
+All report views support using custom indexes for storing event data. They accomplish this with a series of advanced search macros. The queries assume each sourcetype can be stored in it's own index (facts, summary reports, detailed reports, bolt events, action events, Puppet Enterprise metrics).
+
+There is one top level macro, `puppet_index` which defaults to "", if you configure the HEC to use a different index and want all Puppet in that index, change that value here to be `index=puppetindexname`.
+
+If you are using [puppetlabs/splunk_hec](https://forge.puppet.com/puppetlabs/splunk_hec/readme) version 0.5.0 or later, you can specify different HEC tokens for Summary Reports, Facts, and Metrics. Then create an index and an associated HEC token associated with those sourcetypes, and configure both the splunk_hec module in Puppet with those new values. Actions, Bolt Events, and Detailed Reports are all submitted via different tools and would need ot be changed according to use a different HEC token. Then the corresponding macro's updated to use those indexes.
+
+For example, if you want most Puppet data to go to one index, but Facts, Metrics, and Detailed Reports to go to their own indexes, one would follow these steps:
+- Create four indexes: puppet_data, puppet_facts_data, puppet_metrics_data, and puppet_detailed_data (or whatever name makes sense), each with their desired timespan, retention, etc.
+- Create four HEC's (example names):
+1. `puppet` with sourcetype of `puppet:summary` and the index `puppet_data`
+2. `puppet_facts` with sourcetype of `puppet:facts` and the index of `puppet_facts_data`
+3. `puppet_metrics` with sourcetype of `puppet:metrics` and the index of `puppet_metrics_data`
+4. `puppet_detailed` with sourcetype of `puppet:detailed` and the index of `puppet_detailed_data`
+- Configure the `splunk_hec` module with the corresponding tokens
+1. `splunk_hec::token` with the value from the `puppet` HEC (since you want all Puppet using splunk_hec plugin to go here, except for facts and metrics)
+2. `splunk_hec::token_facts` with the value from the `puppet_facts` HEC
+3. `splunk_hec::token_metrics` with the value from the `puppet_metrics` HEC
+- Update the Puppet Report Viewer's configuration to use the `puppet_detailed` HEC token, because detailed reports are pulled from Puppet and generated by the alert action in this application
+- Update the advanced search macros to use the new values:
+1. Open Advanced Search under the Settings -> Knowledge menu
+2. Select `Search Macros`
+3. Select `puppet_index` and change the definition to `index=puppet_data`, click save
+4. Select `puppet_facts_index` and change the definition to `index=puppet_facts_data`, click save
+5. Select `puppet_metrics_index` and change the definition to `index=puppet_metrics_data`, click save
+6. Select `puppet_detailed_index` and change the definition to `index=puppet_detailed_data`, click save
+- Reload the main view of the Puppet Report Viewer app, and you should see data, or perform the following search:
+``` 
+`puppet_all_index` sourcetype=puppet:*
+```
+
 
 More information
 ----------------

README/DEVELOPING.md

@@ -10,7 +10,7 @@ In order to load this module properly into the Splunk Add-On builder for develop
 
 ```
 $ git checkout -b 'my working branch'
-$ tar -C .. --exclude=".git" --exclude="tmpdir" -czvf tmpdir/TA-puppet-report-viewer.tar.gz TA-puppet-report-viewer
+$ COPYFILE_DISABLE=1 tar -C .. --exclude=".git" --exclude="local/" --exclude="metadata/local.meta" --exclude="tmpdir" -czvf tmpdir/TA-puppet-report-viewer.tar.gz TA-puppet-report-viewer
 ```
 
 To add your finished work back to the repo:
@@ -22,7 +22,7 @@ To add your finished work back to the repo:
 
 ```
 $ cd tmpdir
-$ tar xzvf TA-puppet-tasks-actionable_2_0_1_export.tgz
+$ tar xzvf TA-puppet-report-viewer_2_0_1_export.tgz
 $ cd ..
-$ rsync -vr tmpdir/TA-puppet-tasks-actionable_2_0_1_export/* ./
+$ rsync -vr tmpdir/TA-puppet-report-viewer_2_0_1_export/* ./
 ```

README/publishing.md

@@ -0,0 +1,13 @@
+Puppet Report Viewer Publishing Guide
+==============
+
+This documents how this plugin is published.
+
+- Finalize testing
+- Bundle this app up using the export options in the developing.md guide
+- Import tar.gz into Splunk AddOn Builder for final validation preflight check
+- Complete and fix validation steps if needed
+- Make needed changes, tag build with final release number and publish a release in GitHub adding notes from releasenotes.md
+- Rename file name to be .spl instead of .tar.gz
+- Import .spl version to Splunk to verify that package installs properly
+- Upload .spl to Splunkbase

README/releasenotes.md

@@ -0,0 +1,16 @@
+Release Notes
+==============
+
+1.5.1:
+New Features:
+- Full dashboard updates
+- Support for Facts sourcetype (puppet:facts), and dashboards to use it
+- Introduces "Report Builder" page to help a user build reports and then craft custom search from the iterface to use for alerts or their own uses
+- Introduces Advanced Search macros to allow for customized indexs without requiring to modify the app. See Advanced Configuration section of the readme
+- Add's sourcetypes of puppet:action, puppet:metrics, for future use
+- Example Alert added, the search to generate a detailed report for any summary report that isn't "unchanged" has been added to the app, but set as disabled
+
+Fixes:
+- Duplicate item entry fixed, sourcetype's are now configured to extract KV from json only once
+- [Updated documentation](https://github.com/puppetlabs/ta-puppet-report-viewer)
+

app.manifest

@@ -5,7 +5,7 @@
     "id": {
       "group": null,
       "name": "TA-puppet-report-viewer",
-      "version": "1.4.0"
+      "version": "1.5.1"
     },
     "author": [
       {

default/app.conf

@@ -7,7 +7,7 @@ build = 3
 
 [launcher]
 author = Puppet, Inc.
-version = 1.4.0
+version = 1.5.1
 
 [ui]
 is_visible = 1

default/data/ui/views/all_reports.xml

@@ -13,7 +13,7 @@
       <fieldForLabel>os.family</fieldForLabel>
       <fieldForValue>os.family</fieldForValue>
       <search>
-        <query>sourcetype="puppet:facts"
+        <query>`puppet_facts_index` sourcetype="puppet:facts"
 | top os.family limit=100</query>
         <earliest>$reportTimeRange.earliest$</earliest>
         <latest>$reportTimeRange.latest$</latest>
@@ -24,7 +24,7 @@
       <fieldForLabel>environment</fieldForLabel>
       <fieldForValue>environment</fieldForValue>
       <search>
-        <query>sourcetype="puppet:facts"
+        <query>`puppet_facts_index` sourcetype="puppet:facts"
 | top environment limit=100</query>
         <earliest>$reportTimeRange.earliest$</earliest>
         <latest>$reportTimeRange.latest$</latest>
@@ -35,7 +35,7 @@
       <fieldForLabel>networking.domain</fieldForLabel>
       <fieldForValue>networking.domain</fieldForValue>
       <search>
-        <query>sourcetype="puppet:facts"
+        <query>`puppet_facts_index` sourcetype="puppet:facts"
 | top networking.domain limit=100</query>
         <earliest>$reportTimeRange.earliest$</earliest>
         <latest>$reportTimeRange.latest$</latest>
@@ -46,7 +46,7 @@
     <panel>
       <table>
         <search>
-          <query>sourcetype="puppet:summary"
+          <query>`puppet_summary_index` sourcetype="puppet:summary"
 | eval "certname"=mvdedup('certname'),"run_time"=mvdedup('metrics.time.total')
 | eval strf_time=strftime(_time, "%Y-%m-%d %T %:z")
 | eval run_time_rnd=round(run_time,1)

default/data/ui/views/bolt.xml

@@ -14,7 +14,7 @@
       <title>All Bolt runs</title>
       <chart>
         <search>
-          <query>sourcetype="puppet:bolt" event_type="bolt_apply"
+          <query>`puppet_bolt_index` sourcetype="puppet:bolt" event_type="bolt_apply"
 | timechart count(_raw)</query>
           <earliest>$boltTimeRange.earliest$</earliest>
           <latest>$boltTimeRange.latest$</latest>
@@ -37,7 +37,7 @@
       <title>Bolt run details (by last reported time)</title>
       <table>
         <search>
-          <query>sourcetype="puppet:bolt" event_type="bolt_apply"
+          <query>`puppet_bolt_index` sourcetype="puppet:bolt" event_type="bolt_apply"
 | eval "certname"=mvdedup('host'), "log"=mvdedup('logs{}.message')
 | eval "strf_time"=strftime(_time, "%Y-%m-%d %T %:z")
 | rename "certname" as "Host"

default/data/ui/views/detailed_reports.xml

@@ -13,7 +13,7 @@
       <fieldForLabel>os.family</fieldForLabel>
       <fieldForValue>os.family</fieldForValue>
       <search>
-        <query>sourcetype="puppet:facts"
+        <query>`puppet_facts_index` sourcetype="puppet:facts"
 | top os.family limit=100</query>
         <earliest>$detailedreportTimeRange.earliest$</earliest>
         <latest>$detailedreportTimeRange.latest$</latest>
@@ -24,7 +24,7 @@
       <fieldForLabel>environment</fieldForLabel>
       <fieldForValue>environment</fieldForValue>
       <search>
-        <query>sourcetype="puppet:facts"
+        <query>`puppet_facts_index` sourcetype="puppet:facts"
 | top environment limit=100</query>
         <earliest>$detailedreportTimeRange.earliest$</earliest>
         <latest>$detailedreportTimeRange.latest$</latest>
@@ -35,7 +35,7 @@
       <fieldForLabel>networking.domain</fieldForLabel>
       <fieldForValue>networking.domain</fieldForValue>
       <search>
-        <query>sourcetype="puppet:facts"
+        <query>`puppet_facts_index` sourcetype="puppet:facts"
 | top networking.domain limit=100</query>
         <earliest>$detailedreportTimeRange.earliest$</earliest>
         <latest>$detailedreportTimeRange.latest$</latest>
@@ -46,7 +46,7 @@
     <panel>
       <table>
         <search>
-          <query>sourcetype="puppet:detailed"
+          <query>`puppet_detailed_index` sourcetype="puppet:detailed"
 | eval "certname"=mvdedup('certname'),"run_time"=mvdedup('metrics.time.total')
 | eval strf_time=strftime(_time, "%Y-%m-%d %T %:z")
 | eval run_time_rnd=round(run_time,1)

default/data/ui/views/details.xml

@@ -14,7 +14,7 @@
       <title>Hosts by OS family</title>
       <chart>
         <search>
-          <query>sourcetype="puppet:detailed" 
+          <query>`puppet_detailed_index` sourcetype="puppet:detailed" 
 | top facts.osfamily limit=100</query>
           <earliest>$detailsTimeRange.earliest$</earliest>
           <latest>$detailsTimeRange.latest$</latest>
@@ -28,7 +28,7 @@
       <title>Free memory by host (average)</title>
       <chart>
         <search>
-          <query>sourcetype="puppet:detailed" 
+          <query>`puppet_detailed_index` sourcetype="puppet:detailed" 
 | chart avg(facts.memoryfree_mb) by host</query>
           <earliest>$detailsTimeRange.earliest$</earliest>
           <latest>$detailsTimeRange.latest$</latest>
@@ -54,7 +54,7 @@
       <title>Classes in use</title>
       <table>
         <search>
-          <query>sourcetype="puppet:detailed"
+          <query>`puppet_detailed_index` sourcetype="puppet:detailed"
 | top limit=100 "logs.data{}.source"
 | search "logs.data{}.source"=Class*
 | rename "logs.data{}.source" as "Class"
@@ -79,7 +79,7 @@
       <title>Facts by host (from detailed reports, last report shown)</title>
       <table>
         <search>
-          <query>sourcetype="puppet:detailed" 
+          <query>`puppet_detailed_index` sourcetype="puppet:detailed" 
 | dedup 1 certname
 | eval "certname"=mvdedup('certname'),"facts.osfamily"=mvdedup('facts.osfamily'),"facts.uptime"=mvdedup('facts.uptime'),"facts.architecture"=mvdedup('facts.architecture'),"facts.architecture"=mvdedup('facts.architecture'),"facts.networking.ip"=mvdedup('facts.networking.ip')
 | eval strf_time=strftime(_time, "%Y-%m-%d %T %:z")

default/data/ui/views/hosts_with_corrective_changes.xml

@@ -13,7 +13,7 @@
     <panel>
       <table>
         <search>
-          <query>sourcetype="puppet:summary"
+          <query>`puppet_summary_index` sourcetype="puppet:summary"
 | search corrective_change=true
 | dedup 1 certname
 | eval "certname"=mvdedup('certname'),"corrective_change"=mvdedup('corrective_change'),"run_time"=mvdedup('metrics.time.total')

default/data/ui/views/hosts_with_failures.xml

@@ -13,7 +13,7 @@
     <panel>
       <table>
         <search>
-          <query>sourcetype="puppet:summary"
+          <query>`puppet_summary_index` sourcetype="puppet:summary"
 | search status=failed 
 | dedup 1 certname
 | eval "certname"=mvdedup('certname'),"run_time"=mvdedup('metrics.time.total')

default/data/ui/views/inventory.xml

@@ -16,7 +16,7 @@
       <fieldForLabel>pe_console</fieldForLabel>
       <fieldForValue>pe_console</fieldForValue>
       <search>
-        <query>sourcetype="puppet:facts" | dedup pe_console | table pe_console</query>
+        <query>`puppet_facts_index` sourcetype="puppet:facts" | dedup pe_console | table pe_console</query>
         <earliest>-24h@h</earliest>
         <latest>now</latest>
       </search>
@@ -29,7 +29,7 @@
       <title>Hosts by OS family</title>
       <chart>
         <search>
-          <query>sourcetype="puppet:facts" $pe_console$ | dedup host | top 10 os.name</query>
+          <query>`puppet_facts_index` sourcetype="puppet:facts" $pe_console$ | dedup host | top 10 os.name</query>
           <earliest>$detailsTimeRange.earliest$</earliest>
           <latest>$detailsTimeRange.latest$</latest>
         </search>
@@ -42,7 +42,7 @@
       <title>Memory usage by percentage</title>
       <table>
         <search>
-          <query>sourcetype="puppet:facts" $pe_console$ | convert rmunit(memory.system.capacity) |dedup host | rename networking.hostname as "Host" | rename memory.system.capacity as "Memory Used" | table "Host" "Memory Used" | sort - "Memory Used"</query>
+          <query>`puppet_facts_index` sourcetype="puppet:facts" $pe_console$ | convert rmunit(memory.system.capacity) |dedup host | rename host as "Host" | rename memory.system.capacity as "Memory Used" | table "Host" "Memory Used" | sort - "Memory Used"</query>
           <earliest>$detailsTimeRange.earliest$</earliest>
           <latest>$detailsTimeRange.latest$</latest>
         </search>
@@ -61,7 +61,7 @@
       <title>System load average</title>
       <table>
         <search>
-          <query>sourcetype="puppet:facts" $pe_console$ | stats avg(load_averages.15m) by networking.hostname | eval avg(load_averages.15m)=round('avg(load_averages.15m)',2) | table networking.hostname avg(load_averages.15m) | rename avg(load_averages.15m) as "Load Average" | rename "networking.hostname" as "Host" | sort - "Load Average"</query>
+          <query>`puppet_facts_index` sourcetype="puppet:facts" $pe_console$ | stats avg(load_averages.15m) by host | eval avg(load_averages.15m)=round('avg(load_averages.15m)',2) | table host avg(load_averages.15m) | rename avg(load_averages.15m) as "Load Average" | rename host as "Host" | sort - "Load Average"</query>
           <earliest>$detailsTimeRange.earliest$</earliest>
           <latest>$detailsTimeRange.latest$</latest>
         </search>
@@ -83,7 +83,7 @@
       <chart depends="$title1$">
         <title>Memory usage on $title1$ over 24 Hours</title>
         <search>
-          <query>sourcetype="puppet:facts" networking.hostname=$title1$ $pe_console$| convert rmunit(memory.system.capacity) | stats avg(memory.system.capacity) by _time | rename "avg(memory.system.capacity)" as "Avg Memory" | predict "Avg Memory"</query>
+          <query>`puppet_facts_index` sourcetype="puppet:facts" host=$title1$ $pe_console$| convert rmunit(memory.system.capacity) | stats avg(memory.system.capacity) by _time | rename "avg(memory.system.capacity)" as "Avg Memory" | predict "Avg Memory"</query>
           <earliest>-24h@h</earliest>
           <latest>now</latest>
         </search>
@@ -98,7 +98,7 @@
       <chart depends="$title2$">
         <title>System load average on $title2$ over 24 Hours</title>
         <search>
-          <query>sourcetype="puppet:facts" networking.hostname=$title2$ $pe_console$ | stats avg(load_averages.15m) by _time | eval avg(load_averages.15m)=round('avg(load_averages.15m)',2) | rename avg(load_averages.15m) as LoadAvg15 | predict LoadAvg15</query>
+          <query>`puppet_facts_index` sourcetype="puppet:facts" host=$title2$ $pe_console$ | stats avg(load_averages.15m) by _time | eval avg(load_averages.15m)=round('avg(load_averages.15m)',2) | rename avg(load_averages.15m) as LoadAvg15 | predict LoadAvg15</query>
           <earliest>-24h@h</earliest>
           <latest>now</latest>
         </search>
@@ -110,31 +110,40 @@
   </row>
   <row>
     <panel>
-      <title>Facts by host (from detailed reports, last report shown)</title>
-      <input type="text" token="value" searchWhenChanged="true">
-        <label>Fact Name</label>
+      <title>Filter hosts by Inventory Data</title>
+      <input type="dropdown" token="factname" searchWhenChanged="true">
+        <label>Inventory Type</label>
+        <fieldForLabel>field</fieldForLabel>
+        <fieldForValue>field</fieldForValue>
+        <search>
+          <query>`puppet_facts_index` sourcetype="puppet:facts" $pe_console$ | fieldsummary | fields field</query>
+          <earliest>$detailsTimeRange.earliest$</earliest>
+          <latest>$detailsTimeRange.latest$</latest>
+        </search>
         <default>os.family</default>
-        <initialValue>*</initialValue>
-        <suffix>="</suffix>
       </input>
-      <input type="text" searchWhenChanged="true" token="facts">
-        <label>Fact Value</label>
+      <input type="dropdown" token="factvalue" searchWhenChanged="true">
+        <label>Inventory Value</label>
         <default>*</default>
-        <prefix>$value$</prefix>
-        <suffix>"</suffix>
-        <initialValue>*</initialValue>
+        <fieldForLabel>$factname$</fieldForLabel>
+        <fieldForValue>$factname$</fieldForValue>
+        <search>
+          <query>`puppet_facts_index` sourcetype="puppet:facts" $pe_console$ | dedup $factname$ | table $factname$</query>
+          <earliest>$detailsTimeRange.earliest$</earliest>
+          <latest>$detailsTimeRange.latest$</latest>
+        </search>
       </input>
       <table>
         <search>
-          <query>sourcetype="puppet:facts" $facts$ $pe_console$
+          <query>`puppet_facts_index` sourcetype="puppet:facts" $factname$=$factvalue$ $pe_console$
 | dedup 1 trusted.certname 
 | eval strf_time=strftime(_time, "%Y-%m-%d %T %:z")
 | rename "trusted.certname" as "Host"
 | rename "os.name" as "OS"
 | rename "os.family" as "OS Family"
 | rename "system_uptime.uptime" as "Uptime"
 | rename "os.architecture" as "Architecture"
-| rename "networking.ip" as "IPv4 address"
+| rename "ipaddress" as "IPv4 address"
 | rename environment as "Environment"
 | rename pe_console as "Puppet Install"
 | rename os.release.full as "OS Version"