Merge pull request #50 from carbonblack/release-note-and-config-test

Release note and config test

README.md

@@ -1,5 +1,12 @@
 # Carbon Black Cloud Syslog Connector
 
+## Breaking Changes with v1.1.0
+
+* Leef format logs have been rewritten to better utilize common variables and include as much information as possible
+* `cb_defense_syslog.py` has been renamed to `cbc_syslog.py`
+* Example config file and cacert have moved from the `cb-defense-syslog` folder to a `cbc-syslog` folder
+* Example cron job file has been renamed to cbc-syslog and uses the new file and folder locations
+
 The syslog connector lets administrators forward alert notifications and audit logs from their Carbon Black Cloud instance to local, on-premise systems, and:
 
 * Generates pipe-delimited syslog messages with alert metadata identified by the streaming prevention system

changelog.md

@@ -1,77 +1,86 @@
 # Changelog
 All notable changes to this project will be documented in this file.
 
+## Version 1.1.0
+
+**Breaking Changes:**
+* Leef output has been rewritten to better utilize common properties and include as much information as possible
+* Code and files have been refactored and renamed to better align with the product
+
+**Bug fixes:**
+* Config file no longer fails when output_format is leef
+* Added Python 3 support with the addition of the python six package.
+* Tests have been rewritten and additional tests around the config file have been added
+* https_ssl_verify allows for false value
+
 ## Version 1.0.1
 
 ### Bug Fix
 
-Removed the package fcntl and replaced it with the package psutils. This ensures multi-platform functionality for 
+Removed the package fcntl and replaced it with the package psutils. This ensures multi-platform functionality for
 the connector.
 
 ## Version 1.0.0
 
-### New Installation instructions 
+### New Installation instructions
 
 The Syslog Connector will now be moved to a pip install. Please see README.md for more information about the
 installation instructions.  The previous yum installation will be deprecated.
 
 ### Back Up Directory Feature
 
 In the configuration file, a Backup Directory location can now be added. This allows backup files to be stored
-in the case that the Connector fails to send the data to Syslog. 
+in the case that the Connector fails to send the data to Syslog.
 
-Please see the following example: 
+Please see the following example:
 
     back_up_dir = /Users/jdoe/Documents/
-    
+
 > **Note**: These fields are not optional and must be present in the config file.
 
 ### API Key
 
-In the configuration file, a API Key is now available to be added. This will allow Audit Logs to be pulled from each 
-server in the configuration file. 
+In the configuration file, a API Key is now available to be added. This will allow Audit Logs to be pulled from each
+server in the configuration file.
 
 Please see the following example:
 
     [cbdefense1]
     api_connector_id = GO5M953111
     api_key = BYCRM7BRNSH0CXZR5V1Y3111
 
-> **Note**: These fields are not optional and must be present in the config file. If no API Key is needed, please 
+> **Note**: These fields are not optional and must be present in the config file. If no API Key is needed, please
 leave the field blank as shown below:
 
     [cbdefense1]
-    api_connector_id = 
-    api_key = 
+    api_connector_id =
+    api_key =
 
 
 ### Audit Logs
 
-Audit Logs are now available to be pulled from the Syslog Connector. To set up the program to pull Audit Logs, please 
-see the API Key section above. When the Syslog Connector is executing, the program will grab the Audit Logs that have 
-been generated since the last time the Connector was run. The following file formats are compatible with Audit Logs: 
+Audit Logs are now available to be pulled from the Syslog Connector. To set up the program to pull Audit Logs, please
+see the API Key section above. When the Syslog Connector is executing, the program will grab the Audit Logs that have
+been generated since the last time the Connector was run. The following file formats are compatible with Audit Logs:
 CEF,LEEF, JSON
 
-> **Note**: All events types will be pulled from the Syslog Connector. As of now, no additional filtering is 
+> **Note**: All events types will be pulled from the Syslog Connector. As of now, no additional filtering is
 compatible for Audit Logs.
 
 
 ### ThreatHunter
 
-ThreatHunter notifications are now available to be pulled from the Syslog Connector. To set up the Connector to pull 
-ThreatHunter notifications you need to add the API Key as shown below in the configuration file: 
+ThreatHunter notifications are now available to be pulled from the Syslog Connector. To set up the Connector to pull
+ThreatHunter notifications you need to add the API Key as shown below in the configuration file:
 
 
-    cbdefense1]
+    [cbdefense1]
     siem_connector_id = UEUWR4U111
     siem_api_key = XNS5UKWZXZMCC3CYC7DFM111
 
 
-The file formats that are compatible with ThreatHunter Notifications are: LEEF, CEF, JSON. Just like with Audit Logs, the 
-program will grab only the notifications that have been generated since the last time the Connector was run. 
+The file formats that are compatible with ThreatHunter Notifications are: LEEF, CEF, JSON. Just like with Audit Logs, the
+program will grab only the notifications that have been generated since the last time the Connector was run.
 
-> **Note**: All events types will be pulled from the Syslog Connector. As of now, no additional filtering is 
+> **Note**: All events types will be pulled from the Syslog Connector. As of now, no additional filtering is
 compatible for the ThreatHunter Notifications.
-
-
-

src/cbc_syslog/config.py

@@ -1,5 +1,4 @@
 import ast
-import json
 import os
 import sys
 
@@ -80,7 +79,11 @@ def verify_config(config):
 
     output_params['output_type'] = output_type
     output_params['output_format'] = output_format
-    output_params['policy_action_severity'] = config.get('general', 'policy_action_severity', fallback=1)
+
+    try:
+        output_params['policy_action_severity'] = config.get('general', 'policy_action_severity')
+    except Exception:
+        output_params['policy_action_severity'] = 1
 
     if output_type == 'tcp':
         if not config.has_option('general', 'tcp_out'):
@@ -126,7 +129,11 @@ def verify_config(config):
         else:
             output_params['tls_cert'] = config.get('tls', 'cert')
             output_params['tls_key'] = config.get('tls', 'key')
-            output_params['tls_key_password'] = config.get('tls', 'key_password', fallback=None)
+
+            try:
+                output_params['tls_key_password'] = config.get('tls', 'key_password')
+            except Exception:
+                output_params['tls_key_password'] = None
 
         try:
             output_params['tls_verify'] = config.getboolean('tls', 'tls_verify')
@@ -155,16 +162,16 @@ def verify_config(config):
         output_params['http_headers'] = {'content-type': 'application/json'}
         if config.has_option('general', 'http_headers'):
             try:
-                headers = config.get('general', 'http_headers').strip()  # Get the headers from config file
-                headers = ast.literal_eval(headers)                      # Convert the str to a dict
-                output_params['http_headers'] = json.dumps(headers)      # Convert the dict to JSON
+                headers = config.get('general', 'http_headers').strip()    # Get the headers from config file
+                output_params['http_headers'] = ast.literal_eval(headers)  # Convert the str to a dict
             except Exception as e:
                 logger.error(str(e))
                 logger.error("Invalid http_headers: unable to parse JSON")
                 sys.exit(-1)
 
         if config.has_option('general', 'https_ssl_verify'):
-            output_params['https_ssl_verify'] = bool(config.get('general', 'https_ssl_verify'))
+            output_params['https_ssl_verify'] = False if \
+                config.get('general', 'https_ssl_verify') in ['False', 'false', '0'] else True
         else:
             output_params['https_ssl_verify'] = True
 

src/test/fixtures/cef.conf

@@ -0,0 +1,20 @@
+[general]
+
+template = {{source}} {{version}}|{{vendor}}|{{product}}|{{dev_version}}|{{signature}}|{{name}}|{{severity}}|{{extension}}
+
+back_up_dir = /Users/jdoe/Documents/
+
+policy_action_severity = 4
+
+output_format=cef
+
+output_type=udp
+udp_out=0.0.0.0:8886
+
+
+[CarbonBlackCloudServer]
+api_connector_id = RANDOM_ID
+api_key = RANDOM_SECRET
+siem_connector_id = RANDOM_ID
+siem_api_key = RANDOM_SECRET
+server_url = http://0.0.0.0:5001

src/test/fixtures/json.conf

@@ -0,0 +1,16 @@
+[general]
+back_up_dir = /Users/avanbrunt/Desktop/backdir
+policy_action_severity = 4
+output_format=json
+
+output_type=http
+http_out= http://0.0.0.0:5001/http_out
+http_headers= { 'content-type': 'application/json' }
+https_ssl_verify=False
+
+[CarbonBlackCloudServer]
+api_connector_id = RANDOM_ID
+api_key = RANDOM_SECRET
+siem_connector_id = RANDOM_ID
+siem_api_key = RANDOM_SECRET
+server_url = http://0.0.0.0:5001

src/test/fixtures/leef.conf

@@ -0,0 +1,28 @@
+[general]
+
+back_up_dir = /Users/jdoe/Documents/
+
+output_format=leef
+
+output_type=tcp+tls
+tcp_out=0.0.0.0:8888
+
+
+[tls]
+ca_cert = /etc/cb/integrations/cbc-syslog/ca.pem
+cert = /etc/cb/integrations/cbc-syslog/cert.pem
+key = /etc/cb/integrations/cbc-syslog/cert.key
+tls_verify = true
+
+[CarbonBlackCloudServer]
+api_connector_id = RANDOM_ID
+api_key = RANDOM_SECRET
+siem_connector_id = RANDOM_ID
+siem_api_key = RANDOM_SECRET
+server_url = http://0.0.0.0:5001
+
+
+[CarbonBlackCloudServerSecondary]
+api_connector_id = RANDOM_ID
+api_key = RANDOM_SECRET
+server_url = http://0.0.0.0:5001

src/test/test_config.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python
+import logging.handlers
+import unittest
+import os
+
+from cbc_syslog.config import parse_config, verify_config
+
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+
+formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+
+
+class TestConfig(unittest.TestCase):
+
+    def setUp(self):
+        super(TestConfig, self).setUp()
+        self.addTypeEqualityFunc(str, self.assertMultiLineEqual)
+        self.addTypeEqualityFunc(dict, self.assertDictEqual)
+        self.addTypeEqualityFunc(list, self.assertListEqual)
+        self.addTypeEqualityFunc(tuple, self.assertTupleEqual)
+        self.addTypeEqualityFunc(set, self.assertSetEqual)
+        self.addTypeEqualityFunc(frozenset, self.assertSetEqual)
+        self.maxDiff = None
+
+    def test_cef_config(self):
+        config = parse_config(os.path.dirname(__file__) + '/fixtures/cef.conf')
+        output_params, server_list = verify_config(config)
+
+        self.assertEqual(output_params['output_format'], 'cef')
+        self.assertEqual(output_params['output_type'], 'udp')
+        self.assertEqual(output_params['output_host'], '0.0.0.0')
+        self.assertEqual(output_params['output_port'], 8886)
+
+        self.assertEqual(len(server_list), 1)
+
+    def test_leef_config(self):
+        config = parse_config(os.path.dirname(__file__) + '/fixtures/leef.conf')
+        output_params, server_list = verify_config(config)
+
+        self.assertEqual(output_params['output_format'], 'leef')
+        self.assertEqual(output_params['output_type'], 'tcp+tls')
+        self.assertEqual(output_params['output_host'], '0.0.0.0')
+        self.assertEqual(output_params['output_port'], 8888)
+
+        self.assertEqual(output_params['ca_cert'], '/etc/cb/integrations/cbc-syslog/ca.pem')
+        self.assertEqual(output_params['tls_cert'], '/etc/cb/integrations/cbc-syslog/cert.pem')
+        self.assertEqual(output_params['tls_key'], '/etc/cb/integrations/cbc-syslog/cert.key')
+        self.assertEqual(output_params['tls_verify'], True)
+
+        self.assertEqual(len(server_list), 2)
+
+    def test_json_config(self):
+        config = parse_config(os.path.dirname(__file__) + '/fixtures/json.conf')
+        output_params, server_list = verify_config(config)
+
+        self.assertEqual(output_params['output_format'], 'json')
+        self.assertEqual(output_params['output_type'], 'http')
+        self.assertEqual(output_params['output_host'], 'http://0.0.0.0:5001/http_out')
+        self.assertEqual(output_params['output_port'], None)
+        self.assertEqual(output_params['http_headers'], {'content-type': 'application/json'})
+        self.assertEqual(output_params['https_ssl_verify'], False)
+
+        self.assertEqual(len(server_list), 1)
+
+
+if __name__ == '__main__':
+    unittest.main()

src/test/test_syslog.py

@@ -21,10 +21,10 @@
     get_unicode_string = str
 
 
-class TestCbDefenseSyslogConnector(unittest.TestCase):
+class TestCBCSyslogConnector(unittest.TestCase):
 
     def setUp(self):
-        super(TestCbDefenseSyslogConnector, self).setUp()
+        super(TestCBCSyslogConnector, self).setUp()
         self.addTypeEqualityFunc(str, self.assertMultiLineEqual)
         self.addTypeEqualityFunc(dict, self.assertDictEqual)
         self.addTypeEqualityFunc(list, self.assertListEqual)