Merge pull request #30 from carbonblack/feature-cb-30152

CB-30152: Release Yara v2.1.1

.gitignore

@@ -1,3 +1,7 @@
+################################################################################
+## Set of file patterns to prevent being checked into the repo
+################################################################################
+
 *.py[cod]
 
 # C extensions
@@ -29,12 +33,26 @@ nosetests.xml
 # Translations
 *.mo
 
+# Virtual Environments
+.venv
+
 # Mr Developer
 .mr.developer.cfg
 .project
 .pydevproject
 
 # IDE exclusions
 .idea
+*.iml
 
 config.ini
+
+# created local DB
+binary.db
+
+# incase inadvertantly left by unit testing
+conf-testing.conf
+
+# Build files
+RPMS/
+

.travis.yml

@@ -1,6 +1,6 @@
 language: python
 python:
-  - "2.7"
+  - "3.7"
 # command to install dependencies
 install:
     - "pip install nose"

MANIFEST6

@@ -0,0 +1,11 @@
+/usr/share/cb/integrations/cb-yara-connector/yaraconnector
+/usr/share/cb/integrations/cb-yara-connector/yara-logo.png
+%dir /usr/share/cb/integrations/cb-yara-connector
+%dir /var/log/cb/integrations/cb-yara-connector
+/var/log/cb/integrations/cb-yara-connector/yaraconnector.log
+/etc/cb/integrations/cb-yara-connector/yara_rules
+/etc/cb/integrations/cb-yara-connector/yaraconnector.conf.example
+%dir /var/cb/data/cb-yara-connector
+%dir /var/cb/data/cb-yara-connector/feed_db
+/tmp/yaraconnectorceleryworker
+/etc/init.d/cb-yara-connector

MANIFEST7

@@ -0,0 +1,11 @@
+/usr/share/cb/integrations/cb-yara-connector/yaraconnector
+/usr/share/cb/integrations/cb-yara-connector/yara-logo.png
+%dir /usr/share/cb/integrations/cb-yara-connector
+%dir /var/log/cb/integrations/cb-yara-connector
+/var/log/cb/integrations/cb-yara-connector/yaraconnector.log
+/etc/cb/integrations/cb-yara-connector/yara_rules
+/etc/cb/integrations/cb-yara-connector/yaraconnector.conf.example
+%dir /var/cb/data/cb-yara-connector
+%dir /var/cb/data/cb-yara-connector/feed_db
+/tmp/yaraconnectorceleryworker
+/etc/systemd/system/cb-yara-connector.service

README.md

@@ -1,70 +1,201 @@
-# Carbon Black - Yara Connector
-
-[Yara](http://plusvic.github.io/yara/) is the linga franca of malware analysts. 
-With a robust language to define byte strings and clean, well-designed interfaces, 
-many IR and security operations shops keep the results of their analysis in a local
-repository of yara rules.
-
-However, monitoring activity across your network for matches to your yara rules is 
-difficult.  If possible at all, it usually involves infrequent, time-consuming scans.  
-Since Carbon Black collects all executed binaries and has a robust API, it is possible
-to configure your Carbon Black server to act as a "Yara Monitor" and automatically trigger
-notification for any binary executed across your network matching any of your Yara rules.
-
-## Installation Quickstart
-
-As root on your Carbon Black or other RPM based 64-bit Linux distribution server:
+# Installing YARA Agent (CentOS/RHEL 6/7)
+
+[YARA](https://virustotal.github.io/yara/) Integration is made up of two parts -- a master and one or more workers. The master service must be installed on the same system as CB EDR, while workers are usually installed on other systems (but can also be on the master system, if so desired). The YARA connector itself uses [Celery](http://www.celeryproject.org/) to distribute work to and remote (or local) workers - you will need to install and configure a [broker](https://docs.celeryproject.org/en/latest/getting-started/brokers/) (e.g., [Redis](https://redis.io/)) that is accessible to both the task-master and the remote worker instance(s).
+
+The connector reads YARA rules from a configured directory to efficiently scan binaries as they are seen by the CB EDR server. The generated threat information is used to produce an intelligence feed for ingest by the CB EDR Server.
+
+1. Install the CbOpenSource repository if it isn't already present:
+
+    ```
+    cd /etc/yum.repos.d
+    curl -O https://opensource.carbonblack.com/CbOpenSource.repo
+    ``` 
+
+1. Install the RPM:
+    ```
+    yum install python-cb-yara-connector
+    ```
+
+1. Enable the service:
+    
+    1. For Centos/Red Had 6:
+        
+        ```
+        chkconfig cb-yara-connector on
+        ```
+    1. For Centos/Red Had 7:
+        
+        ```
+        systemctl enable cb-yara-connector
+        ```
+
+# Create YARA Connector Config
+
+The installation process creates a sample configuration file: `/etc/cb/integrations/cb-yara-connector/yaraconnector.conf.sample`.  Copy
+this sample template to `/etc/cb/integrations/cb-yara-connector/yaraconnector.conf`,
+which is the filename and location that the connector expects.  You will likely have to edit this
+configuration file on each system (master and workers) to supply any missing
+information:
+* There are two operating modes to support the two roles: `mode=master` and `mode=worker`. Both modes require a broker for Celery communications. Worker systems will need to change the mode to `worker`; 
+
+* Remote worker systems will require the master's URL for `cb_server_url` (local workers need no modification);
+ they also require  the token of a global admin user for `cb_server_token`. 
+* Remote workers will require the URL of the master's Redis server 
+
+The daemon will attempt to load the PostgreSQL credentials from the CB EDR server's `cb.conf` file, 
+if available, falling back to the PostgreSQL connection information in the master's configuration file using the `postgres_xxxx` keys in the config. The REST API location and credentials are specified in the `cb_server_url` and `cb_server_token` keys, respectively. 
+
+```ini
+;
+; Cb Response PostgreSQL Database settings, required for 'master' and 'master+worker' systems
+; The seever will attempt to read from local cb.conf file first and fall back
+; to these settings if it cannot do so.
+;
+postgres_host=127.0.0.1
+postgres_username=cb
+postgres_password=<POSTGRES PASSWORD GOES HERE>
+postgres_db=cb
+postgres_port=5002
 ```
-cd /etc/yum.repos.d
-curl -O https://opensource.carbonblack.com/release/x86_64/CbOpenSource.repo
-yum install python-cb-yara-connector
-```
-
-Once the software is installed via YUM, copy the `/etc/cb/integrations/yara/connector.conf.example` file to 
-`/etc/cb/integrations/yara/connector.conf`. Edit this file and place your Carbon Black API key into the 
-`carbonblack_server_token` variable and your Carbon Black server's base URL into the `carbonblack_server_url` variable.
-Also, point the Yara connector to a directory of yara rule files by editing the `yara_rule_directory` variable. A set
-of example rules are included in the `/usr/share/cb/integrations/yara/example_rules` directory.
 
-To start the service, run `service cb-yara-connector start` as root. Any errors will be logged into `/var/log/cb/integrations/yara/yara.log`.
+```ini
+;
+; Cb EDR server settings, required for 'worker' and 'master+worker' systems
+; For remote workers, the cb_server_url mus be that of the master
+;
+cb_server_url=https://127.0.0.1
+cb_server_token=<API TOKEN GOES HERE>
+```
 
-## Troubleshooting
+You must configure `broker=` which sets the broker and results_backend for Celery. 
+Set this appropriately as per the [Celery documentation](https://docs.celeryproject.org/en/latest/getting-started/brokers/).
 
-If you suspect a problem, please first look at the Yara connector logs found here: `/var/log/cb/integrations/yara/yara.log`
-(There might be multiple files as the logger "rolls over" when the log file hits a certain size).
+```ini
+;
+; URL of the Redis server, defaulting to the local CB EDR server Redis for the master.  If this is a worker
+; system, alter to point to the master system.  If you are using a standalone Redis server, both master and
+; workers must point to the same server.
+;
+broker_url=redis://127.0.0.1
+```
+## Create your YARA rules
 
-If you want to re-run the analysis across your binaries:
+The YARA connector monitors the directory `/etc/cb/integrations/cb-yara-connector/yara_rules` for files (`.yar`) each 
+specifying one or more YARA rule. Your rules must have `meta` section with a 
+`score = [1-10]` tag to appropriately score matching binaries.  This directory is 
+configurable in your configuration file. C-style comments are supported.
 
-1. Stop the service: `service cb-yara-connector stop`
-2. Remove the database file: `rm /usr/share/cb/integrations/yara/db/sqlite.db`
-3. Remove the feed from your Cb server's Threat Intelligence page
-4. Restart the service: `service cb-yara-connector start`
+#### Sample YARA Rule File
+```
+// Sample rule to match binaries over 100kb in size
+
+rule matchover100kb {
+	meta:
+		score = 10
+	condition:
+		filesize > 100KB
+}
+```
 
-## Building yara-python with crypto
+## Controlling the YARA Agent 
+
+#### CentOS / Red Hat 6
+
+| Action | Command |
+| ------ | ------- |
+| Start the service | `service cb-yara-connector start` |
+| Stop the service | `service cb-yara-connector stop` |
+| Display service status | `service cb-yara-connector status` | 
+
+#### CentOS / Red Hat 7
+
+| Action | Command |
+| ------ | ------- |
+| Start the service | `systemctl start cb-yara-connector` |
+| Stop the service | `systemctl stop cb-yara-connector` |
+| Display service status | `systemctl status -l cb-yara-connector` |
+| Displaying verbose logs | `journalctl -u cb-yara-connector` |
+
+# Development Notes	
+
+## Utility Script
+Included with this version is a feature for discretionary use by advanced users and
+should be used with caution.
+
+When `utility_interval` is defined with a value greater than 0, it represents the interval
+in minutes at which the YARA connector will pause its work and execute an external
+shell script.  A sample script, `vacuumscript.sh`  is provided within the `scripts` folder
+of the current YARA connector installation. After execution, the YARA connector continues with
+its work.
+
+> _**NOTE:** As a safety for this feature, if an interval is defined but no script is defined, nothing is done.
+> By default, no script is defined._
+
+```ini
+;
+; The use of the utility script is an ADVANCED FEATURE and should be used with caution!
+;
+; If "utility_interval" is greater than 0 it represents the interval in minutes after which the YARA connector will
+; pause to execute a shell script for general maintenance. This can present risks. Be careful what you allow the
+; script to do, and use this option at your own discretion.
+;
+utility_interval=-1
+utility_script=./scripts/vacuumscript.sh
+```
 
-This is only needed if you are building the connector from scratch.
+## YARA Agent Build Instructions 
+
+The dockerfile in the top-level of the repo contains a CentOS 7 environment for running, building, and testing 
+the connector. 
+
+The provided script `docker-build-rpm.sh` will use docker to build the project, and place the RPM(s) in `${PWD}/RPMS`. 
+
+
+##### Command-line Options
+```text
+usage: yaraconnector [-h] --config-file CONFIG_FILE [--log-file LOG_FILE]
+                     [--output-file OUTPUT_FILE] [--working-dir WORKING_DIR]
+                     [--pid-file PID_FILE] [--daemon]
+                     [--validate-yara-rules] [--debug]
+
+Yara Agent for Yara Connector
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --config-file CONFIG_FILE
+                        location of the config file
+  --log-file LOG_FILE   file location for log output
+  --output-file OUTPUT_FILE
+                        file location for feed file
+  --working-dir WORKING_DIR
+                        working directory
+  --pid-file PID_FILE   pid file location - if not supplied, will not write a
+                        pid file
+  --daemon              run in daemon mode (run as a service)
+  --validate-yara-rules
+                        only validate the yara rules, then exit
+  --debug               enabled debug level logging
+```
+###### --config-file
+Provides the path of the configuration file to be used _**(REQUIRED)**_
 
-1. `git clone --recursive https://github.com/VirusTotal/yara-python`
-2. `cd yara-python/yara`
-3. `./bootstrap.sh`
-3. `./configure --with-crypto`
-4. `make`
-5. `mktmpenv`
-6. `python setup.py build --dynamic-linking`
-7. `python setup.py install`
+###### --log-file
+Provides the path of the YARA log file.  If not supplied, defaults to `local/yara_agent.log`
+within the current YARA package.
 
-## Contacting Carbon Black Developer Relations Support
+###### --output-file
+Provides the path containing the feed description file.  If not supplied, defaults to
+`feed.json` in the same location as the configured `feed_database_dir` folder.
 
-Web: https://developer.carbonblack.com
-E-mail: [email protected]
+###### --validate-yara-rules
+If supplied, YARA rules will be validated and the script will exit.
 
-### Reporting Problems
+---
+# Dev install 
 
-When you contact Bit9 Developer Relations Technical Support with an issue, please provide the following:
+Use Git to retrieve the project, create a new virtual environment using Python 3.6+, and use pip to install the requirements:
 
-* Your name, company name, telephone number, and e-mail address
-* Product name/version, CB Server version, CB Sensor version
-* Hardware configuration of the Carbon Black Server or computer (processor, memory, and RAM) 
-* For documentation issues, specify the version of the manual you are using. 
-* Action causing the problem, error message returned, and event log output (as appropriate) 
-* Problem severity
+```
+git clone https://github.com/carbonblack/cb-yara-connector
+pip3 install -r requirements.txt
+```