Development Guide

Welcome to the toolkit development guide!

The Kenna toolkit is a consolidated toolkit of functions for security professionals using Kenna and intended to be run in a docker container in production. During development, however, most developers prefer to set up an environment on their local system.

Getting the code and dependencies

Local Ruby development

1. you should have a ruby setup, preferably set up with rbenv. The current Ruby version used by the connector is 3.2.2.
2. in the toolkit directory, install bundler using the command gem install bundler
3. install the dependencies using bundler bundle install

Using the container

docker build . -t toolkit:latest
docker run -v $(pwd):/opt/app/toolkit --rm -it --entrypoint bash toolkit

First steps

A good place to start with development on the toolkit is by looking at the example task. Two methods matter:

• self.metadata: a method on the Task class, which self-describes how the task can be used, and what options it takes
• run: the method that operates on the data it's passed.

One easy thing to do for new developers is to copy the toolkit/tasks/connectors/_sample_task folder to a new folder and adjust the name of the folder and the task (which should always match) to something like 'hello'. This creates a new task that is automatically available in the toolkit and can be called simply by running (in the root) bundle exec ruby ./toolkit.rb task=hello

Code Organization

We recommend splitting the code into 2 sections: the Task itself and an APIClient. The APIClient is responsible only for interactions with the service (scanner) to obtain the data needed by the task. The APIClient can also format the obtained data to ease the Task process. The Task is responsible for the creation of Kenna objects, upload, and execution of Kenna processes.

The common folder layout is this:

|-- sample_task
|   |-- lib
|   |   |-- sample_api_client.rb
|   |   |-- sample_custom_helpers.rb
|   |-- sample_readme.md
|   |-- sample_task.rb

1. sample_task.rb is expected to contain your Task class which defines the run and self.metadata methods.
2. sample_readme.md is the connector's documentation markdown file.
3. The lib folder is where client classes and helper modules live.

Main run method

We recommend balancing the code in several methods avoiding putting too much in the main run method.

The following is a simplified and fully commented code snippet of the entire process and can be used as guideline:

def run
  initialize_options # Process set options from command line parameters
  client = Client.new(user_id, user_token) # Instantiate the client using options passed as parameters
  page = 1
  loop do
    page_data = client.get_page(page) # Get data from the client in batches

    page_data.each do |issue| # For each resulting issue ...
      asset = extract_asset(issue) # Builds an asset for Kenna
      finding = extract_finding(issue) # Builds the finding (issue) object for Kenna
      definition = extract_definition(issue) # Builds the issue definition (unique definitions)
      create_kdi_asset_finding(asset, finding) # Creates the association in current Kenna batch
      create_kdi_vuln_def(definition) # Creates the definition (deduplicated) in current Kenna batch
    end

    # Below line uploads current batch to Kenna
    kdi_upload(@output_directory, "report_#{page}.json", @kenna_connector_id, @kenna_api_host, @kenna_api_key, @skip_autoclose, @retries, @kdi_version)
    break if page_data.empty? # Stop loop if there is no more data
    page += 1
  end
  # Below line starts the import process in Kenna for all uploaded batches
  kdi_connector_kickoff(@kenna_connector_id, @kenna_api_host, @kenna_api_key)
rescue ApiError => e # Api exception handler for the entire process
  fail_task e.message
end

In the example above the methods extract_asset, extract_finding and extract_definition should return a hash with JSON data in the format specified by the KDI Json Format.

Depending on the final destination for the data upload, you need to use one of create_kdi_asset_finding or create_kdi_asset_vuln helper methods.

Please, refer to the provided sample for specific details on Client implementation, exception handling, and log tracing.

Batching

The connector process runs in a constrained environment and must wisely use the memory and processor resources, making use of batching or pagination techniques.

Usually batch_size and/or page_size are added to the connector parameters with common defaults, normally a number between 100 and 500. In the code, these parameters should be used to split API calls to the source application as well as Kenna, contributing this way to memory usage reduction on both ends.

Helpers

The Toolkit provides several helper methods included in the Kenna::Toolkit::BaseTask class, which is the common ancestor for Task classes.

class BaseTask
      include Kenna::Toolkit::Helpers
      include Kenna::Toolkit::KdiHelpers
...
end

You can find the source code in toolkit/lib/helpers.rb and toolkit/lib/kdi/kdi_helpers.rb.

Most commonly used methods in Helpers are:

• print(message = nil) # Log useful information
• print_good(message = nil) # Log something was successfull
• print_error(message = nil) # Log and error (doesn't break task execution)
• print_debug(message = nil) # Log message only if @options[:debug] == true
• fail_task(message) # Log message as error and terminate execution returning a non zero exit code
• remove_html_tags(string) # Return the argument string whithout any html tags

Most commonly used methods in KdiHelpers are:

• create_kdi_asset(asset_hash) # Create kdi asset with the hash argument and keep it in memory up to next upload
• create_kdi_asset_vuln(asset_hash, vuln_hash) # Create kdi asset with associated vuln
• create_kdi_asset_finding(asset_hash, finding_hash) # Create kdi asset with associated finding
• kdi_upload(output_dir, filename, kenna_connector_id, ...) # Upload the current batch of assets, vulns and findings to Kenna
• kdi_connector_kickoff(kenna_connector_id, kenna_api_host, ...) # Instruct Kenna to process all previous pending uploads

Submitting a Pull Request

Follow instructions in the README.md to submit a pull request. Your PR will get reviewed faster if it has specs and all checks and specs pass!