This is an output plugin for Logstash. It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
This guide is for the users just want to download the binary and make the plugin work. For the developer, please refer to the Developer Guide
Create a Sumo Logic free account if you currently don't have one.
Create a HTTP source in your account and get the URL for this source. It should be something like:
https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX
Following this instruction to download and install LogStash. This plugin requires Logstash 2.3 or higher to work.
3. Install latest Logstash Sumo Logic Output plugin from RubyGems
bin/logstash-plugin install logstash-output-sumologicIn the Logstash home, running:
bin/logstash -e "input{stdin{}}output{sumologic{url=>'<URL from step 1>'}}"This will send any input from console to Sumo Logic cloud service.
Set the URL got from step 1 as environment variable:
export sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXXLaunch sample with:
bin/logstash -f samples/log.confThe input from console will be sent to Sumo Logic cloud service as log lines.
Set the URL got from step 1 as environment variable:
export sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXXInstall plugins-filters-metrics plugin
Launch sample with:
bin/logstash -f samples/metrics.confMocked events will be sent to Sumo Logic server as 1 minute and 15 minutes rate metrics.
Logon to Sumo Logic web app and run
- Try it with different input/filter/codec plugins
- Start LogStash as a service/daemon in your production environment
- Report any issue or idea through Git Hub
| Parameter | Type | Required? | Default value | Description |
|---|---|---|---|---|
url |
string | Yes | HTTP Source URL | |
source_category |
string | No | Logstash |
Source category to appear when searching in Sumo Logic by _sourceCategory. Using empty string if want keep source category of the HTTP source. |
source_name |
string | No | logstash-output-sumologic |
Source name to appear when searching in Sumo Logic by _sourceName. Using empty string if want keep source name of the HTTP source. |
source_host |
string | No | machine name | Source host to appear when searching in Sumo Logic by _sourceHost. Using empty string if want keep source host of the HTTP source. |
extra_headers |
hash | No | Extra fields need to be send in HTTP headers. | |
compress |
boolean | No | false |
Enable or disable compression. |
compress_encoding |
string | No | "deflate" |
Encoding method of comressing, can only be "deflate" or "gzip". |
interval |
number | No | 0 |
The maximum time for waiting before sending the message pile, in seconds. |
pile_max |
number | No | 102400 |
The maximum size of message pile, in bytes. |
queue_max |
number | No | 4096 |
The maximum message piles can be hold in memory. |
sender_max |
number | No | 100 |
The maximum HTTP senders working in parallel. |
format |
string | No | "%{@timestamp} %{host} %{message}" |
For log only, the formatter of log lines. Use %{@json} as the placeholder for whole event json. |
json_mapping |
hash | No | Override the structure of {@json} tag with the given key value pairs.For example: json_mapping => { "foo" => "%{@timestamp}" "bar" => "%{message}" }will create messages as: {"foo":"2016-07-27T18:37:59.460Z","bar":"hello world"}{"foo":"2016-07-27T18:38:01.222Z","bar":"bye!"} |
|
metrics |
hash | No | If defined, the event will be sent as metrics. Keys will be the metrics name and values will be the metrics value. | |
metrics_format |
string | No | "cabon2" |
Metrics format, can only be "graphite" or "carbon2". |
metrics_name |
string | No | "*" |
Define the metric name looking, the placeholder "*" will be replaced with the actual metric name. |
intrinsic_tags |
hash | No | For carbon2 format only, send extra intrinsic key-value pairs other than metric (which is the metric name). |
|
meta_tags |
hash | No | For carbon2 format only, send metadata key-value pairs. | |
fields_as_metrics |
boolean | No | false |
If true, all fields in logstash event with number value will be sent as a metrics (with filtering by fields_include and fields_exclude ; the metics parameter is ignored. |
fields_include |
array | No | all fields | Working with fields_as_metrics parameter, only the fields which full name matching these RegEx pattern(s) will be included in metrics. |
fields_exclude |
array | No | none | Working with fields_as_metrics parameter, the fields which full name matching these RegEx pattern(s) will be ignored in metrics. |
sleep_before_requeue |
number | No | 30 |
The message failed to send to server will be retried after (x) seconds. Not retried if negative number set |
stats_category |
string | No | Logstash.stats |
The source category this plugin's stats will be sent with when stats_enabled is true |
stats_enabled |
boolean | No | false |
If true, stats of this plugin will be sent as metrics |
stats_interval |
number | No | 60 |
The stats will be sent every (x) seconds |
This plugin is based on logstash-mixin-http_client thus also supports all HTTP layer parameters like proxy, authentication, timeout etc.
Logstash is using log4j2 framework for logging. Starting with 5.0, each individual plugin can configure the logging strategy. Here is a sample log4j2.properties to print plugin log to console and a rotating file.
The throughput can be tuning with following parameters:
- Messages will be piled before sending if both
intervalandpile_maxare larger than0. (e.g. multiple messages will sent in single HTTP request); The maximum size of pile is defined inpile_maxand if there is no more message comes in, piled message will be sent out everyintervalseconds. A higher number of these parameters normally means more messages will be piled together so overall reduce the overhead in transmission and benefit for compressing efficiency; but it may make a larger latency because messages may be hold in plugin for longer before sending; - Message piles will be cached before sending in a memory queue. The maximum piles can stay in queue is defined with
queue_max. A larger setting may be helpful if input is blocked by the plugin consuming speed, but may also consume more RAM (which can be set in JVM options) - The plugin will use up to
sender_maxHTTP senders in parallel for talking to Sumo Logic server. This number is also limited by the max TCP connections - Depends on the content pattern, adjusting
compress/compress_encodingfor balancing between the CPU consumption and package size
On the other side, this version is marked as thread safe so if necessary, multiple plugins can work in parallel as workers
If your Sumo Logic account supports metrics feature, you can enable the stats monitor of this plugin with configuring stats_enabled to true. For every stats_interval seconds, a batch of metrics data points will be sent to Sumo Logic with source category specified by stats_category parameter:
| Metric | Description |
|---|---|
total_input_events |
Total number of events handled from the plugin startup |
total_input_bytes |
Total bytes of inputs after encoded to payload |
total_metrics_datapoints |
Total metrics data points generated from input |
total_log_lines |
Total log lines generated from input |
total_output_requests |
Total number of HTTP requests sent to Sumo Logic server |
total_output_bytes |
Total bytes of payloads sent to Sumo Logic server |
total_output_bytes_compressed |
Total bytes of payloads sent to Sumo Logic server (after compressing) |
total_response_times |
Total number of responses acknowledged by Sumo Logic server |
total_response_success |
Total number of accepted(200) acknowledged by Sumo Logic server |
NOTE: The data points will consume DPM quota
Sumo Logic only accepts connections from clients using TLS version 1.2 or greater. To utilize the content of this repo, ensure that it's running in an execution environment that is configured to use TLS 1.2 or greater.