EventTracer is a thin wrapper to aggregate multiple logging services as a single component with a common interface for utilising the different underlying services.
This gem currently supports only:
- Base logger (payload in JSON format): Can be initialised around the default loggers like those of Rails or Hanami
- Appsignal: Empty wrapper around the custom metric distributions
- increment_counter
- add_distribution_value
- set_gauge
- Datadog: Empty wrapper around the custom metric distributions
- count
- distribution
- set
- gauge
- histogram
- DynamoDB
- Prometheus
- counter
- gauge
Add this line to your application's Gemfile:
gem 'event_tracer'And then execute:
$ bundle
Or install it yourself as:
$ gem install event_tracer
There're 2 sections to using the gem
1. Initialising the individual loggers
Each individual logger should receive the relevant logging services to wrap onto.
base_logger = EventTracer::BaseLogger.new(your_logger)
appsignal_logger = EventTracer::AppsignalLogger.new(Appsignal)
datadog_logger = EventTracer::DatadogLogger.new(
statsd,
allowed_tags: [], # only the tags defined here are allowed to be sent to Datadog
default_tags: {} # the tags and values defined here are always sent to Datadog
)2. Registering the wrapped loggers
Each initialised logger is then registered to EventTracer.
EventTracer.register :base, base_logger
EventTracer.register :appsignal, appsignal_logger
EventTracer.register :datadog, datadog_logger
EventTracer.register :dynamo_db, dynamo_db_loggerAs this is a registry, you can set it up with your own implemented wrapper as long as
it responds to the following LOG_TYPES methods: debug, info, warn, error
Top-level Controls
You can control the loggers to use when sending the event by using the top-level loggers key to specify the logger service(s) to apply to.
| Key | Key type | Required | Values |
|---|---|---|---|
| loggers | Array[Symbol] | N | Array of symbolised logger codes registered for use |
| action | String | Y | Action label to prepend log messages with |
| message | String | Y | This is the basic message to be used for all services |
This accepts an array of the loggers' codes which will be used to select the loggers to send messages for. Invalid/ empty values will be treated as blank and all loggers will be invoked in such a scenario.
1. Base Logger
The base logger only with the following format using a message,
action and all remaining arguments are rendered in a JSON payload
# Sample usage
EventTracer.info action: 'Action', message: 'Message', other_args: 'data'
=> "[Action] message {\"other_args\":\"data\"}"2. Metrics
EventTracer allows sending metrics together with your log. Currently two monitoring services are supported: AppSignal and DataDog.
All metrics are sent in metrics fields, for example:
EventTracer.info(
action: 'Action',
message: 'There is an action',
metrics: {
metric_1: { type: :counter, value: 12 },
metric_2: { type: :gauge, value: 1 },
metric_3: { type: :distribution, value: 10 }
}
)Extra data in the payload can also be filtered to create tags for each metric:
EventTracer.register :appsignal, AppsignalLogger.new(Appsignal, allowed_tags: [:extra_data])Currently, tags apply for all metrics, we don't have support individual tagging yet.
Appsignal >= 2.5 is currently supported for the following metric functions:
| AppSignal function | EventTracer key |
|---|---|
| increment_counter | counter |
| add_distribution_value | distribution |
| set_gauge | gauge |
We can also add tags for metric:
EventTracer.info(
action: 'Action',
message: 'Message',
metrics: [:counter_1],
region: 'eu'
)
# This calls .increment_counter on Appsignal once with additional tag
# counter_1, 1, region: 'eu'Datadog via dogstatsd-ruby (version >= 4.8) is currently supported for the following metric functions:
| DataDog function | EventTracer key |
|---|---|
| increment | counter |
| distribution | distribution |
| gauge | gauge |
| set | set |
| histogram | histogram |
EventTracer.info action: 'Action', message: 'Message',
metrics: {
counter_1: { type: :counter, value: 1 },
counter_2: { type: :counter, value: 2 }
}
# This calls .count on Datadog twice with the 2 sets of arguments
# counter_1, 1
# counter_2, 2Prerequisites:
- Sidekiq
- AWS DynamoDB SDK
Before using this logger, you need to require the logger and define some config:
EventTracer::Config.configure do |config|
config.app_name = 'guardhouse'.freeze # app name that will be sent with each log to DynamoDB
config.dynamo_db_table_name = ENV.fetch('AWS_DYNAMODB_LOGGING_TABLE', 'logs') # send logs to this DynamoDB table
config.dynamo_db_client = Aws::DynamoDB::Client.new # this value is set by default
config.dynamo_db_queue_name = 'low' # defaults to 'low'
end
require "event_tracer/dynamo_db/logger" # NOTE: needs to be required after configuring EventTracerPreparing payload (optional) If you have any pre-processing of the payload to be done, you can supply an instance of a log processor as an argument, e.g.
log_processor = YourLogProcessor.new # defaults to EventTracer::DynamoDB::DefaultProcessor.new
EventTracer.register :dynamodb, EventTracer::DynamoDB::Logger.new(log_processor: log_processor) # note the difference in namespace from the rest of the loggersThis processor needs to respond to .call and accept the same arguments you would normally pass to DynamoDBLogger, namely: log_type, action:, message:, args: and return a Hash
Buffer for network/IO optimization (optional)
For this logger, a thread-safe buffer has been implemented to allow batch sending of logs. To utilise the buffer, define a buffer with optional keyword arguments buffer_size and flush_interval, as such:
buffer = EventTracer::Buffer.new(
buffer_size: ENV.fetch('EVENT_TRACER_BUFFER_SIZE', EventTracer::Buffer::DEFAULT_BUFFER_SIZE).to_i,
flush_interval: ENV.fetch('EVENT_TRACER_FLUSH_INTERVAL', EventTracer::Buffer::DEFAULT_FLUSH_INTERVAL).to_i
)
EventTracer.register :dynamodb, EventTracer::DynamoDBLogger.new(buffer: buffer)buffer_size refers to the number of items that can be stored in the buffer before all items are flushed
flush_interval defines the maximum time between adding the first and penultimate items in the buffer (in seconds). However, note that the buffer is only flushed when the next call is made, so the items could potentially remain in buffer for a very long time if calls are sparse
NOTE: Hanami apps use shotgun gem to reload the app during development. This means that every HTTP request is a new fork with a new (empty) buffer, and the buffered items will simply disappear on the next request. To properly run manual tests in development, comment out shotgun and bundle install again.
If you prefer not to use the buffer, simply initialize without an argument:
EventTracer.register :dynamodb, EventTracer::DynamoDBLogger.new- For Ruby app, Prometheus is supported by Prometheus Client. Checkout its Rack middleware to setup
/metricsendpoint. - For Sidekiq, Sidekiq Prometheus provides the same interface with the above gem. Checkout its Usage to setup for Sidekiq.
In initializers/event_tracer.rb, you can register the logger like this
registry = if Sidekiq.server?
SidekiqPrometheus.registry
else
Prometheus.registry
end
logger = EventTracer::Prometheus.new(registry, allowed_tags: [], default_tags: {})
EventTracer.register :prometheus, loggerThen you can track your metrics using the same interface that EventTracer provides for other loggers.
Notes
-
For multi-processes app, we need to choose the
DirectFileStorefor Prometheus's data store. Read on Data store for more information. -
Prometheus requires every metrics to be pre-registered before we can track them. In
EventTracer, by default it will raise the error for any unregistered metrics. Alternative, we provide araise_if_missingflag to allow just-in-time metric registration.
logger = EventTracer::Prometheus.new(
registry,
allowed_tags: [],
default_tags: {},
raise_if_missing: false # this will register the missing metric instead of raising error
)However, doing so defeats the purpose of being clear on what metrics we want to track and can also result in a performance penalty. To make the metrics registration less tedious, we recommend you to standardize your custom metrics name.
By default, EventTracer passes through errors that are raised by the loggers. However, you can provide a proc or instance of a class to handle the error and payload. You can configure it as such:
EventTracer::Config.configure do |config|
config.error_handler = EventTracerExtension::ErrorHandler.new
end
The error handler will receive two positional arguments, i.e. EventTracerExtension::ErrorHandler.new.call(error, payload):
error - the rescued error
payload - the original arguments or payloads that were logged
In addition, it returns a EventTracer::Result object that logs the success/failure of the outcome of your execution in case you'd like to investigate why your services ain't working. NOTE: if an error is raised, no results are returned unless you handle the error above.
Each log result is mapped to the code of the activated logger.
Sample
result = EventTracer.info action: '123', message: '' # <EventTracer::Result @records={:base=>#<struct EventTracer::LogResult :success?=true, error=nil>}>
result.records[:base].success? => true
result.records[:base].error => nilIn all the generated interface for EventTracer logging could look something like this
EventTracer.info(
loggers: %(base appsignal custom_logging_service datadog),
action: 'NewTransaction',
message: "New transaction created by API",
metrics: {
counter_1: { type: :counter, value: 1 },
distribution_2: { type: :distribution, value: 10 }
},
region: 'eu',
tenant: 'SomeTenant'
)After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.
Bug reports and pull requests are welcome on GitHub at https://github.com/Kaligo/event_tracer.
The gem is available as open source under the terms of the MIT License.