Skip to content

Commit

Permalink
Merge pull request #3 from bdurand/read_encrypted_options_from_job
Browse files Browse the repository at this point in the history
Read encrypted options from job
  • Loading branch information
bdurand authored Jun 13, 2020
2 parents 95f655a + 9b78a1e commit fbc49e6
Show file tree
Hide file tree
Showing 11 changed files with 441 additions and 172 deletions.
12 changes: 11 additions & 1 deletion CHANGE_LOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,14 @@
# Change log
# Change Log

## 1.1.0

* Use `to_json` if it is defined when serializing encrypted args to JSON.
* Add client middleware to the server default configuration. This ensures that arguments will be encrypted if a worker enqueues a job with encrypted arguments.
* Client middleware now reads sidekiq options from the job hash instead of from the worker class so that the list of encrypted arguments is always in sync on the job payload.
* Don't blow up if class name that is not defined is passed to client middleware.
* Added additional option to specify encrypted args with array of argument indexes.
* Deprecated setting encrypted args as hash or array of booleans.
* Client middleware is prepended while server middleware is appended.

## 1.0.2

Expand Down
8 changes: 4 additions & 4 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
PATH
remote: .
specs:
sidekiq-encrypted_args (1.0.2)
sidekiq-encrypted_args (1.1.0)
secret_keys
sidekiq (>= 4.0)

GEM
remote: https://rubygems.org/
specs:
appraisal (2.2.0)
appraisal (2.3.0)
bundler
rake
thor (>= 0.14.0)
ast (2.4.0)
ast (2.4.1)
climate_control (0.2.0)
connection_pool (2.2.3)
diff-lcs (1.3)
Expand All @@ -24,7 +24,7 @@ GEM
rack
rainbow (3.0.0)
rake (13.0.1)
redis (4.1.4)
redis (4.2.1)
rexml (3.2.4)
rspec (3.9.0)
rspec-core (~> 3.9.0)
Expand Down
50 changes: 26 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This can be an even bigger issue if you use scheduled jobs since sensitive data

## Solution

This gem adds Sidekiq middleware that allows you to specify job arguments for your workers that should be encrypted in Redis. You do this by adding `encrypted_args` to the `sidekiq_options` in the worker. Jobs for these workers will have their arguments encrypted in Redis and decrypted when passed to `perform` method.
This gem adds Sidekiq middleware that allows you to specify job arguments for your workers that should be encrypted in Redis. You do this by adding `encrypted_args` to the `sidekiq_options` in the worker. Jobs for these workers will have their arguments encrypted in Redis and decrypted when passed to the `perform` method.

To use the gem, you will need to specify a secret that will be used to encrypt the arguments as well as add the middleware to your Sidekiq client and server middleware stacks. You can set that up by adding this to the end of your Sidekiq initialization:

Expand All @@ -24,21 +24,29 @@ Sidekiq::EncryptedArgs.configure!(secret: "YourSecretKey")

If the secret is not set, the value of the `SIDEKIQ_ENCRYPTED_ARGS_SECRET` environment variable will be used as the secret. If this variable is not set, job arguments will not be encrypted.

The call to `Sidekiq::EncryptedArgs.configure!` will append the encryption middleware to the end of the client and server middleware chains. You add the middlewares manually if you need more control over where they appear in the stacks.
The call to `Sidekiq::EncryptedArgs.configure!` will **prepend** the client encryption middleware and **append** server decryption middleware. By doing this, any other middleware you register will only receive the encrypted parameters (e.g. logging middleware will receive the encrypted parameters).

You can add the middleware manually if you need more control over where they appear in the stacks.

```ruby
Sidekiq::EncryptedArgs.secret = "YourSecretKey"

Sidekiq.configure_client do |config|
config.client_middleware do |chain|
chain.add Sidekiq::EncryptedArgs::ClientMiddleware
chain.prepend Sidekiq::EncryptedArgs::ClientMiddleware
end
end

Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add Sidekiq::EncryptedArgs::ServerMiddleware
end

# register client middleware on the server so that starting jobs in a Sidekiq::Worker also get encrypted args
# https://github.com/mperham/sidekiq/wiki/Middleware#client-middleware-registered-in-both-places
config.client_middleware do |chain|
chain.prepend Sidekiq::EncryptedArgs::ClientMiddleware
end
end
```

Expand All @@ -59,30 +67,24 @@ class SecretWorker
end
```

You can also encrypt just specific arguments with a hash or an array. This can be useful to preserve visibility into non-sensitive arguments that might be useful for troubleshooting or other reasons. All of these examples will encrypt just the second argument to the `perform` method.

```ruby
# Pass in a list of argument names that should be encrypted
sidekiq_options encrypted_args: [:arg_2]

def perform(arg_1, arg_2, arg_3)
end
```
You can also choose to only encrypt specific arguments with an array of either argument names (symbols or strings) or indexes. This is useful to preserve visibility into non-sensitive arguments for troubleshooting or other reasons. Both of these examples encrypt just the second argument to the `perform` method.

```ruby
# Pass in a hash with values indicating which arguments should be encrypted
sidekiq_options encrypted_args: { arg_2: true, arg_1: false }
# Pass in a list of argument names that should be encrypted
sidekiq_options encrypted_args: [:arg_2]
# or
sidekiq_options encrypted_args: ["arg_2"]

def perform(arg_1, arg_2, arg_3)
end
def perform(arg_1, arg_2, arg_3)
end
```

```ruby
# Pass in an array of boolean values indicating which argument positions should be encrypted
sidekiq_options encrypted_args: [false, true]
# Pass in an array of integers indicating which argument positions should be encrypted
sidekiq_options encrypted_args: [1]

def perform(arg_1, arg_2, arg_3)
end
def perform(arg_1, arg_2, arg_3)
end
```

You don't need to change anything else about your workers. All of the arguments passed to the `perform` method will already be unencrypted when the method is called.
Expand All @@ -92,15 +94,15 @@ You don't need to change anything else about your workers. All of the arguments
If you need to roll your secret, you can simply provide an array when setting the secret.

```ruby
Sidekiq::EncryptedArgs.secret = ["CurrentSecret", "OldSecret"]
Sidekiq::EncryptedArgs.secret = ["CurrentSecret", "OldSecret", "EvenOlderSecret"]
```

The left most key will be considered the current key and will be used for encrypting arguments. However, all of the keys will be tried when decrypting. This allows you to switch you secret keys without breaking jobs already enqueued in Redis.
The first (left most) key will be considered the current key, and is used for encrypting arguments. When decrypting, we iterate over the secrets list until we find the correct one. This allows you to switch you secret keys without breaking jobs already enqueued in Redis.

If you are using the `SIDEKIQ_ENCRYPTED_ARGS_SECRET` envrionment variable to specify your secret, you can delimit multiple keys with a spaces.
If you are using the `SIDEKIQ_ENCRYPTED_ARGS_SECRET` environment variable to specify your secret, you can delimit multiple keys with a spaces.

You can also safely add encryption to an existing worker. Any jobs that are already enqueued will still run even without having the arguments encrypted in Redis.

## Encryption

Encrypted arguments are stored using AES-256-GCM with a key derived from your secret using PBKDF2.
Encrypted arguments are stored using AES-256-GCM with a key derived from your secret using PBKDF2. For more info on the underlying encryption, refer to the [SecretKeys](https://github.com/bdurand/secret_keys) gem.
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.0.2
1.1.0
123 changes: 87 additions & 36 deletions lib/sidekiq/encrypted_args.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,54 +6,65 @@

module Sidekiq
module EncryptedArgs
# Error thrown when the
# Error thrown when the secret is invalid
class InvalidSecretError < StandardError
def initialize
super("Cannot decrypt. Invalid secret provided.")
end
end

class << self
# Set the secret key used for encrypting arguments. If this is not set,
# the value will be loaded from the `SIDEKIQ_ENCRYPTED_ARGS_SECRET` environment
# variable. If that value is not set, arguments will not be encrypted.
#
# @param [String] value One or more secrets to use for encrypting arguments.
#
# @note You can set multiple secrets by passing an array if you need to roll your secrets.
# You can set multiple secrets by passing an array if you need to roll your secrets.
# The left most value in the array will be used as the encryption secret, but
# all the values will be tried when decrypting. That way if you have scheduled
# jobs that were encrypted with a different secret, you can still make it available
# when decrypting the arguments when the job gets run. If you are using the
# environment variable, separate the keys with spaces.
#
# @param [String] value One or more secrets to use for encrypting arguments.
# @return [void]
def secret=(value)
@encryptors = make_encryptors(value)
end

# Calling this method will add the client and server middleware to the Sidekiq
# Add the client and server middleware to the Sidekiq
# middleware chains. If you need to ensure the order of where the middleware is
# added, you can forgo this method and add it yourself.
#
# This method prepends client middleware and appends server middleware.
#
# @param [String] secret optionally set the secret here. See {.secret=}
def configure!(secret: nil)
self.secret = secret unless secret.nil?

Sidekiq.configure_client do |config|
config.client_middleware do |chain|
chain.add Sidekiq::EncryptedArgs::ClientMiddleware
chain.prepend Sidekiq::EncryptedArgs::ClientMiddleware
end
end

Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add Sidekiq::EncryptedArgs::ServerMiddleware
end
config.client_middleware do |chain|
chain.prepend Sidekiq::EncryptedArgs::ClientMiddleware
end
end
end

# Encrypt a value.
#
# @param [Object] data Data to encrypt. You can pass any JSON compatible data types or structures.
# @param [#to_json, Object] data Data to encrypt. You can pass any JSON compatible data types or structures.
#
# @return [String]
def encrypt(data)
return nil if data.nil?
json = JSON.dump(data)
json = (data.respond_to?(:to_json) ? data.to_json : JSON.generate(data))
encrypted = encrypt_string(json)
if encrypted == json
data
Expand All @@ -67,37 +78,55 @@ def encrypt(data)
# @param [String] encrypted_data Data that was previously encrypted. If the value passed in is
# an unencrypted string, then the string itself will be returned.
#
# @return [String]
# @return [Object]
def decrypt(encrypted_data)
return encrypted_data unless SecretKeys::Encryptor.encrypted?(encrypted_data)
json = decrypt_string(encrypted_data)
JSON.parse(json)
end

protected

# Helper method to get the encrypted args option from an options hash. The value of this option
# Private helper method to get the encrypted args option from an options hash. The value of this option
# can be `true` or an array indicating if each positional argument should be encrypted, or a hash
# with keys for the argument position and true as the value.
def encrypted_args_option(worker_class)
sidekiq_options = worker_class.sidekiq_options
option = sidekiq_options.fetch(:encrypted_args, sidekiq_options["encrypted_args"])

#
# @private
def encrypted_args_option(worker_class, job)
option = job["encrypted_args"]
return nil if option.nil?

return Hash.new(true) if option == true

return replace_argument_positions(worker_class, option) if option.is_a?(Hash)

hash = {}
Array(option).each_with_index do |val, position|
if val.is_a?(Symbol) || val.is_a?(String)
hash[val] = true
else
hash[position] = val
return [] if option == false

indexes = []
if option == true
job["args"].size.times { |i| indexes << i }
elsif option.is_a?(Hash)
deprecation_warning("hash")
indexes = replace_argument_positions(worker_class, option)
else
array_type = nil
deprecation_message = nil
Array(option).each_with_index do |val, position|
current_type = nil
if val.is_a?(Integer)
indexes << val
current_type = :integer
elsif val.is_a?(Symbol) || val.is_a?(String)
worker_class = constantize(worker_class) if worker_class.is_a?(String)
position = perform_method_parameter_index(worker_class, val)
indexes << position if position
current_type = :symbol
else
deprecation_message = "boolean array"
indexes << position if val
end
if array_type && current_type
deprecation_message = "array of mixed types"
else
array_type ||= current_type
end
end
deprecation_warning(deprecation_message) if deprecation_message
end
replace_argument_positions(worker_class, hash)
indexes
end

private
Expand Down Expand Up @@ -135,18 +164,40 @@ def make_encryptors(secrets)
Array(secrets).map { |val| val.nil? ? nil : SecretKeys::Encryptor.from_password(val, SALT) }
end

def replace_argument_positions(worker_class, encrypt_option)
updated = {}
encrypt_option.each do |key, value|
def deprecation_warning(message)
warn("Sidekiq::EncryptedArgs: setting encrypted_args to #{message} is deprecated; support will be removed in version 1.2.")
end

# @param [String] class_name name of a class
# @return [Class] class that was referenced by name
def constantize(class_name)
names = class_name.split("::")
# Clear leading :: for root namespace since we're already calling from object
names.shift if names.empty? || names.first.empty?
# Map reduce to the constant. Use inherit=false to not accidentally search
# parent modules
names.inject(Object) { |constant, name| constant.const_get(name, false) }
end

def replace_argument_positions(worker_class, encrypt_option_hash)
encrypted_indexes = []
encrypt_option_hash.each do |key, value|
next unless value
if key.is_a?(Symbol) || key.is_a?(String)
key = key.to_sym
position = worker_class.instance_method(:perform).parameters.find_index { |_, name| name == key }
updated[position] = value if position
position = perform_method_parameter_index(worker_class, key)
encrypted_indexes << position if position
elsif key.is_a?(Integer)
updated[key] = value
encrypted_indexes << key
end
end
updated
encrypted_indexes
end

def perform_method_parameter_index(worker_class, parameter)
if worker_class
parameter = parameter.to_sym
worker_class.instance_method(:perform).parameters.find_index { |_, name| name == parameter }
end
end
end
end
Expand Down
Loading

0 comments on commit fbc49e6

Please sign in to comment.