Laravel Data Shipper

Laravel Data Shipper is a tool that helps relieve stress from secondary data sources (e.g. Elasticsearch) by delaying non time-sensitive data updates that is managed in a queue based system in Redis.

Installation

You can install the package via composer:

composer require autoklose/laravel-data-shipper

You can publish and run the migrations with:

php artisan vendor:publish --tag="data-shipper-migrations"
php artisan migrate

You can publish the config file with:

php artisan vendor:publish --tag="data-shipper-config"

You can customize how often data shipments happen by changing the following values in the config file:

max_size

• How many updates are held in the queue for a model before it will be acted upon.

max_wait_minutes

• How many minutes should Data Shipper wait until shipping changes regardless of not yet reaching the max queue size.

max_shipments_per_minute

• How many times a shipment can be handled per a minute.

max_retries

• How many times a failed shipment can be retried before no longer being handled.

return [
    'subscribers' => ['elasticsearch'],
    'shipments' => [
        'max_size' => 10,
        'max_wait_minutes' => 5,
        'max_shipments_per_minute' => 10,
        'max_retries' => 3
    ]
];

In order for updates/changes passed to DataShipper to be acted upon, you must add the following command to your Laravel project's scheduler:

protected function schedule(Schedule $schedule)
{
  // ...

  $schedule->command('data-shipper:ship-it')->everyMinute();
}

All models that are passed through Data Shipper must use the HasDataSubscriber trait.

use Illuminate\Database\Eloquent\Model;
use Autoklose\DataShipper\Traits\HasDataSubscribers;

class Record extends Model
{
  use HasDataSubscribers;
}

Usage

Data Shipper can be made use of anywhere in your app using the Data Shipper Facade.

Shipping Changes to a Model

Data Shipper provides support to automatically detect changes to a model and add them to the shipment queue.

However, you must push the model before saving the model instance.

$record->foo = 'bar';
DataShipper::pushModel($record);

$record->save();

In cases where you only want certain fields to be written to the shipment queue, you can specifiy what columns should be observed.

$record->foo = 'bar';
$record->bar = 'foo';

// Only changes made to the 'foo' field will be acted on
DataShipper::pushModel($record, ['foo']);

$record->save();

Shipping Specific Data

In some cases you may want to provide a custom made array of data that should be passed to data subscribers.

In order to do this, provide the class of the model the changes are related to, an array of changes and the identifier for the model you are applying the changes to.

DataShipper::push(Record::class, ['text_field' => 'changed text'], $record->key());

Shipping Multiple Changes

If you have a large set of changes you want to provide to subscribers you can do so by using Data Shippers pushMany method.

You can provide an array of changes, however each array item must have an identifier.

$changes = [
  ['id' => 1, 'text_field' => 'change 1'],
  ['id' => 2, 'text_field' => 'change 2'],
];

DataShipper::pushMany(Record::class, $changes, 'id');

Renaming Columns per Subscriber

In some subscribers you may have a column named differently than you do than in your primary data source. You can easily remap these columns in your changes by adding a column map to your model

use Illuminate\Database\Eloquent\Model;
use Autoklose\DataShipper\Traits\HasDataSubscribers;

class Record extends Model
{
  use HasDataSubscribers;

  protected $elasticsearchMap = [
    'sql_column_name' => 'elastic_search_column_name'
  ];
}

Transforming Data Before Shipping

If you need to massage the data before it is handled by a subscriber you can make use of the transformData method on your model.

use Illuminate\Database\Eloquent\Model;
use Autoklose\DataShipper\Traits\HasDataSubscribers;

class Record extends Model
{
  use HasDataSubscribers;

  public function transformData($value, $key, $subscriber) {
      if ($subscriber === 'elasticsearch') {
        if ($key === 'foo') {
          return $value * 2;
        } else if ($key === 'bar') {
          return $value . ' added';
        }
      }

      return $value;
  }
}

Available Data Subscribers

Elasticsearch

Elasticsearch is available as the default data subscriber. In order to work with this subscriber, each class that is passed through Data Shipper must have an elasticsearch_index field that is publically available.

It can be as simple as a public string property on your class.

use Illuminate\Database\Eloquent\Model;
use Autoklose\DataShipper\Traits\HasDataSubscribers;

class Record extends Model
{
  use HasDataSubscribers;

  public string $elasticsearch_index = 'record_index';
}

Alternatively if you need to retrieve the index name programmatically we recommend making use of an attribute. Please note that you should not rely on class properties when creating the index attribute as the class will not have any data loaded.

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Casts\Attribute;
use Autoklose\DataShipper\Traits\HasDataSubscribers;

class Record extends Model
{
  use HasDataSubscribers;

  public function elasticsearchIndex(): Attribute
  {
    return Attribute::make(get: fn() => config('elasticsearch.recordIndex'));
  }
}

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

• Simon Chawla
• All Contributors

License

This project is licensed under the Apache License, Version 2.0. Please see License File for more information.