Skip to content

Commit

Permalink
Add split event processor for dataprepper (opensearch-project#6682)
Browse files Browse the repository at this point in the history
* Add split event processor for dataprepper

Signed-off-by: srigovs <[email protected]>

* Adding usage example

* Update split-event.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: srigovs <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
  • Loading branch information
3 people authored Mar 27, 2024
1 parent 6f862fa commit 5a873ab
Showing 1 changed file with 52 additions and 0 deletions.
52 changes: 52 additions & 0 deletions _data-prepper/pipelines/configuration/processors/split-event.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
---
layout: default
title: split-event
parent: Processors
grand_parent: Pipelines
nav_order: 56
---

# split-event

The `split-event` processor is used to split events based on a delimiter and generates multiple events from a user-specified field.

## Configuration

The following table describes the configuration options for the `split-event` processor.

| Option | Type | Description |
|------------------|---------|-----------------------------------------------------------------------------------------------|
| `field` | String | The event field to be split. |
| `delimiter_regex`| String | The regular expression used as the delimiter for splitting the field. |
| `delimiter` | String | The delimiter used for splitting the field. If not specified, the default delimiter is used. |

# Usage

To use the `split-event` processor, add the following to your `pipelines.yaml` file:

```
split-event-pipeline:
source:
http:
processor:
- split_event:
field: query
delimiter: ' '
sink:
- stdout:
```
{% include copy.html %}

When an event contains the following example input:

```
{"query" : "open source", "some_other_field" : "abc" }
```

The input will be split into multiple events based on the `query` field, with the delimiter set as white space, as shown in the following example:

```
{"query" : "open", "some_other_field" : "abc" }
{"query" : "source", "some_other_field" : "abc" }
```

0 comments on commit 5a873ab

Please sign in to comment.