[aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups

[Kavindu-Dodan]

Proposed commit message

Introduce ignore_older and start_timestamp properties to AWS S3 input. This is a follow-up for #41694.

The configurations introduced here act as input object filters. If the object fails to match derived filters, the entries will be cleaned up from the registry, reducing filebeat memory consumption.

Introduced configurations are,

• ignore_older : Accepts a time duration in which entries are accepted for processing
• start_timestamp: A timestamp from which objects are accepted for processing

For both inputs, the object's last modified timestamp is taken into comparison. See Use cases section for further explanation

Note - a follow-up for #41694. Hence diff contains all changes

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None as defaults are disabled. However, when configurations introduced here are used, the following can have an impact on the user,

• Whenstart_timestamp is defined, then objects with the last modified timestamps prior to the timestamp are ignored from processing (documented ¹)
• When ignore_older is defined, then objects that do not fall within the look-back period when processing started (polling run) are ignored (documented ¹)
• When both start_timestamp & ignore_older are defined, the initial run will process all entries up to start_timestamp. The subsequent runs will not include entries that do not fall within ignore_older even if processing failed for an object. (documented ¹)

How to test this PR locally

• Build filebeat from the changest included in the PR
• Source S3 bucket with objects (you may use this tool ² to create entries)
• Try configuring filebeat with alternative values for ignore_older & start_timestamp to see how data ingestion change with their values. See Use cases section for further explanation

Related issues

• Depends on [aws] feat: aws-s3 input registry cleanup for untracked s3 objects #41694
• Fixes aws-s3 input's bucket polling accumulates state in the registry #39116

Use cases

Consider below diagrams where there're 3 objects Object A, Object B and Object C with their respect last modified timestamps t1, t2 and t3.

And consider how filebeat processes and track registry entries based on following scnearios

Default behavior

If none of the configurations used, then filebeat will process and internal registry will track all objects continuously unless they are removed from the bucket.

Use start_timestamp

If start_timestamp is used, then objects newer than the timestamp are accepted for processing. The registry will grow unless objects are removed from the bucket.

Use ignore_older

If ignore_older is defined, input will process objects within the provided duration, calculated from the current time. The registry will track objects within the current timeframe and others will get cleaned up eventually by subsequent runs.

Use both ignore_older & start_timestamp

If both properties are defined,

• The initial run will include entries that fall within the start_timestamp (ignoring ignore_older duration).
• Subsequent runs will only consider entries that fall within the ignore_older duration.

Footnotes

1. https://github.com/elastic/beats/pull/41817/files#diff-422765b7341c5bbf6de7af38927e34e00a5073b188585a7af3c4fee1175b64a6R574-R597 ↩ ↩² ↩³

2. https://github.com/Kavindu-Dodan/data-gen ↩

[Kavindu-Dodan]

todo - fix this when #41694 is merged. This is here as changes are based on #41694