[filebeat][gcs] - Added support for retry config #41862

ShourieG · 2024-12-03T10:14:28Z

Type of change

Enhancement

Proposed commit message

Added support for retry config

NOTE

Ignore the commit history as this resulted separating this PR from an earlier PR and the subsequent merges that occurred to keep the changes consistent across.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify · 2024-12-03T10:15:05Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-12-03T10:15:06Z

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

…removed changes that resulted for an earlier merge

elasticmachine · 2024-12-03T10:16:58Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

botelastic · 2024-12-03T10:17:28Z

This pull request doesn't have a Team:<team> label.

x-pack/filebeat/input/gcs/config.go

ShourieG · 2024-12-03T12:00:20Z

/test

efd6 · 2024-12-03T21:48:59Z

CHANGELOG.next.asciidoc

+- Added support for retry config in GCS input. {issue}11580[11580] {pull}41862[41862]
+


Suggested change

- Added support for retry config in GCS input. {issue}11580[11580] {pull}41862[41862]

- Added support for retry configuration in GCS input. {issue}11580[11580] {pull}41862[41862]

efd6 · 2024-12-03T21:50:47Z

x-pack/filebeat/docs/inputs/input-gcs.asciidoc

+[float]
+==== `retry`
+
+This attribute can be used to configure a list of sub attributes that directly control how the input should behave when a download for a file/object fails or gets interrupted. The list of sub attributes are as follows :-


Suggested change

This attribute can be used to configure a list of sub attributes that directly control how the input should behave when a download for a file/object fails or gets interrupted. The list of sub attributes are as follows :-

This attribute can be used to configure a list of sub attributes that directly control how the input should behave when a download for a file/object fails or gets interrupted.

Also suggest a <ul> rather than an <ol> since the ordering of the properties does not matter.

efd6 · 2024-12-03T21:54:44Z

x-pack/filebeat/docs/inputs/input-gcs.asciidoc

+This attribute can be used to configure a list of sub attributes that directly control how the input should behave when a download for a file/object fails or gets interrupted. The list of sub attributes are as follows :-
+
+    1. `max_attempts`: This attribute defines the maximum number of retry attempts(including the initial api call) that should be attempted for a retryable error. The default value for this is `3`. 
+    2. `initial_backoff_duration`: This attribute defines the initial backoff time in seconds. The default value for this is `1s`.


Suggested change

2. `initial_backoff_duration`: This attribute defines the initial backoff time in seconds. The default value for this is `1s`.

2. `initial_backoff_duration`: This attribute defines the initial backoff time. The default value for this is `1s`.

This is a time.Duration and has a unit awareness. Perhaps note this rather than saying that it's in seconds. Also below.

efd6 · 2024-12-03T21:55:48Z

x-pack/filebeat/docs/inputs/input-gcs.asciidoc

+    bucket_timeout: 10m
+----
+
+While configuring the `retry` attribute, the user should take into consideration the `bucket_timeout` value. The `retry` attribute should be configured in such a way that the retries are completed within the `bucket_timeout` window. If the `retry` attribute is configured in such a way that the retries are not completed successfully within the `bucket_timeout` window, then the input will suffer a `context timeout` for that specific object/file which it was retrying. This can cause gaps in ingested data to pile up over time.


Suggested change

While configuring the `retry` attribute, the user should take into consideration the `bucket_timeout` value. The `retry` attribute should be configured in such a way that the retries are completed within the `bucket_timeout` window. If the `retry` attribute is configured in such a way that the retries are not completed successfully within the `bucket_timeout` window, then the input will suffer a `context timeout` for that specific object/file which it was retrying. This can cause gaps in ingested data to pile up over time.

When configuring the `retry` attribute, the user should consider the `bucket_timeout` value. The `retry` attribute should be configured in such a way that the retries are completed within the `bucket_timeout` window. If the `retry` attribute is configured in such a way that the retries are not completed successfully within the `bucket_timeout` window, the input will suffer a `context timeout` for that specific object/file which it was retrying. This can cause gaps in ingested data to pile up over time.

efd6 · 2024-12-03T21:56:46Z

x-pack/filebeat/input/gcs/config.go

@@ -126,5 +141,11 @@ func defaultConfig() config {
 		PollInterval:  5 * time.Minute,
 		BucketTimeOut: 120 * time.Second,
 		ParseJSON:     false,
+		Retry: retryConfig{
+			MaxAttempts:            3,
+			InitialBackOffDuration: 1 * time.Second,


Suggested change

InitialBackOffDuration: 1 * time.Second,

InitialBackOffDuration: time.Second,

mergify · 2024-12-03T22:44:50Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b gcs/add_retry_config upstream/gcs/add_retry_config
git merge upstream/main
git push upstream gcs/add_retry_config

ShourieG · 2024-12-04T05:05:40Z

@efd6, I've addressed the PR suggestions.

efd6 · 2024-12-04T05:12:10Z

x-pack/filebeat/docs/inputs/input-gcs.asciidoc

+    - `max_backoff_duration`: This attribute defines the maximum backoff time. The default value for this is `30s`.
+    - `backoff_multiplier`: This attribute defines the backoff multiplication factor. The default value for this is `2`.
+
+NOTE: Attributes such as `initial_backoff_duration` and `max_backoff_duration` can be defined in the following formats: `{{x}}s`, `{{x}}m`, `{{x}}h`, here `s = seconds`, `m = minutes` and `h = hours`. The value `{{x}}` can be anything we wish.


I think something simpler like "NOTE: The initial_backoff_duration and max_backoff_duration attributes must have time units. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

efd6 · 2024-12-04T05:19:26Z

x-pack/filebeat/docs/inputs/input-gcs.asciidoc

+    - `max_backoff_duration`: This attribute defines the maximum backoff time. The default value for this is `30s`.
+    - `backoff_multiplier`: This attribute defines the backoff multiplication factor. The default value for this is `2`.
+
+NOTE: The `initial_backoff_duration` and `max_backoff_duration` attributes must have time units. Valid time units are `ns`, `us (or µs)`, `ms`, `s`, `m`, `h`


Suggested change

NOTE: The `initial_backoff_duration` and `max_backoff_duration` attributes must have time units. Valid time units are `ns`, `us (or µs)`, `ms`, `s`, `m`, `h`

NOTE: The `initial_backoff_duration` and `max_backoff_duration` attributes must have time units. Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`.

Committed this

Co-authored-by: Dan Kortschak <[email protected]>

* Added support for retry config along with necessary documentation and tests (cherry picked from commit 3f51793)

* Added support for retry config along with necessary documentation and tests (cherry picked from commit 3f51793) Co-authored-by: ShourieG <[email protected]>

ShourieG added 6 commits November 20, 2024 15:26

added retry config, refactored the defauyltConfig() and updated docs

631669f

Merge remote-tracking branch 'upstream/main' into gcs/retry_config

c8677db

updated changelog

2884058

Merge remote-tracking branch 'upstream/main' into gcs/retry_config

90ba607

Merge remote-tracking branch 'upstream/main' into gcs/add_retry_config

0fb5bdf

resolved merge conflicts

7fde3fa

ShourieG requested a review from a team as a code owner December 3, 2024 10:14

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 3, 2024

mergify bot assigned ShourieG Dec 3, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Dec 3, 2024

ShourieG added 2 commits December 3, 2024 15:45

updated changelog, added new test to check minimum value failure and …

3ab87bc

…removed changes that resulted for an earlier merge

updated changelog pull req number

6ee6dae

ShourieG added the Team:Security-Service Integrations Security Service Integrations Team label Dec 3, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 3, 2024

ShourieG added Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label labels Dec 3, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 3, 2024

ShourieG added enhancement needs_team Indicates that the issue/PR needs a Team:* label labels Dec 3, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 3, 2024

ShourieG added needs_team Indicates that the issue/PR needs a Team:* label input:GCS labels Dec 3, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 3, 2024

ShourieG assigned ShourieG and unassigned ShourieG Dec 3, 2024

ShourieG requested a review from efd6 December 3, 2024 10:19

ShourieG commented Dec 3, 2024

View reviewed changes

x-pack/filebeat/input/gcs/config.go Show resolved Hide resolved

efd6 reviewed Dec 3, 2024

View reviewed changes

ShourieG added 2 commits December 4, 2024 10:23

Merge remote-tracking branch 'upstream/main' into gcs/add_retry_config

6903122

addressed Dan's PR suggestions regarding docs and default config values

774e4b8

efd6 reviewed Dec 4, 2024

View reviewed changes

addressed latest PR suggestions

0b8a8a4

efd6 reviewed Dec 4, 2024

View reviewed changes

Update input-gcs.asciidoc

353180b

Co-authored-by: Dan Kortschak <[email protected]>

efd6 approved these changes Dec 4, 2024

View reviewed changes

ShourieG merged commit 3f51793 into elastic:main Dec 4, 2024
20 of 22 checks passed

mergify bot pushed a commit that referenced this pull request Dec 4, 2024

[filebeat][gcs] - Added support for retry config (#41862)

8f1c3db

* Added support for retry config along with necessary documentation and tests (cherry picked from commit 3f51793)

mergify bot mentioned this pull request Dec 4, 2024

[8.x](backport #41862) [filebeat][gcs] - Added support for retry config #41885

Merged

5 tasks

ShourieG deleted the gcs/add_retry_config branch December 4, 2024 07:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[filebeat][gcs] - Added support for retry config #41862

[filebeat][gcs] - Added support for retry config #41862

ShourieG commented Dec 3, 2024 •

edited

Loading

mergify bot commented Dec 3, 2024

mergify bot commented Dec 3, 2024

elasticmachine commented Dec 3, 2024

botelastic bot commented Dec 3, 2024

ShourieG commented Dec 3, 2024

efd6 Dec 3, 2024

efd6 Dec 3, 2024

efd6 Dec 3, 2024

efd6 Dec 3, 2024

efd6 Dec 3, 2024

mergify bot commented Dec 3, 2024

ShourieG commented Dec 4, 2024

efd6 Dec 4, 2024

ShourieG Dec 4, 2024

efd6 Dec 4, 2024

ShourieG Dec 4, 2024

		- Added support for retry config in GCS input. {issue}11580[11580] {pull}41862[41862]

	- Added support for retry config in GCS input. {issue}11580[11580] {pull}41862[41862]
	- Added support for retry configuration in GCS input. {issue}11580[11580] {pull}41862[41862]

	This attribute can be used to configure a list of sub attributes that directly control how the input should behave when a download for a file/object fails or gets interrupted. The list of sub attributes are as follows :-
	This attribute can be used to configure a list of sub attributes that directly control how the input should behave when a download for a file/object fails or gets interrupted.

	2. `initial_backoff_duration`: This attribute defines the initial backoff time in seconds. The default value for this is `1s`.
	2. `initial_backoff_duration`: This attribute defines the initial backoff time. The default value for this is `1s`.

	InitialBackOffDuration: 1 * time.Second,
	InitialBackOffDuration: time.Second,

	NOTE: The `initial_backoff_duration` and `max_backoff_duration` attributes must have time units. Valid time units are `ns`, `us (or µs)`, `ms`, `s`, `m`, `h`
	NOTE: The `initial_backoff_duration` and `max_backoff_duration` attributes must have time units. Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`.

[filebeat][gcs] - Added support for retry config #41862

[filebeat][gcs] - Added support for retry config #41862

Conversation

ShourieG commented Dec 3, 2024 • edited Loading

Type of change

Proposed commit message

NOTE

Checklist

Disruptive User Impact

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Dec 3, 2024

mergify bot commented Dec 3, 2024

elasticmachine commented Dec 3, 2024

botelastic bot commented Dec 3, 2024

ShourieG commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Dec 3, 2024

ShourieG commented Dec 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShourieG commented Dec 3, 2024 •

edited

Loading