Update Beats to the current Sarama fork

[khushijain21]

This PR updates Beats to the current version of Elastic's Sarama fork. It upgrades sarama version to 1.43.3.

The import path has been changed from github.com/Shopify/sarama to github.com/elastic/sarama due to change in ownership.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kafka upstream/kafka
git merge upstream/main
git push upstream kafka

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[shmsr]

qq: Why don't we directly point to the fork i.e., github.com/elastic/sarama@beats-fork? Asking this because we have faced some problem with this before.

See the PR description: #39840 (comment). As it was breaking for the apm team, upon investigation, I found the problem and that's why we moved to using the fork directly. If required, I can also send the related Slack conversation in Slack.

So basically what'd happen is anyone using beats as a dependency, the replace directive won't work for them.

[khushijain21]

I think we can. @faec could you please confirm? Any particular reason we don't directly use the fork

[faec]

could you please confirm? Any particular reason we don't directly use the fork

I don't see a reason we can't, but the reason we don't is probably that we would prefer to revert to upstream someday. Though there admittedly has not been much movement on that, so I don't think that's an overriding concern if it simplifies dependency handling overall.

[mauri870]

As a side note, you can automate changes in import paths using gofmt:

gofmt -w -r '"github.com/Shopify/sarama" -> "github.com/IBM/sarama"' .

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kafka upstream/kafka
git merge upstream/main
git push upstream kafka

[khushijain21]

It uses an old version of curl which fails when you build locally. So this explicit upgrade was required

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kafka upstream/kafka
git merge upstream/main
git push upstream kafka

[khushijain21]

I think we should also update the kafka application version in our integration tests. I plan to do this in a separate PR.

[mauri870]

LGTM, thanks!

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.