Add configurable commitAllMetricUpdates

[FuRyanf]

Description of Changes

This update introduces a configurable option to control the invocation of commitUpdates within the updateMetrics method in the SamzaMetricsContainer class. The new configuration option, beam.samza.metrics.commitAllMetricUpdates, allows toggling the behavior of committing metrics after they have been processed.

Changes:

• New Configuration:

  • Added a new configuration property, beam.samza.metrics.commitAllMetricUpdates, which determines whether commitUpdates is called for each metrics container after metrics are updated.

• Code Logic:

  • The line stepNameList.stream().map(metricsContainers::getContainer).forEach(MetricsContainerImpl::commitUpdates); ensures that metrics are marked as committed, preventing them from being reprocessed in subsequent steps. This is particularly important for avoiding the accumulation of metrics and the associated CPU overhead.

Rationale:

• Performance Considerations:

  • The commitUpdates method, similar to getUpdates, may have a significant CPU cost. Preliminary analysis shows that getUpdates contributed 8% of CPU time for the updateMetrics call. Given that commitUpdates could incur similar overhead, the configuration option provides flexibility to disable this behavior for jobs where all metrics in a step are always used, making the commit operation unnecessary.

• Testing and Flexibility:

  • By making the commit behavior configurable, we can test the impact of this change on various jobs and optimize performance accordingly. This allows us to determine if the additional CPU cost of committing updates is justified based on the specific needs of different workloads.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.