v2.8.0

[heitorlessa]

Summary

This release is packed with new features and enhancements, specifically to Batch Processing, Metrics and Idempotency utility.

⭐ Huge welcome to new contributors: @BakasuraRCE, @prudnikov, @kavichu, @marcraminv

High-concurrency async processor

Docs

Thanks to @BakasuraRCE, we’re adding a new specialized processor for high concurrency jobs that can process thousands of messages sub-second (thanks to @sthulb for load testing) - AsyncBatchProcessor and async_batch_processor.

These will use Python’s asyncio.gather to call your record handler with all records at the same time. During load testing, we’ve managed to process 5500 messages in approximately 300ms compared to 1.2 seconds with BatchProcessor synchronous counterpart.

1-second resolution metrics

Docs

You can now create high-resolution (1s) metrics that are common for use cases like IoT, telemetry, time-series, real-time incident, etc. Previously, metrics were only created with 60-second resolution. This is an opt-in feature.

Inheriting dimensions in single_metric

Docs

Thanks to @prudnikov, you can now optionally set default dimensions for single metrics. These are individual application or operational metrics that despite being isolated from a larger set of metrics, they do share the same set of dimensions (avoid repetition, sparse metrics).

Idempotent_function decorator is now thread-safe

Thanks to @mploski, any Python function decorated with idempotent_function can now safely be called from threads. Previously, we used to create higher-level AWS SDK clients which didn’t support threading - calling functions decorated back then could result in unforeseen side-effects.

Example

import os
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from threading import current_thread

from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    idempotent_function,
)

TABLE_NAME = os.getenv("IdempotencyTable", "")
persistence_layer = DynamoDBPersistenceLayer(table_name=TABLE_NAME)
threads_count = 2


@idempotent_function(persistence_store=persistence_layer, data_keyword_argument="record")
def record_handler(record):
    time_now = time.time()
    return {"thread_name": current_thread().name, "time": str(time_now)}


def lambda_handler(event, context):
    with ThreadPoolExecutor(max_workers=threads_count) as executor:
        futures = [executor.submit(record_handler, **{"record": event}) for _ in range(threads_count)]

    return [
        {"state": future._state, "exception": future.exception(), "output": future.result()}
        for future in as_completed(futures)
    ]

Changes

🌟New features and non-breaking changes

• feat(batch): add async_batch_processor for concurrent processing (feat(batch): add async_batch_processor for concurrent processing #1724) by @BakasuraRCE
• feat(metrics): add default_dimensions to single_metric (feat(metrics): add default_dimensions to single_metric #1880) by @prudnikov

📜 Documentation updates

• docs(homepage): Replace poetry command to add group parameter (docs(homepage): Replace poetry command to add group parameter #1917) by @marcraminv
• feat(metrics) - add support for high resolution metrics (feat(metrics) - add support for high resolution metrics #1915) by @leandrodamascena
• feat(batch): add async_batch_processor for concurrent processing (feat(batch): add async_batch_processor for concurrent processing #1724) by @BakasuraRCE
• fix(idempotency): make idempotent_function decorator thread safe (fix(idempotency): make idempotent_function decorator thread safe #1899) by @mploski
• docs(idempotency): add IAM permissions section (docs(idempotency): add IAM permissions section #1902) by @leandrodamascena
• docs(homepage): set url for end-of-support in announce block (docs(homepage): set url for end-of-support in announce block #1893) by @kavichu
• feat(metrics): add default_dimensions to single_metric (feat(metrics): add default_dimensions to single_metric #1880) by @prudnikov

🐛 Bug and hot fixes

• fix(idempotency): make idempotent_function decorator thread safe (fix(idempotency): make idempotent_function decorator thread safe #1899) by @mploski

🔧 Maintenance

• feat(batch): add async_batch_processor for concurrent processing (feat(batch): add async_batch_processor for concurrent processing #1724) by @BakasuraRCE
• chore(deps-dev): bump mkdocs-material from 9.0.11 to 9.0.12 (chore(deps-dev): bump mkdocs-material from 9.0.11 to 9.0.12 #1919) by @dependabot
• chore(deps-dev): bump aws-cdk-lib from 2.63.2 to 2.64.0 (chore(deps-dev): bump aws-cdk-lib from 2.63.2 to 2.64.0 #1918) by @dependabot
• chore(pypi): add new links to Pypi package homepage (chore(pypi): add new links to Pypi package homepage #1912) by @leandrodamascena
• chore(deps-dev): bump types-requests from 2.28.11.8 to 2.28.11.12 (chore(deps-dev): bump types-requests from 2.28.11.8 to 2.28.11.12 #1906) by @dependabot
• chore(deps-dev): bump pytest-xdist from 3.1.0 to 3.2.0 (chore(deps-dev): bump pytest-xdist from 3.1.0 to 3.2.0 #1905) by @dependabot
• chore(deps): bump docker/setup-buildx-action from 2.4.0 to 2.4.1 (chore(deps): bump docker/setup-buildx-action from 2.4.0 to 2.4.1 #1903) by @dependabot
• chore(deps-dev): bump aws-cdk-lib from 2.63.0 to 2.63.2 (chore(deps-dev): bump aws-cdk-lib from 2.63.0 to 2.63.2 #1904) by @dependabot
• chore(deps-dev): bump mkdocs-material from 9.0.10 to 9.0.11 (chore(deps-dev): bump mkdocs-material from 9.0.10 to 9.0.11 #1896) by @dependabot
• chore(deps-dev): bump mypy-boto3-appconfig from 1.26.0.post1 to 1.26.63 (chore(deps-dev): bump mypy-boto3-appconfig from 1.26.0.post1 to 1.26.63 #1895) by @dependabot
• docs(homepage): set url for end-of-support in announce block (docs(homepage): set url for end-of-support in announce block #1893) by @kavichu
• chore(deps-dev): bump mkdocs-material from 9.0.9 to 9.0.10 (chore(deps-dev): bump mkdocs-material from 9.0.9 to 9.0.10 #1888) by @dependabot
• chore(deps-dev): bump mypy-boto3-s3 from 1.26.58 to 1.26.62 (chore(deps-dev): bump mypy-boto3-s3 from 1.26.58 to 1.26.62 #1889) by @dependabot
• chore(deps-dev): bump black from 22.12.0 to 23.1.0 (chore(deps-dev): bump black from 22.12.0 to 23.1.0 #1886) by @dependabot
• chore(deps-dev): bump aws-cdk-lib from 2.62.2 to 2.63.0 (chore(deps-dev): bump aws-cdk-lib from 2.62.2 to 2.63.0 #1887) by @dependabot

This release was made possible by the following contributors:

@BakasuraRCE, @dependabot, @dependabot[bot], @heitorlessa, @kavichu, @leandrodamascena, @marcraminv, @mploski, @prudnikov and Release bot

---

This discussion was created from the release v2.8.0.