Skip to content

v2.25.0
Compare
Choose a tag to compare
@github-actions github-actions released this 15 Sep 15:05
· 1259 commits to refs/heads/develop since this release
v2.25.0
781fbba

Summary

This release simplifies data transformation with Amazon Kinesis Data Firehose, and handling secret rotation events from Amazon Secrets Manager.

🌟 Huge welcome to our new contributor @TonySherman. Tony documented how to use Event Handler with micro Lambda functions.

Data Transformation code snippet

Data transformation

Docs

When using Kinesis Firehose, you can use a Lambda function to perform data transformation. For each transformed record, you can choose to either:

  • A) Put them back to the delivery stream (default)
  • B) Drop them so consumers don't receive them (e.g., data validation)
  • C) Indicate a record failed data transformation and should be retried

To make this process easier, you can now use KinesisFirehoseDataTransformationResponse and serialization functions to quickly encode payloads into base64 data for the stream.

Example where you might want to drop unwanted records from the stream.

from json import JSONDecodeError
from typing import Dict

from aws_lambda_powertools.utilities.data_classes import (
    KinesisFirehoseDataTransformationRecord,
    KinesisFirehoseDataTransformationResponse,
    KinesisFirehoseEvent,
    event_source,
)
from aws_lambda_powertools.utilities.serialization import base64_from_json
from aws_lambda_powertools.utilities.typing import LambdaContext


@event_source(data_class=KinesisFirehoseEvent)
def lambda_handler(event: KinesisFirehoseEvent, context: LambdaContext):
    result = KinesisFirehoseDataTransformationResponse()

    for record in event.records:
        try:
            payload: Dict = record.data_as_json  # decodes and deserialize base64 JSON string

            ## generate data to return
            transformed_data = {"tool_used": "powertools_dataclass", "original_payload": payload}

            processed_record = KinesisFirehoseDataTransformationRecord(
                record_id=record.record_id,
                data=base64_from_json(transformed_data),
            )
        except JSONDecodeError:  
            # our producers ingest JSON payloads only; drop malformed records from the stream
            processed_record = KinesisFirehoseDataTransformationRecord(
                record_id=record.record_id,
                data=record.data,
                result="Dropped",
            )

        result.add_record(processed_record)

    # return transformed records
    return result.asdict()

Rotating secrets

Docs

When rotating secrets with Secrets Manager, it invokes your Lambda function in four potential steps:

  • createSecret. Create a new version of the secret.
  • setSecret. Change the credentials in the database or service.
  • testSecret. Test the new secret version.
  • finishSecret. Finish the rotation.

You can now use SecretsManagerEvent to more easily access the event structure, and combine Parameters to get secrets to perform secret operations.

from aws_lambda_powertools.utilities import parameters
from aws_lambda_powertools.utilities.data_classes import SecretsManagerEvent, event_source

secrets_provider = parameters.SecretsProvider()


@event_source(data_class=SecretsManagerEvent)
def lambda_handler(event: SecretsManagerEvent, context):
    # Getting secret value using Parameter utility
    # See https://docs.powertools.aws.dev/lambda/python/latest/utilities/parameters/
    secret = secrets_provider.get(event.secret_id, VersionId=event.version_id, VersionStage="AWSCURRENT")

    if event.step == "setSecret":
        # Perform any secret rotation logic, e.g., change DB password
        # Check more examples: https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas
        print("Rotating secret...")

    return secret

Changes

🌟New features and non-breaking changes

  • feat(event_source): add Kinesis Firehose Data Transformation data class (#3029) by @roger-zhangg
  • feat(event_sources): add Secrets Manager secret rotation event (#3061) by @roger-zhangg

📜 Documentation updates

🔧 Maintenance

  • chore(deps-dev): bump aws-cdk from 2.96.0 to 2.96.1 (#3093) by @dependabot
  • chore(typing): move backwards compat types to shared types (#3092) by @heitorlessa
  • refactor(parameters): BaseProvider._get to also support Dict (#3090) by @leandrodamascena
  • chore(deps): bump docker/setup-qemu-action from 2.2.0 to 3.0.0 (#3081) by @dependabot
  • chore(deps): bump docker/setup-buildx-action from 2.10.0 to 3.0.0 (#3083) by @dependabot
  • chore(deps-dev): bump cfn-lint from 0.79.10 to 0.79.11 (#3088) by @dependabot
  • chore(deps-dev): bump sentry-sdk from 1.30.0 to 1.31.0 (#3086) by @dependabot
  • chore(deps-dev): bump aws-cdk from 2.95.1 to 2.96.0 (#3087) by @dependabot
  • chore(deps-dev): bump the boto-typing group with 1 update (#3085) by @dependabot
  • chore(deps-dev): bump ruff from 0.0.288 to 0.0.289 (#3080) by @dependabot
  • chore(deps-dev): bump cfn-lint from 0.79.9 to 0.79.10 (#3077) by @dependabot
  • chore(deps-dev): bump hvac from 1.2.0 to 1.2.1 (#3075) by @dependabot
  • chore(deps): bump squidfunk/mkdocs-material from dd1770c to c4890ab in /docs (#3078) by @dependabot
  • chore(deps-dev): bump ruff from 0.0.287 to 0.0.288 (#3076) by @dependabot
  • chore(deps-dev): bump aws-cdk from 2.95.0 to 2.95.1 (#3074) by @dependabot
  • chore(deps): bump actions/dependency-review-action from 3.0.8 to 3.1.0 (#3071) by @dependabot
  • chore(deps-dev): bump aws-cdk from 2.94.0 to 2.95.0 (#3070) by @dependabot
  • chore(automation): remove previous labels when PR is updated (#3066) by @sthulb

This release was made possible by the following contributors:

@TonySherman, @dependabot, @dependabot[bot], @github-actions, @github-actions[bot], @heitorlessa, @leandrodamascena, @roger-zhangg and @sthulb

Contributors

sthulb, heitorlessa, and 4 other contributors
Assets 3
Loading