How to convert DynamoDBEvent images to DynamoDBv2.Document

[dscpinheiro]

Discussed in #1654

^{Originally posted by Dreamescaper January 16, 2024}
I have a lambda subscribed to dynamoDb events. I want to convert event's records to regular JSON, currently we use something like that:

public async Task FunctionHandler(DynamoDBEvent input, ILambdaContext context)
{
        foreach (var record in input.Records)
        {
                var image = record.Dynamodb.NewImage;
                var document = Document.FromAttributeMap(image);
                var json = document.ToJson();
                /*   ....    */
        }
}

However, it doesn't work after the #1648 is merged.
Could anyone suggest what is the easiest way to do same after the update?

[adam-knights]

We've just hit this as well. We can no longer do

var document = Document.FromAttributeMap(attributeValues);
return _dynamoDBContext.FromDocument<MyObject>(document);

I can understand wanting to break the dependency but its weird to have AttributeValue duplicated as different classes across the two libraries, is the plan to have it live in a shared model library?

[ashovlin]

We've released version 3.1.0 of Amazon.Lambda.DynamoDBEvents, which adds a ToJson and ToJsonPretty that can be used with DynamoDBEvent.

We still kept the event definition separate from the SDK definition in AWSSDK.DynamoDBv2, which we split in version 3.0.0 via #1648. This avoids some interaction with code relevant to the SDK but not to Lambda (request marshallers), and may reduce the package size for cases where one only needs to read the event without using the full DynamoDB SDK.

You can now use the ToJson method on either OldImage or NewImage to:

1. Convert to JSON
2. Convert it to the SDK's Document
3. Convert it to the SDK's object persistence classes.

Note that the JSON conversion has the same limitations as the SDK: the sets (SS, NS, BS) will be converted to JSON arrays, and binary (B) will be converted to Base64 strings

foreach (var record in dynamoEvent.Records)
{
    // Convert the event to a JSON string
    var json = record.Dynamodb.NewImage.ToJson();

    // Which you can convert to the mid-level document model
    var document = Document.FromJson(json);
    
    // And then to the high-level object model using an IDynamoDBContext
    var myClass = context.FromDocument<T>(document);
}

I think that will address both use cases reported above, but let us know if you're still seeing limitations after 3.1.0. Thanks.

[psdanielhosseini]

@ashovlin Thanks, everything worked great!

However, we have a scenario (testing) where we are building a DynamoDB stream event based of an object - see code

 public static DynamoDBEvent CreateDynamoDbEvent(OperationType type, object newObj, object prevObj)
    {
        return new DynamoDBEvent
        {
            Records = new List<DynamodbStreamRecord>
            {
                new()
                {
                    Dynamodb = new DynamoDBEvent.StreamRecord
                    {
                        NewImage = ToDynamoDbAttributes(newObj),
                        OldImage = ToDynamoDbAttributes(prevObj)
                    },
                    EventName = new OperationType(type)
                }
            }
        };
    }

private static Dictionary<string, DynamoDBEvent.AttributeValue> ToDynamoDbAttributes(object obj)
    {
        if (obj == null) return null;

        var attributes = Document.FromJson(obj.ToJson()).ToAttributeMap();
        return attributes;
    }

Now I want to return an Dictionary<string, DynamoDBEvent.AttributeValue, but not really sure how I would go about it now that ToAttributeMap() returns a different AttributeValue. What's the best approach here?

[adam-knights]

The only thing we spotted so far is that one of our unit tests shows An exception of type 'System.InvalidOperationException' occurred in System.Text.Json.dll but was not handled in user code: 'Cannot write a JSON property name following another property name. A JSON value is missing.'

I think this might be caused by the WriteJsonValue of

aws-lambda-dotnet/Libraries/src/Amazon.Lambda.DynamoDBEvents/ExtensionMethods.cs

Line 65 in 9569c1a

private static void WriteJson(Utf8JsonWriter writer, Dictionary<string, AttributeValue> item)

- namely there's no final else, so we get the error in the stack:

   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource, Int32 currentDepth, Int32 maxDepth, Byte token, JsonTokenType tokenType)
   at System.Text.Json.Utf8JsonWriter.WriteStringByOptionsPropertyName(ReadOnlySpan`1 propertyName)
   at System.Text.Json.Utf8JsonWriter.WritePropertyName(ReadOnlySpan`1 propertyName)
   at Amazon.Lambda.DynamoDBEvents.ExtensionMethods.WriteJson(Utf8JsonWriter writer, Dictionary`2 item)
   at Amazon.Lambda.DynamoDBEvents.ExtensionMethods.ToJson(Dictionary`2 item, Boolean prettyPrint)
   at Amazon.Lambda.DynamoDBEvents.ExtensionMethods.ToJson(Dictionary`2 item)

This was actually a 'bug' in our unit test where we were setting up an AttributeValue in an image Dictionary incorrectly

as new DynamoDBEvent.AttributeValue { N = null } or new DynamoDBEvent.AttributeValue()

So like:

public static void Main()
{
	var image = CreateBasicImage();
	var json = image.ToJson();
	Console.WriteLine(json);
}

private static Dictionary<string, DynamoDBEvent.AttributeValue> CreateBasicImage() =>
        new()
        {
            { "PublishThing", new DynamoDBEvent.AttributeValue { N = null } }, // Or new DynamoDBEvent.AttributeValue()
	    { "SomeDefaultThing", new DynamoDBEvent.AttributeValue { S = "foo" } }
        };

See https://dotnetfiddle.net/Fomy9m

So reporting as this was a change in behaviour for us versus FromAttributeMap, and also reporting incase you want to handle this in some way in your WriteJsonValue, rather than allowing STJ to fail. But we've fixed our test and are good.

[ashovlin]

1. @psdanielhosseini - ah, I see. So you're ultimately trying to go from a JSON string to Dictionary<string, DynamoDBEvent.AttributeValue>. But this is no longer possible since the FromJson -> Document -> ToAttributeValues path produces the SDK's AttributeValue, not the newly defined DynamoDBEvent.AttributeValue in the event package.

   I'll take this back to the team for prioritization, we were initially focused on ToJson since that seemed more relevant to production code, but I understand now how this is problematic for tests.

2. @adam-knights - that was inadvertent, thanks for the report. I'll take a look at better handling the null case.

[ashovlin]

@adam-knights - we just released Amazon.Lambda.DynamoDBEvents v3.1.1, which should handle the "empty" case for AttributeValue when serializing to JSON.

[psdanielhosseini]

@ashovlin - Do you have any update regarding the first point?

[Dreamescaper]

@ashovlin

So you're ultimately trying to go from a JSON string to Dictionary<string, DynamoDBEvent.AttributeValue>.

Any workaround using AWSSDK.DynamoDBv2 would be fine, e.g. JSON -> DynamoDBv2.Document -> Dictionary<string, DynamoDBEvent.AttributeValue>.

I've tried to get a "DynamoDB" JSON from Document, so I could deserialize it to Dictionary<string, DynamoDBEvent.AttributeValue>. Unfortunately, I haven't found an easy way to do that.

[ashovlin]

@psdanielhosseini / @Dreamescaper - I separated this request over to #1700. I'll leave this issue #1657 focused on going from the 3.0.0+ DynamoDBEvent to the SDK types, and then #1700 for the opposite direction. We don't have any work started to address this yet, but will review with the team.

[jevvo-trimble]

Hello @ashovlin,

I'm looking for a way of converting Dictionary<string, Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.AttributeValue> to Dictionary<string, Amazon.DynamoDBv2.Model.AttributeValue> or Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.AttributeValue to Amazon.DynamoDBv2.Model.AttributeValue at least.

Do you provide an "out of the box" solution? I could see something similar in
aws-lambda-java-libs

Kind regards

[chrischappell-rgare]

I have just run into this issue with mapping from DynamoDBEvent models as well. In my case I am mapping the new image to an object persistence class that has a property using an IPropertyConverter to store the value in a binary field with gzip compression. The ToJson() method doesn't work because that will convert the binary property into a string. The string is base 64 encoded and contains a byte order mark that the converter does not normally encounter. DynamoDBEntry doesn't seem to have any method for exposing the attribute type either. AsByteArray or AsMemoryStream throw InvalidOperationException if the conversion is not supported.

[starkcolin]

Any update on this? An implementation of ToAttributeMap() is all I need to be able to update to the newest version

[andrew-malone-cko]

@starkcolin I was stuck with the same issue and got around it for now by creating extension methods for mapping from the lambda dynamodb event type to the model type. This seems to work but it would be nice for this to be something built into the nuget package for mapping from x to y and back. Hope some of this helps someone else.

        /// <summary>
        /// Converting Dictionary of Dynamo model to lambda events
        /// </summary>
        /// <param name="source"></param>
        /// <returns></returns>
        public static Dictionary<string, Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.AttributeValue>
            ToLambdaAttributeValue(
                this Dictionary<string, Amazon.DynamoDBv2.Model.AttributeValue> source)
        {
            if (source == null)
            {
                return null!;
            }

            var target = new Dictionary<string, Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.AttributeValue>();

            foreach (var kvp in source)
            {
                target[kvp.Key] = kvp.Value.ToLambdaAttributeValue();
            }

            return target;
        }

        /// <summary>
        /// DynamoDBv2 AttributeValue to Lambda AttributeValue
        /// </summary>
        /// <param name="source"></param>
        /// <returns></returns>
        public static Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.AttributeValue ToLambdaAttributeValue(
            this Amazon.DynamoDBv2.Model.AttributeValue source)
        {
            if (source == null)
            {
                return null!;
            }

            var attribute = new Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.AttributeValue
            {
                S = source.S,
                N = source.N,
                B = source.B,
                SS = source.SS,
                NS = source.NS,
                BS = source.BS,
                M = source.M?.ToLambdaAttributeValue(),
                L = source.L?.Select(attr => attr?.ToLambdaAttributeValue()).ToList(),
                NULL = source.NULL,
                BOOL = source.BOOL
            };
            return attribute;
        }

        public static Dictionary<string, AttributeValue> ToDynamoDBv2AttributeValue(
            this Dictionary<string,  Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.AttributeValue> source)
        {
            if (source == null)
            {
                return null!;
            }

            var target = new Dictionary<string, AttributeValue>();

            foreach (var kvp in source)
            {
                target[kvp.Key] = kvp.Value.ToDynamoDBv2AttributeValue();
            }

            return target;
        }

        // Convert Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.AttributeValue to Amazon.DynamoDBv2.Model.AttributeValue
        public static AttributeValue ToDynamoDBv2AttributeValue(
            this  Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.AttributeValue source)
        {
            if (source == null)
            {
                return null!;
            }

            var at =  new AttributeValue
            {
                S = source.S,
                N = source.N,
                B = source.B,
                SS = source.SS,
                NS = source.NS,
                BS = source.BS,
                M = source.M?.ToDynamoDBv2AttributeValue(),
                L = source.L?.Select(attr => attr.ToDynamoDBv2AttributeValue()).ToList()
            };

            return at;
        }

        public static Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.StreamRecord ToDynamoDBEventStreamRecord(
            this Amazon.DynamoDBv2.Model.StreamRecord record)
        {
            return new Amazon.Lambda.DynamoDBEvents.DynamoDBEvent.StreamRecord()
            {
                Keys = record.Keys.ToLambdaAttributeValue()
            };
        }
        
        public static DynamoDBEvent GenerateDynamoDbEvent(this IEnumerable<Dictionary<string, DynamoDBEvent.AttributeValue>> attributeMaps)
        {
            return new DynamoDBEvent { Records = attributeMaps.Select(GenerateDynamoDbRecord).ToList() };
        }
        
        public static DynamoDBEvent.DynamodbStreamRecord GenerateDynamoDbRecord(Dictionary<string, DynamoDBEvent.AttributeValue> attributeMap)
        {
            return new DynamoDBEvent.DynamodbStreamRecord
            {
                EventSource = "aws:dynamodb",
                EventName = "INSERT",
                Dynamodb = new DynamoDBEvent.StreamRecord
                {
                    NewImage = attributeMap,
                    Keys = new Dictionary<string, DynamoDBEvent.AttributeValue>()
                    {
                        { "payment_id", new DynamoDBEvent.AttributeValue { S = attributeMap["payment_id"].S } }
                    }
                },
            };
        }

[rami-hamati-ttl]

All these options depend on the Context. Is there a plan to add support for a deserializer which doesn't use the context?

[michelemottini]

To convert from event to document the suggested:

    var json = record.Dynamodb.NewImage.ToJson();

    var document = Document.FromJson(json);

does not really work (?) - for example if NewImage contains a binary attribute the resulting JSON is

{
    "theBinaryAttribute": "FdagZAT9+k2bZplXbijX8w=="
   . . . 
}

that gets converted to a document that has that attribute as a string, not binary

Also going from a dictionary to a JSON string and then parse that seems pretty inefficient?

Using DynamoDBEvents 3.1.1