About the "at-least-onces" message delivery guarantee for Activity Functions (DurableFunctions)

[jonathanharr]

This question originated from an issue I opened and closed where I got some help from @davidmrdavid. After an email I sent regarding another question I had about the "at-least-once" message delivery guarantee we aggreed that it would be best if this was posted in the forums so that others can make use of the knowledge shared in the answer.

How does the guarantee work? Let's say I call the same activity function several times, just how does Azure or the Durable Function know that the activity has or has not been made?

Example:

Here is how the DurableOrchestrationContext calls the function:

        for (int i = 0; i < numbers.Count; i++)
        {
            outputs.Add(await context.CallActivityAsync<string>("PrintNumber", numbers[i].ToString()));
        }

And here is the function itself:

    [FunctionName("PrintNumber")]
    public static async Task<string> SayHello([ActivityTrigger] string number, ILogger log)
    {
        log.LogInformation($"Printed number: {number}.");
        return $"Printed {number}!";
    }

This is how the function calls are stored inside of Azure Storage:

As you can see in the above image, the same TaskScheduled EventType has been raised for the same name of the Activity Function 5 times. Does the replay functionality count the amount of EventId's to make sure it has reached the number it is supposed to reach or just how does Azure know that it has scheduled the 3rd call to the Activity Functiion named "PrintNumber"?

So my question is, what it is it that the durable extension framework does to make sure that it has executed call 0, 1, 2, 3, 4 to PrintNumber with the value in the array?

I also have a side question, if the Activity Function times out due to it reaching the max limit.. Will the Durable Function just keep rescheduling it over & over again or does it have a limit for retries? Or does the Durable Function also time out after a while?

[cgillum]

what it is it that the durable extension framework does to make sure that it has executed call 0, 1, 2, 3, 4 to PrintNumber with the value in the array?

The framework uses sequence numbers to know whether an activity has been scheduled. These numbers always start from 0 and are incremented by 1 for each new task that is scheduled. When processing the history, the framework checks to see if a call to CallActivityAsync has a corresponding TaskScheduledEvent already. If it doesn't then we know it hasn't been scheduled yet. If it does, then we know it's already been scheduled and we're just waiting for a response. There is also a TaskCompletedEvent that corresponds to every TaskScheduledEvent that includes a matching sequence number. This is used to know whether a particular task has completed. These sequence numbers are found in the EventId and TaskScheduledEventId properties that you can see in Azure Storage (more info on these history events can be found here). Only basic validation is done to determine whether the code matches the history (i.e. checking the name of the activity being called), but the primary mechanism is the sequence numbers.

I also have a side question, if the Activity Function times out due to it reaching the max limit.. Will the Durable Function just keep rescheduling it over & over again or does it have a limit for retries? Or does the Durable Function also time out after a while?

Today, we retry forever. However, we're looking into adding a Poison message feature that will help in cases where retrying forever is not desirable.

[jonathanharr]

Thanks for your reply @cgillum
So to clarify, if I have an ActivityFunction that times out, my Durable Function will reschedule it. It will keep rescheduling it indefinitely if the ActivityFunction times out each time?

[cgillum]

Yes, until we implement poison message handling, it's possible that you may end up with an infinite retry loop. This isn't 100% guaranteed, however, since the exact recycle behavior of the Functions host is not always consistent (there can be races). In some cases, you may observe that the activity fails with a timeout exception. This behavior isn't guaranteed, however.