Skip to content

Latest commit

 

History

History
177 lines (137 loc) · 8.43 KB

function_validation.md

File metadata and controls

177 lines (137 loc) · 8.43 KB
Raw

Enhancing Robustness: Using Pydantic Validation Decorator with ActionWeaver

In this post, we are going to showcase how we combine Pydantic validation decorator with ActionWeaver to create robust and self healing LLM function-calling applications.

  • ActionWeaver simplifies the development of LLM applications by providing straightforward tools for structured data parsing, function dispatching and orchestration.
  • Pydantic is the most widely used data validation library for Python. Its @validate_call decorator enables the validation of function arguments based on the function's annotations before the function is executed.

By the end of this blog post, you will learn how to use Pydantic validation decorator with ActionWeaver to make your function calling more robust. For the complete working notebook, please refer to here.

Create OpenAI client, wrap with ActionWeaver.

ActionWeaver wrapper will manage the function calling loop, which includes passing function descriptions, executing functions, and handling exceptions.

from openai import OpenAI
from actionweaver.llms import wrap

llm = wrap(OpenAI())

Define Pydantic models with field validation

Next, let's define some Pydantic models used in the function calling. Within the Pydantic model UserModel, we've specified certain field_validators to ensure that both the name and phone_number adhere to specific formats.

import re
from pydantic import BaseModel, ValidationError, validator, PrivateAttr

class UserModel(BaseModel):
    _uid: UUID = PrivateAttr(default_factory=uuid4)
    _created_at: datetime = PrivateAttr(default_factory=datetime.now)
    name: str
    phone_number: str

    @field_validator('name')
    @classmethod
    def validate_name(cls, v: str) -> str:
        # Split the input string into first and last names
        names = v.split()
        
        # Check if both first and last names are present
        if len(names) != 2:
            raise ValueError('Name must contain a first name and a last name separated by a space')
        
        # Check if the formatted name is not in uppercase
        if v != v.upper():
            raise ValueError('Name must be in uppercase')
        
        return v

    @field_validator('phone_number')
    @classmethod
    def validate_phone_number(cls, v: str) -> str:
        # Define a regular expression pattern for a phone number with country code
        pattern = r'^\+\d{1,3}\s*\(\d{3}\)\s*\d{3}-\d{4}$'  # Example: +1 (XXX) XXX-XXXX

        # Check if the phone number matches the pattern
        if not re.match(pattern, v):
            raise ValueError('phone number must be in the format +1 (XXX) XXX-XXXX')
        return v

Create an Action from function

Now, let's use ActionWeaver action decorator to create an action ingest_user_info, which accepts list of Pydantic model UserModel as arguments.

Note: An action represents a tool that can be used by LLM. Each action comprises two main elements: a Pydantic model that is generated to facilitate structured prompting, and a conventional Python function. ActionWeaver will treat the function docstring as function description. Please refer to the ActionWeaver documentation for more information and specifics.

We want the LLM to extract user information from natural language and invoke ingest_user_info, which will then store the validated user information into user_db.

A validate_call decorator is added to apply validation to the arguments.

from pydantic import validate_call

user_db = []

@action(name="SaveUserInfo", stop=True)
@validate_call
def ingest_user_info(users: List[UserModel]):
    """Save user info to database"""
    user_db.append(users)
    return "success"

Handling Exception

Now what if the LLM invoke the function with invalid arguments?

Fortunately, ActionWeaver offers an ExceptionHandler feature. This handler allows users to define how to respond to exceptions within the function calling loop.

For example, we can create a handler that takes the exception message as input for the LLM and permits a maximum number of retries. This setup enables the LLM to respond to error messages effectively.

from actionweaver.llms import ExceptionHandler, ExceptionAction, ChatLoopInfo, Continue, Return

class ExceptionRetryHandler(ExceptionHandler):
    def __init__(self, max_retry=2):
        self.max_retry = max_retry

    def handle_exception(self, e: Exception, info: ChatLoopInfo) -> ExceptionAction:
        if self.max_retry:
            self.max_retry -= 1
            
            print(f"\nRetrying. Retries left: {self.max_retry}")
            print(f"Exception raised: {type(e).__name__}: {str(e)}")
            
            response = info.context['response']
            messages = info.context['messages']
            messages.append(
                    {
                        "role": "tool",
                        "tool_call_id": response.choices[0].message.tool_calls[0].id,
                        "name": response.choices[0].message.tool_calls[0].function.name,
                        "content": f"Exceptions raised: \n{e}",
                    }
                )

            return Continue(functions=info.context['tools'])
        raise e

Demo

Now, let's attempt to prompt the LLM to call ingest_user_info. As you'll notice, the input text below has a different format and may not pass the field validation.

Note: For details on how to force the LLM to execute function, please refer to ActionWeaver Document.

input = """                Name       Phone Number
0  Dr. Danielle King      (844)055-3780
1        John Miller  +1-268-920-5475x5
2    Michael Johnson  +1-758-232-6153x8
"""

messages = [
    {"role": "user", "content": input}
]

response = ingest_user_info.invoke(
    llm, 
    messages=messages,
    model=MODEL,
    stream=False, 
    temperature=1,
    exception_handler = ExceptionRetryHandler(3)
)

We expect the LLM to execute the ingest_user_info function, potentially encountering a ValidationError. If so, it should attempt a retry, ideally leading to the invocation of the function with properly formatted data.

Retrying. Retries left: 2
Exception raised: ValidationError: 6 validation errors for ingest_user_info
users.0.name
  Value error, Name must contain a first name and a last name separated by a space [type=value_error, input_value='Dr. Danielle King', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.0.phone_number
  Value error, phone number must be in the format +1 (XXX) XXX-XXXX [type=value_error, input_value='(844)055-3780', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.1.name
  Value error, Name must be in uppercase [type=value_error, input_value='John Miller', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.1.phone_number
  Value error, phone number must be in the format +1 (XXX) XXX-XXXX [type=value_error, input_value='+1-268-920-5475x5', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.2.name
  Value error, Name must be in uppercase [type=value_error, input_value='Michael Johnson', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error
users.2.phone_number
  Value error, phone number must be in the format +1 (XXX) XXX-XXXX [type=value_error, input_value='+1-758-232-6153x8', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/value_error

Eventually, the formatted data should be successfully inserted into the user_db.

[[UserModel(name='DANIELLE KING', phone_number='+1 (844) 055-3780'),
  UserModel(name='JOHN MILLER', phone_number='+1 (268) 920-5475'),
  UserModel(name='MICHAEL JOHNSON', phone_number='+1 (758) 232-6153')]]

For the complete working notebook, please refer to here.