Injecting Behaviors into Mimic

Behaviors are a way to provide deterministic error injection into Mimic functionality. For instance, mimic will always successfully create a server, immediately. But perhaps you want to see how your service or client handles a server build timing out. Or a 400 response.

Contents:

Goals - what the behaviors API was designed to do
Concepts - terminology and ideas used
Behavior storage - object models used to store behaviors - a little bit of code
Implementation guide - how to implement injected behaviors and CRUD endpoints to manipulate them

Goals

Provide an out-of-band method to specify behaviors, so the actual payload to the system under test does not need to change based on whether it is being run against Mimic or a production system. In other words, your test harness should be able to inject behaviors, and your application does not need to know.
Provide a consistent REST interface for injecting behaviors, so that the many different types of behavior which may have to be injected for many different types of events will follow a similar pattern, and developers can pick up new ones quickly.
Support different behaviors for different functionality at the same time. For instance, we want to be able to cause server creation 500 failures at the same time as identity authentication 403 failures.
Support different behaviors for the same functionality at the same time. For instance, we want to be able to cause server creation 500 failures for some servers, and also server creation 400 failures for some other servers, and also default 202 success for other servers, depending on certain criteria.

Concepts

These explain some terminology and how behaviors are modeled in mimic. Some of these also correspond to objects (in the Python sense) in Mimic, and some only partially correspond to objects in Mimic.

Note that this does not actually cover all the behavior-related objects (in the Python sense) in Mimic. Please see the behavior storage section for that.

Event

When events occur in mimic, for example:

a tenant authenticates against identity
a client requests that a server be created
a client requests that a server be deleted
a client requests that a node is added to a load balancer
a server finishes building within mimic

Clients may wish to customize the way that Mimic reacts to these things. Only tenant authentication and server creation are presently implemented this way, but the long-term plan is to make everything behave like this.

A class of event, like "server creation", is described by a mimic.model.behaviors.EventDescription. Such a class would be represented by a global instance of EventDescription - it'd be declared as: server_creation = EventDescription(). This server_creation object describes how server creation works, not any particular server being created.

An EventDescription has a default behavior, which is the behavior that gets used when there are no injected behaviors specified. This would be successfully creating a server, for instance, in the case of server creation.

Criterion

A pair of (attribute, predicate) to match against to determine whether or not to apply a behavior to an event. By predicate we just mean a callable that returns True or False, given the value for the attribute. For example:

The server name in the JSON passed to the create server endpoint - maybe we only want to apply the behavior if the server name matches "fail_test.*". The criterion would be (server_name, predicate that regexp-matches against "fail_test.*").
The metadata in the JSON passed to the create server endpoint. Maybe we only want to apply the behavior if certain metadata is passed in. So the criterion would be (metadata, predicate that matches the metadata dictionary against a predefined dictionary).

mimic.model.behaviors.Criterion

Criterion takes a name and a predicate. This API is currently in flux - see this issue for more details.

Criteria

A set of criterion to match against to determine whether or not to apply a behavior to an event. All of them have to apply in order for a behavior to apply. Mimic actually always applies criteria (as opposed to a single criterion)

In Mimic, this corresponds to a list of criterion, and no other separate object.

Behavior

A way in which Mimic behaves. This is just a function that does something. Injected behaviors are functions that do something else other than the normal thing.

In Mimic, the normal thing looks like:

@event.declare_default_behavior
def do_my_normal_thing(*args, **kwargs):
    ...

(see the implementation guide for default behaviors). The injected behavior looks like:

@event.declare_behavior_creator("other_behavior_name")
def create_other_behavior_callable(parameters):
    ...
    def do_other_behavior(*args, **kwargs):
        # do something with both the parameters and the args and kwargs
    return do_other_behavior

Note that the *args and **kwargs in the default behavior and in the callable returned by the injected behavior code, while arbitrary, should match.

(see the implementation guide for injected behaviors for more information)

Behavior storage

This section describes the behavior storage mechanism in Mimic.

`EventDescription`

This object describes an event. It describes the set of all the possible behaviors that may apply to the event, and the set of unordered criteria which should be evaluated before a behavior is applied. There is no relationship in an EventDescription between any behavior or any criteria.

That was the more handwavy concept explanation. In reality, an EventDescription contains a set of behavior creators - more general functions that return another function, which is the actual behavior: the inner function will be called when applying a behavior (please see this part of the implementation guide for more details).

`BehaviorRegistryCollection`

This is the is the top-level store, and contains behaviors for multiple events. It is up to the implementer of the control plane where an instance of this should be stored.

When retrieving a BehaviorRegistry from a BehaviorRegistryCollection, if one does not exist for a particular event, one is created and then returned.

`BehaviorRegistry`

This contains a instance of a mimic.model.behaviors.EventDescription, which contains an ordered bijection (a mapping and its reverse) between criteria to behaviors.

A mimic.model.behaviors.BehaviorRegistry contains the currently specified criteria/behaviors which should apply when the event is triggered.

This distinct from the behaviors and criteria contained by mimic.model.behaviors.EventDescription, which are all the possible behavior creators and criteria, without any relationship between them.

A "behavior" may be in the EventDescription, but if it is not in the ordered bijection in the BehaviorRegistry, it will not be applied at all when the event is triggered.

The criteria in a BehaviorRegistry should be a strict subset of the criteria in its EventDescription, and its behaviors should be a strict subset of the behaviors in its EventDescription.

And actually, behaviors in a BehaviorRegistry are the callables that are returned by the behavior creators in EventDescription.

How it works:

Criteria, in BehaviorRegistry, look like:

[Criterion(name="criterion1", predicate=lambda ...),
 Criterion(name="criterion2", predicate=lambda ...),
 Criterion(name="criterion3", predicate=lambda ...)]

When BehaviorRegistry.behavior_for_attributes is called, it takes a dictionary whose keys are some subset of ("criterion1", "criterion2", "criterion3"), and whose values should be evaluatable by the predicates.

It iterates through that mapping/bijection, and the first behavior whose criteria matches the attribute dictionary will be returned.

Uniqueness is not enforced. There could be two behaviors with the same set of criteria, or even the same behavior with the same set of criteria twice. But the first one is always the one that gets returned.

Implementation Guide

For this guide, we will be using Nova server creation as an example. The exact code is in mimic.model.nova_objects, and the code provided here will be more pseudocode and somewhat handwavy.

Let's first describe the spec: how we want behavior injection to work:

The user to make a POST request to a behavior injection endpoint with the following JSON:
```
{
    "criteria": [
        {"server_name": "my_failure_name.*"},
        {"imageRef": "abcd.*"}
    ],
    "name": "fail",
    "parameters": {
        "code": 404,
        "message": "Stuff is broken, what"
    }
}
```
This means that they want the behavior named "fail" to be applied to any server created with a server name that matches the regex expression "my_failure_name.*" and also an image ID that matches "abcd.*".

The "fail" behavior apparently takes the parameter attributes "code" and "message".

Once they post, a 201 response will be returned containing the ID of the behavior that was just registered. It looks like:
```
{
    "id": "this-is-a-uuid-here"
}
```
Now, when the user creates a server, with the name "my_failure_name" and the image ID "abcdef", the creation will fail with a 404 status code and response body: "Stuff is broken, what".
The user issues a DELETE request to the same endpoint, plus the behavior ID, and to remove said behavior.
Now, when the user creates a server, even with the name "my_failure_name" and the image ID "abcdef", the creation will succeed instead of returning a 404.

This guide will accomplish the above in multiple steps:

The default behavior for a single event
Injected behaviors for the same event.
The behavior REST endpoints
Adding additional events

We suggest implementing the default behavior (1) in one PR, some injected behavior(s) and the REST endpoints in another PR (2-3), and additional events (4) in other PRs, whether you are adding a control plane to an existing plugin or writing a new plugin.

Assuming that you are modifying an existing plugin:

The default behavior wouldn't change any existing behavior, so no additional tests are needed.
When adding new behaviors, tests for those new behaviors can be added at the same time. The REST endpoint and tests are already templated and provided for you, and do not require very much code. And the templated tests require that there be at least 1 behavior in addition to the default behavior.

If you are providing a new plugin at the same time as the control plane, we'd suggest either:

Implementing the the plugin first, or at least the part you want to provide injection behavior for first, and then adding the control plane.
Implementing the default behavior from teh start, and only for one event you want to provide a control plane for. Provide tests for the default behavior for that single event. Implement the rest endpoints and additional behaviors in another PR, and then other events in later PRs.

Default Behavior

First, we define what happens normally, without any behavior injection.

Declare an event for which behaviors apply:
```
server_creation = EventDescription()
```
Define a default behavior for the event. This is just a decorated callable (the decorator makes use of the previously declared server_creation) which take arbitrary parameters - the parameters it requires is defined by the code which uses it (see the next step).
```
@server_creation.declare_default_behavior
def default_create_behavior(*args, **kwargs):
    # create server code
    # set response code to 202
    # return server JSON
```
The pseudocode may not be the actual behavior - maybe it just creates the server and returns a server, and the handler sets the response code and returns the server JSON isntead. But this is the general idea.
Create a BehaviorRegistryCollection somewhere in the plugin's code.

We recommend placing this in your regional session store (bottom level of the session store diagram here), so that there is one set of behaviors per plugin per region. You can also place it on your global plugin session store (second-to-last level of bottom level of the session store diagram here), if you want the behaviors to apply to all regions. Or if your plugin does not support regions.

The identity behavior collection, for instance, is stored on the global resource object the behavior being injected affects everything).
In the code that normally handles the event (such as creating a server), call the behavior function.
```
def handle_server_creation(..., behavior_registry_collection, ...):
    server_creation_behavior_registry = (
        behavior_registry_collection.registry_by_event(server_creation))
    behavior_to_apply = (
        server_creation_behavior_registry.behavior_for_attributes({}))
    return behavior_to_apply(*args, **kwargs)
```
This function accepts as a parameter the BehaviorRegistryCollection created in the previous step. It knows that the event it handles is server_creation. From that, it obtains a behavior to apply.

In this case, we pass behavior_for_attributes an empty attribute dictionary, because we have not defined any criteria yet or any other behaviors - this will get is the default behavior for now. This will change in the next section.

Note that the above implementation is just a suggestion. For example, if handle_server_creation is an instance method, as it is in Nova, behavior_registry_collection might be an instance attribute instead. Alternately, the function could accept or refer to just a BehaviorRegistry instead of a whole BehaviorRegistryCollection.

If you are modifying an existing plugin, this function probably previously looked like:
```
def handle_server_creation(...):
    # create server code
    # set response code to 202
    # return server JSON
```
Note that this looks exactly like the previous step's default_create_behavior. That's because the easiest way to create a default behavior is probably to move all the existing code to that function, unless you want to factor some code out for injected error behavior

Injected Behaviors

Note that this builds on the code from the previous section, particularly the event (server_creation) that was declared.

Declare one or more criterion for this event. Note that this API will probably change slightly, but right now, this looks something like:
```
@server_creation.declare_criterion("server_name")
def server_name_criterion(value):
    return Criterion(
        name="server_name",
        predicate=lambda name: re.compile(value).match(name))

@server_creation.declare_criterion("image_id")
def image_id_criterion(value):
    return Criterion(
        name="image_id",
        predicate=lambda _id: re.compile(value).match(_id))
```
This function, when given a value, will create a criterion that specifies that a server name should be equal that value.

The value and predicate can be anything your plugin specifies, so long as it's documented. For example, the value can be just a string, and the predicate checks for equality. The value can be a number, and the predicate can check the length of whatever gets passed in.
- what happens behind the scenes (not something you have to implement):
  
  Let's look at the the criteria part of the REST request body in the implementation spec above:
```
{
    "criteria": [
        {"server_name": "my_failure_name.*"},
        {"image_id": "abcd.*"}
    ],
    ...
}
```
  This will all be handled by the behavior models and REST endpoint template code (e.g. code in mimic.models.behaviors), but what will happen when that request is made, at least in regards to criteria, is that the criteria JSON will be turned into a list of 2 Criterion.
  
  mimic.models.behaviors will look up the declared functions by name, and call those functions to return Criterions. So it will find:
  - server_name_criterion because of "server_name", and call it with "my_failure_name.*", returning a
```
Criterion(
    name="server_name",
    predicate=(
        lambda name: re.compile("my_server_name.*").match(name)))
```
  - image_id_criterion because of "image_id", and call it with "abcd.*", returning a
```
Criterion(name="image_id",
          predicate=lambda _id: re.compile("abcd.*").match(_id))
```
  And mimic.models.behaviors will map the set of these two criteria to a behavior that will be created in the next step.
Declare a non-default behavior creator for this event:
```
@server_creation.declare_behavior_creator("fail")
def make_other_create_behavior(parameters):
    # any prep work that needs to be done with parameters
    def do_other_create_behavior(*args, **kwargs):
        # create server, or not, or do it wrong or delayed or whatever
        #    the behavior is supposed to be
        # set response code (maybe to a failure code)
        # return some string response
```
This decorated function, make_other_create_behavior, actually CREATES the function that will be called when this behavior is applied.

It must take a single argument: parameters, which is a dictionary containing whatever values your behavior needs to function. It's a dictionary because it will be passed via JSON when the REST request is made to use this behavior. You should document what this dictionary looks like - one example would be, if you wanted to pass some arbitrary HTTP status code and failure message:
```
 ```
 {
     "code": 404,
     "message": "This is the failure message"
 }
 ```
```
Note that the inner function's *args, **kwargs must be the exact same as those expected by the default behavior (see the next step for why).
- what happens behind the scenes when a user specifies a behavior (not something you have to implement):
  
  Let's look at the the behavior part of the REST request body in the implementation spec above:
```
{
    ...,
    "name": "fail",
    "parameters": {
        "code": 404,
        "message": "Stuff is broken, what"
    }
}
```
  This will all be handled by the behavior models and REST endpoint template code (e.g. code in mimic.models.behaviors), but what will happen when that request is made, at least in regards to behaviors, is that the behavior JSON will be turned into behavior and added to the BehaviorRegistry.
  
  mimic.models.behaviors will:
  1. Look up the declared function by name in the EventDescription. It will find make_other_create_behavior, because that was declared with the behavior name "fail", which is the name specified in the above JSON.
  2. Call the function with the provided parameters, something like:
```
make_other_create_behavior(
    {"code": 404, "message": "Stuff is broken, what"})
```
    Because {"code": 404, "message": "Stuff is broken, what"} was passed in as the "paramters" attribute.
  3. Store the resultant callable (remember that make_other_create_behavior returned a function?) in the BehaviorRegistry, mapped to the criteria that was created as per the previous step.
Modify the code that normally handles the event (such as creating a server). Remember that after we finished our default behavior implementation, it should look like this:
```
def handle_server_creation(..., behavior_registry_collection, ...):
    server_creation_behavior_registry = (
        behavior_registry_collection.registry_by_event(server_creation))
    behavior_to_apply = (
        server_creation_behavior_registry.behavior_for_attributes({}))
    return behavior_to_apply(*args, **kwargs)
```
We need to modify it to actually look up the parameters for the criteria, and apply it.
```
def handle_server_creation(..., behavior_registry_collection, server_json):
    criteria_attributes = {
        "server_name": server_json['name'],
        "image_id": server_json['imageRef']
    }
    server_creation_behavior_registry = (
        behavior_registry_collection.registry_by_event(server_creation))
    behavior_to_apply = (
        server_creation_behavior_registry.behavior_for_attributes(
            criteria_attributes))
    return behavior_to_apply(*args, **kwargs)
```
Note that the only difference is in which criteria are passed - the call to behavior_to_apply(*args, **kwargs) is the same, which is why the function returned by the behavior creator must have the same signature as the default behavior.
- what happens behind the scenes (not something you have to implement):
  
  When you call server_creation_behavior_registry.behavior_for_attributes that the mimic.model.behaviors.BehaviorRegistry code will iterate through its ordered bijection mappings one by one, find a behavior for which the criteria apply.
  
  For example, for each mapping:
  - If the criteria are:
```
[Criterion(name="server_name", predicate=lambda <1>...),
 Criterion(name="image_id", predicate=lambda <2>...)]
```
  - mimic.models.behaviors.BehaviorRegistry.behavior_for_attribute will evaluate:
    1. (lambda <1>)(criteria_attributes["server_name"])
    2. (lambda <2>)(criteria_attributes["image_id"])
  - if both are True, then mimic.models.behaviors.BehaviorRegistry.behavior_for_attribute will return the corresponding behavior callable mapped to those criteria, otherwise it moves on to the next behavior.
  If there are no more mappings left to iterate through, mimic.models.behaviors.BehaviorRegistry.behavior_for_attribute will return the EventDescription's default behavior.
  
  Then, it's up to you as the plugin author to correctly call the returned behavior, using whatever arguments and keyword arguments you specified.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly