Skip to content

Latest commit

 

History

History
288 lines (235 loc) · 17.9 KB

EiffelTestExecutionRecipeCollectionCreatedEvent.md

File metadata and controls

288 lines (235 loc) · 17.9 KB

EiffelTestExecutionRecipeCollectionCreatedEvent (TERCC)

The EiffelTestExecutionRecipeCollectionCreatedEvent declares that a collection of test execution recipes has been created. In order to clarify what that means, several concepts need to be explained.

Just as Eiffel is an opinionated protocol, EiffelTestExecutionRecipeCollectionCreatedEvent is an opinionated event, and to a certain extent that opinion goes against the grain of traditional test management. This event assumes that activity configurations and test scope definitions are two separate things. To exemplify, when one's CI server triggers the Acceptance Test activity in the pipeline, there is nothing in that activity that says which tests to execute where or with what parameters: that is a separate concern. Instead, it will query a test selection service for that information. This information is encapsulated by the EiffelTestExecutionRecipeCollectionCreatedEvent, which contains all the information needed to configure and execute the tests.

How the test selection service generates the recipe collection is, from the point of view of the Eiffel protocol, irrelevant. It may very well be from a statically defined list of test cases, or from an elaborate test selection algorithm weighing together a host of factors to determine the optimal set of test cases to execute at any particular time, or a combination of the two.

The data object consists of two main parts. data.selectionStrategy identifies the strategy used to select the test cases and generate the recipe collection, while data.batches or data.batchesUri contain or reference, respectively, the recipes. The recipes are grouped in batches, which are used to control the order of execution of test cases. Every batch has a priority to let the test executor order them in sequence, but within each batch no assumptions are made as to the execution order the test cases. This way the recipe collection can either allow the executor a high degree of freedom in scheduling the test executions, and/or prescribe the exact sequential order in which they must be executed. Each event SHALL include one and only one of data.batches and data.batchesUri.

Finally, each recipe (data.batches.recipes) consists of two parts: the test case to execute, and the constraints of that execution. The EiffelTestExecutionRecipeCollectionCreatedEvent does not control the syntax of these constraints, as the nature of such constraints are highly dependent on technology domain and test execution framework. That being said, there are three questions that typically need to be answered: what is the item under test, in what kind of environment is it to be tested, and what are the test parameters? Note the distinction between test case and test execution: it is perfectly legal for a single test case to appear multiple times within the same EiffelTestExecutionRecipeCollectionCreatedEvent, but (presumably) with different constraints.

Data Members

data.selectionStrategy

Type: Object
Required: Yes
Description: The strategy used to select the test cases and generate the recipe collection.

data.selectionStrategy.tracker

Type: String
Required: No
Description: The name of the selection strategy that generated the test execution recipe collection.

data.selectionStrategy.id

Type: String
Required: Yes
Description: The unique identity of the selection strategy that generated the test execution recipe collection.

data.selectionStrategy.uri

Type: String
Required: No
Description: The URI at which the the selection strategy that generated the test execution recipe collection can be retrieved.

data.batchesUri

Type: String
Required: No
Description: A URI at which at which the array of test execution batches can be fetched. The format of the document at this URI SHALL be on the format prescribed by data.batches (i.e. [ { "name": "Batch 1", ...}, {...}]). Each event SHALL include one and only one of data.batches and data.batchesUri.

data.batches

Type: Object[]
Required: No
Description: A list of batches of recipes. Each event SHALL include one and only one of data.batches and data.batchesUri. In the interest of keeping message sizes small, it is recommended to use data.batches only for limited numbers of test cases (on the order of ten executions). For larger numbers of executions, data.batchesUri SHOULD be used instead.

data.batches.name

Type: String
Required: No
Description: The name of the recipe batch.

data.batches.priority

Type: Integer
Required: Yes
Description: The execution priority of the batch, as compared to other batches in the collection. A lower value SHALL be interpreted as a higher priority.

data.batches.recipes

Type: Object[]
Required: Yes
Description: The collection of test execution recipes within the batch.

data.batches.recipes.id

Type: String
Required: Yes
Description: A UUID identifying the unique execution. Note that this is different from the id of a test case, in two ways. First, a test case is a (presumably) persistnent entity, whereas an execution is transient in nature. Second, a test case may be executed any number of times in any given recipe collection.

data.batches.recipes.testCase

Type: Object
Required: Yes
Description: The test case to be executed in this execution recipe.

data.batches.recipes.testCase.tracker

Type: String
Required: No
Description: The name of the test case tracker - typically a test management system.

data.batches.recipes.testCase.id

Type: String
Required: Yes
Description: The unique identity of the test case.

data.batches.recipes.testCase.uri

Type: String
Required: No
Description: A location where a description of the test case can be retrieved.

data.batches.recipes.constraints

Type: Object[]
Required: No
Description: Any constraints of the execution. The nature of such constraints is highly dependent on technology domain and test execution framework. Consequently, there are no pre-defined or required constraints. Instead, this property is a list of key-value pairs on the same format as data.customData. That being said, there are three questions that typically need to be answered: what is the item under test, in what kind of environment is it to be tested, and what are the test parameters?

data.batches.recipes.constraints.key

Type: String
Required: Yes
Description: The key name of constraint.

data.batches.recipes.constraints.value

Type: Any
Required: Yes
Description: The value of the constraint.

data.batches.dependencies

Type: Object[]
Required: No
Description: A list of test case execution dependencies. Ideally, test cases are atomic and can be executed in isolation. In cases where a test case assumes that another test case has been executed previously in the same environment, however, this property can be used to specify that. It serves as an instruction to the test executor to place two executions subsequent to one another in the same environment.

data.batches.dependencies.dependency

Type: String
Required: Yes
Description: The UUID of the dependency execution (data.batches.recipes.id), i.e. the execution that shall be performed prior to that of the dependent.

data.batches.dependencies.dependent

Type: String
Required: Yes
Description: The UUID of the dependent execution (data.batches.recipes.id), i.e. the execution that shall be performed only after that of the dependency.

Links

CAUSE

Required: No
Legal targets: Any
Multiple allowed: Yes
Description: Identifies a cause of the event occurring. SHOULD not be used in conjunction with CONTEXT: individual events providing CAUSE within a larger context gives rise to ambiguity. It is instead recommended to let the root event of the context declare CAUSE.

CONTEXT

Required: No
Legal targets: EiffelActivityTriggeredEvent, EiffelTestSuiteStartedEvent
Multiple allowed: No
Description: Identifies the activity or test suite of which this event constitutes a part.

FLOW_CONTEXT

Required: No
Legal targets: EiffelFlowContextDefinedEvent
Multiple allowed: Yes
Description: Identifies the flow context of the event: which is the continuous integration and delivery flow in which this occurred – e.g. which product, project, track or version this is applicable to.

Meta Members

meta.id

Type: String
Format: UUID
Required: Yes
Description: The unique identity of the event, generated at event creation.

meta.type

Type: String
Format: An event type name
Required: Yes
Description: The type of event. This field is required by the recipient of the event, as each event type has a specific meaning and a specific set of members in the data and links objects.

meta.version

Type: String
Format: Semantic Versioning 2.0.0
Required: Yes
Description: The version of the event type. This field is required by the recipient of the event to interpret the contents. Please see Versioning for more information.

meta.time

Type: Integer
Format: UNIX Epoch time, in milliseconds.
Required: Yes
Description: The event creation timestamp.

meta.tags

Type: String[]
Format: Free text
Required: No
Description: Any tags or keywords associated with the events, for searchability purposes.

meta.source

Type: Object
Format:
Required: No
Description: A description of the source of the event. This object is primarily for traceability purposes, and while optional, some form of identification of the source is HIGHLY RECOMMENDED. It offers multiple methods of identifying the source of the event, techniques which may be select from based on the technology domain and needs in any particular use case.

meta.source.domainId

Type: String
Format: Free text
Required: No
Description: Identifies the domain that produced an event. A domain is an infrastructure topological concept, which may or may not corresponds to an organization or product structures. A good example would be Java packages notation, ex. com.mycompany.product.component or mycompany.site.division. Also, keep in mind that all names are more or less prone to change. Particularly, it is recommended to avoid organizational names or site names, as organizations tend to be volatile and development is easily relocated. Relatively speaking, product and component names tend to be more stable and are therefore encouraged, while code names may be an option. You need to decide what is the most sensible option in your case.

meta.source.host

Type: String
Format: Hostname
Required: No
Description: The hostname of the event sender.

meta.source.name

Type: String
Format: Free text
Required: No
Description: The name of the event sender.

meta.source.serializer

Type: String
Format: purl specification
Required: No
Description: The identity of the serializer software used to construct the event, in purl format.

meta.source.uri

Type: String
Format: URI
Required: No
Description: The URI of, related to or describing the event sender.

meta.security

Type: Object
Format:
Required: No
Description: An optional object for enclosing security related information, particularly supporting data integrity. See Security for further information.

meta.security.authorIdentity

Type: String
Format: Distinguished Name
Required: Yes
Description: The identity of the author of the event. This property is intended to enable the recipient to identify the author of the event contents and/or look up the appropriate public key for decrypting the meta.security.integrityProtection.signature value and thereby verifying author identity and data integrity.

meta.security.integrityProtection

Type: Object
Format:
Required: No
Description: An optional object for enabling information integrity protection via cryptographic signing. To generate a correct meta.security.integrityProtection object:

  1. Generate the entire event, but with the meta.security.integrityProtection.signature value set to an empty string.
  2. Serialize the event on Canonical JSON Form.
  3. Generate the signature using the meta.security.integrityProtection.alg algorithm.
  4. Set the meta.security.integrityProtection.signature value to the resulting signature while maintaining Canonical JSON Form. To verify the integrity of the event, the consumer then resets meta.security.integrityProtection.signature to an empty string and ensures Canonical JSON Form before verifying the signature.
meta.security.integrityProtection.alg

Type: String
Format: A valid JWA RFC 7518 alg parameter value, excluding "none"
Required: Yes
Description: The cryptographic algorithm used to digitally sign the event. If no signing is performed, the meta.security.integrityProtection SHALL be omitted rather than setting meta.security.integrityProtection.alg to "none".

meta.security.integrityProtection.signature

Type: String
Format:
Required: Yes
Description: The signature produced by the signing algorithm.

meta.security.integrityProtection.publicKey

Type: String
Format:
Required: No
Description: The producer of the event may include the relevant public key for convenience, rather than relying a separate key distribution mechanism. Note that this property, along with the rest of the event, is encompassed by the integrity protection offered via meta.security.integrityProtection.

meta.security.sequenceProtection

Type: Object[]
Format:
Required: No
Description: An optional object for enabling verification of intact event sequences in a distributed environment, thereby protecting against data loss, race conditions and replay attacks. It allows event publishers to state the order in which they produce a certain set of events. In other words, it cannot provide any global guarantees as to event sequencing, but rather per-publisher guarantees. Every object in the array represents a named sequence of which this event forms a part. For every event including a given named sequence, the publisher SHALL increment meta.security.sequenceProtection.position by 1. The first event produced in a given named sequence SHALL numbered 1.

meta.security.sequenceProtection.sequenceName

Type: String
Format:
Required: Yes
Description: The name of the sequence. There MUST not be two identical meta.security.sequenceProtection.sequenceName values in the same event.

meta.security.sequenceProtection.position

Type: Integer
Format:
Required: Yes
Description: The number of the event within the named sequence.

Version History

Version Introduced in Changes
4.0.0 edition-agen Improved information integrity protection
3.0.0 dc5ec6f Introduced purl identifiers instead of GAVs (see Issue 182)
2.1.0 edition-toulouse Multiple links of type FLOW_CONTEXT allowed.
2.0.0 f92e001 Changed syntax of data.batches.recipes.constraints from an uncontrolled object to a list of key-value pairs to comply with design guidelines.
1.0.0 edition-bordeaux Initial version.

Examples