ZachG update parametric documentation and add contributing doc

docs/edit/README.md

@@ -8,6 +8,10 @@ System tests allow developers define scenarios and ensure datadog libraries prod
 
 To make changes, you must be able to run tests locally. Instructions for running **end-to-end** tests can be found [here](https://github.com/DataDog/system-tests/blob/main/docs/execute/README.md#run-tests) and for **parametric**, [here](https://github.com/DataDog/system-tests/blob/main/docs/scenarios/parametric.md#running-the-tests).
 
+**Note**
+
+For information on contributing to specifically **parametric** tests, see [here](/docs/scenarios/parametric_contributing.md).
+
 **Callout**
 
 You'll commonly need to run unmerged changes to your library against system tests (e.g. to ensure the feature is up to spec). Instructions for testing against unmerged changes can be found in [enable-test.md](./enable-test.md).

docs/edit/add-new-test.md

@@ -1,4 +1,4 @@
-Whether it's adding a new test or modifying an existing test, a moderate amount of effort will be required. The instructions below cater to end-to-end tests, refer to [placeholder] (TODO: LINK to parametric_contributing.md) for parametric-specific instructions.
+Whether it's adding a new test or modifying an existing test, a moderate amount of effort will be required. The instructions below cater to end-to-end tests, refer to [the parametric contributing doc](/docs/scenarios/parametric_contributing.md)for parametric-specific instructions.
 
 Once the changes are complete, post them in a PR.
 

docs/edit/features.md

@@ -1,4 +1,4 @@
-System tests are feature-oriented; put another way, tests certify which features are supported in each client library (and the supported library versions). Each test class must belong to a "feature", where "features" map to entries in the [Feature Parity Dashboard](https://feature-parity.us1.prod.dog/). We use the @features decorators to achieve this.
+System tests are feature-oriented; put another way, tests certify which features are supported in each client library (and the supported library versions). Each test class must belong to a "feature", where "features" map to entries in the [Feature Parity Dashboard](https://feature-parity.us1.prod.dog/). We use the `@features` decorators to achieve this.
 
 For example, you have a new feature called `Awesome feature`, which is part of a meta feature called `stuffs`. We add a new file called `tests/test_stuffs.py` and add a test class with some boilerplate code, and a basic test:
 
@@ -12,3 +12,28 @@ class Test_AwesomeFeature:
     def test_basic(self)
         assert P==NP
 ```
+
+Several key points:
+
+* Each new feature should be defined in [_features.py](/utils/_features.py). This consists of adding a feature in [Feature Parity Dashbaord](https://feature-parity.us1.prod.dog/), get the feature id and copying one of the already added features, changing the name and the feature id in the url, and the feature number. In this case we'd add
+
+```python
+
+    @staticmethod
+    def awesome_feature(test_object):
+        """
+        Awesome Feature for Awesomeness
+
+        https://feature-parity.us1.prod.dog/#/?feature=291
+        """
+        pytest.mark.features(feature_id=291)(test_object)
+        return test_object
+```
+
+* One class test one feature
+* One class can have several tests
+* Files can be nested (`tests/test_product/test_stuffs.py::Test_AwesomeFeature`), and how files are organized does not make any difference. Use you common sense, or ask on [slack](https://dd.enterprise.slack.com/archives/C025TJ4RZ8X).
+
+## Skip tests
+
+See [skip-tests.md](/docs/edit/skip-tests.md)

docs/scenarios/parametric.md

@@ -50,10 +50,10 @@ def test_datadog_spans(library_env, test_library, test_agent):
 ```
 
 - This test case runs against all the APM libraries and is parameterized with two different environments specifying two different values of the environment variable `DD_ENV`.
-- The test case creates a new span and sets a tag on it using the shared GRPC/HTTP interface.
-- The implementations of the GRPC/HTTP interface, by language, are in `utils/build/docker/<lang>/parametric`.
+- `test_library.dd_start_span` creates a new span using the shared HTTP interface.
+- The request is sent to a HTTP server by language. Implementations can be found in `utils/build/docker/<lang>/parametric`. More information in [Http Server Implementations](#http-server-implementations).
 - Data is flushed to the test agent after the with test_library block closes.
-- Data is retrieved using the `test_agent` fixture and asserted on.
+- Data (usually traces) are retrieved using the `test_agent` fixture and we assert that they look the way we'd expect.
 
 
 ## Usage
@@ -93,7 +93,7 @@ TEST_LIBRARY=dotnet ./run.sh PARAMETRIC -k test_metrics_
 Tests can be aborted using CTRL-C but note that containers maybe still be running and will have to be shut down.
 
 ### Running the tests for a custom tracer
-To run tests against custom tracers, refer to the [Binaries Documentation](../execute/binaries.md)
+To run tests against custom tracer builds, refer to the [Binaries Documentation](../execute/binaries.md)
 
 #### After Testing with a Custom Tracer:
 Note: Most of the ways to run system-tests with a custom tracer version involve modifying the binaries directory. Modifying the binaries will alter the tracer version used across your local computer. Once you're done testing with the custom tracer, ensure you **remove** it. For example for Python:
@@ -199,19 +199,23 @@ See the steps below in the HTTP section to run the Python server and view the sp
 ### Shared Interface
 
 To view the available HTTP endpoints , follow these steps:
+Note: These are based off of the Python tracer's http server which should be held as the standard example interface across implementations.
+
 
 1. `./utils/scripts/parametric/run_reference_http.sh`
 2. Navigate to http://localhost:8000/docs in your web browser to access the documentation.
 3. You can download the OpenAPI schema from http://localhost:8000/openapi.json. This schema can be imported into tools like [Postman](https://learning.postman.com/docs/integrations/available-integrations/working-with-openAPI/) or other API clients to facilitate development and testing.
 
+Not all endpoint implementations per language are up to spec with regards to their parameters and return values. To see these please reference the [feature parity board](https://feature-parity.us1.prod.dog/#/?runDateFilter=7d&feature=339)
+
 ### Architecture: How System-tests work
 
 Below is an overview of how the testing architecture is structured:
 
 - Shared Tests in Python: We write shared test cases using Python's pytest framework. These tests are designed to be generic and interact with the tracers through an HTTP interface.
-- HTTP Servers in Docker: For each language tracer, we build and run an HTTP server within a Docker container. These servers expose the required endpoints defined in the OpenAPI schema and handle the tracer-specific logic.
+- [HTTP Servers in Docker](#http-server-implementations): For each language tracer, we build and run an HTTP server within a Docker container. These servers expose the required endpoints defined in the OpenAPI schema and handle the tracer-specific logic.
 - [Test Agent](https://github.com/DataDog/dd-apm-test-agent/) in Docker: We start a test agent in a separate Docker container. This agent collects data (such as spans and traces) submitted by the HTTP servers. It serves as a centralized point for aggregating and accessing test data.
-- Test Execution: The Python test cases use an HTTP client to communicate with the servers. The servers generate data based on the interactions, which is then sent to the test agent. The tests can query the test agent to retrieve  data (usually traces) and perform assertions to verify correct behavior.
+- Test Execution: The Python test cases use a [HTTP client](/utils/parametric/_library_client.py) to communicate with the servers. The servers generate data based on the interactions, which is then sent to the test agent. The tests can query the test agent to retrieve data (often traces) and perform assertions to verify correct behavior.
 
 An example of how to get a span from the test agent:
 ```python
@@ -220,6 +224,23 @@ span = find_only_span(test_agent.wait_for_num_traces(1))
 
 This architecture allows us to ensure that all tracers conform to the same interface and behavior, making it easier to maintain consistency across different languages and implementations.
 
+#### Http Server Implementations
+
+The http server implementations for each tracer can be found at the following locations:
+*Note:* For some languages there is both an Otel and a Datadog server. This is simply to separate the available Otel endpoints from the available Datadog endpoints that can be hit by the client. If a language only has a single server, then both endpoints for Otel and Datadog exist there.
+
+* [Python](/utils/build/docker/python/parametric/apm_test_client/server.py)
+* [Ruby](utils/build/docker/ruby/parametric/server.rb)
+* [Php](utils/build/docker/php/parametric/server.php)
+* [Nodejs](utils/build/docker/nodejs/parametric/server.js)
+* [Java Datadog](utils/build/docker/java/parametric/src/main/java/com/datadoghq/trace/opentracing/controller/OpenTracingController.java)
+* [Java Otel](utils/build/docker/java/parametric/src/main/java/com/datadoghq/trace/opentelemetry/controller/OpenTelemetryController.java)
+* [Dotnet Datadog](utils/build/docker/dotnet/parametric/Endpoints/ApmTestApi.cs)
+* [Dotnet Otel](utils/build/docker/dotnet/parametric/Endpoints/ApmTestApiOtel.cs)
+* [Go Datadog](utils/build/docker/golang/parametric/main.go)
+* [Go Otel](utils/build/docker/golang/parametric/otel.go)
+
+
 ![image](https://github.com/user-attachments/assets/fc144fc1-95aa-4d50-97c5-cda8fdbcefef)
 
 <img width="869" alt="image" src="https://user-images.githubusercontent.com/6321485/182887064-e241d65c-5e29-451b-a8a8-e8d18328c083.png">

docs/scenarios/parametric_contributing.md

@@ -0,0 +1,68 @@
+# Contributing to Parametric System-tests
+
+Note: a more in-depth overview of parametric system-tests can be found in [parametric.md](parametric.md).
+
+**MUST:** Acquaint yourself with [how system tests work](parametric.md#architecture-how-system-tests-work) before proceeding.
+
+## Use cases
+
+Let's figure out if your feature is a good candidate to be tested with parametric system-tests.
+
+System-tests in general are great for assuring uniform behavior between different dd-trace repos (tracing, ASM, DI, profiling, etc.). There are two types of system-tests, [end-to-end](/docs/README.md) and [parametric](/docs/scenarios/parametric.md).
+
+Parametric tests in the Datadog system test repository validate the behavior of APM Client Libraries by interacting only with their public interfaces. These tests ensure the telemetry generated (spans, metrics, instrumentation telemetry) is consistent and accurate when libraries handle different input parameters (e.g., calling a Tracer's startSpan method with a specific type) and configurations (e.g., sampling rates, distributed tracing, remote settings). They run against web applications in languages like Java, Go, Python, PHP, Node.js, C++, and .NET, which expose endpoints simulating real-world library usage. The generated telemetry is sent to a Datadog agent, queried, and verified by system tests to confirm proper library functionality across scenarios.
+
+If your usage does not require different parameter values, then [end-to-end system-tests](/docs/README.md) should be used as they will achieve the same level of behavior uniformity verification and test the feature on real world use cases, catching more issues. End-to-end tests are also what should be used for verify behavior between tracer integrations.
+For more on the differences between end-to-end and parametric tests, see [here](/docs/scenarios/README.md#scenarios)
+System-tests are **not** for testing internal or niche library behavior. Unit tests are a better fit for that case.
+
+## Getting set up
+
+We usually add new system tests when validating a new feature. To begin, set up the system-tests repo to run with a version of the library that has already implemented the feature you'd like to test (published or on a branch).
+Follow [Binaries Documentation](../execute/binaries.md) for your particular tracer language to set this up.
+
+[Verify that you can run some (any) parametric tests with your custom tracer](parametric.md#running-the-tests). Make sure some pass — no need to run the whole suite (you can stop the tests from running with `ctrl+c`). If you have any issues, checkout the [debugging section](parametric.md#debugging) to troubleshoot.
+
+## Writing the tests
+
+Now that we're all setup with a working test suite and a tracer with the implemented feature, we can begin writing the new tests.
+
+**MUST:** If you haven't yet, please acquaint yourself with [how system tests work](parametric.md#architecture-how-system-tests-work) before proceeding and reference it throughout this section.
+
+Before writing a new test, check the [existing tests](/tests/parametric) to see if you can use the same methods or endpoints for similar scenarios; in many cases, new endpoints do not need to be added.
+
+For a list of client methods that already exist, refer to `class APMLibrary` in the [_library_client.py](/utils/parametric/_library_client.py). If you're wondering what the methods do, you can take at look at the respective endpoints they're calling in that same file in `class APMLibraryClient`.
+
+The endpoints (where the actual tracer code runs) are defined in the Http Server implementations per tracer [listed here](parametric.md#http-server-implementations). Click on the one for your language to take a look at the endpoints. In some cases you may need to just slightly modify an endpoint rather than add a new one.
+
+### If you need to add additional endpoints to test your new feature
+
+*Note:* please refer to the [architecture section](parametric.md#architecture-how-system-tests-work) if you're confused throughout this process.
+
+Then we need to do the following:
+
+* Determine what you want the endpoint to be called and what you need it to do, and add it to your tracer's http server.
+
+*Note:* If adding a new endpoint please let a Python tracer implementer know so they can add it as well [see](parametric.md#shared-interface)
+*Note*: Only add new endpoints that operate on the public API and execute ONE operation. Endpoints that execute complex operations or validate tracer internals will not be accepted.
+* In [_library_client.py](/utils/parametric/_library_client.py) Add both the endpoint call in `class APMLibraryClient` and the method that invokes it in `class APMLibrary`. Use other implementations for reference.
+
+* Ok we now have our new method! Use it in the tests you write using the [below section](#if-the-methods-you-need-to-run-your-tests-are-already-written)
+
+### If the methods you need to run your tests are already written
+
+If it makes sense to add your tests to a file that already exists, great! Otherwise make a new test file in `tests/parametric`.
+
+Next copy the testing code you want to use as a base/guideline (usually the class (if using a new file) and one of the test methods in it).
+
+Then:
+
+* [Change the name of the feature annotation it'll fit under for the feature parity board](/docs/edit/features.md) (Not always needed e.g. `@features.datadog_headers_propagation` is used for all the propagation features)
+* Change the class and method name to fit what you're testing.
+* [Change your tracer's respective manifest.yml file](/docs/edit/manifest.md) or else the script won't know to run your new test. If you're confused at how to do this properly, search for the file you copied the test from in the manifest file and see how it's specified, you can probably copy that for your new file (make sure the path is the same).
+For the version value, to make sure your test runs, specify the current release your tracer is on. This is the minimum value that the script will run your test with. If you make it too high, the script will skip your test.
+* Write the test pulling from examples of other tests written. Remember you're almost always follwing the pattern of making spans, getting them from the trace_agent, and then verifying values on them.
+
+**Finally:**
+[Try running your test!](parametric.md#running-the-tests)
+If you have an issue, checkout the [debugging section](parametric.md#debugging) to troubleshoot.