Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create smoke test python workflow for cluster deployment and testing #5164

Open
zelinh opened this issue Nov 4, 2024 · 0 comments
Open

Create smoke test python workflow for cluster deployment and testing #5164

zelinh opened this issue Nov 4, 2024 · 0 comments
Assignees
Labels
enhancement New Enhancement test

Comments

@zelinh
Copy link
Member

zelinh commented Nov 4, 2024

Subtask from #4551

A new smoke test python workflow would be introduced as part of smoke test framework we designed. The smoke test runner is python will be implemented.
The new introduced smoke test runner automates OpenSearch cluster deployment, loads test manifests, sends API requests, and validates responses against the OpenSearch API specification with third-party OpenAPI-core python client. It ensures all key components function immediately after a build.

Acceptance criteria of the Smoke Test python workflow.

  1. Cluster Deployment: The smoke test python runner will automate the deployment of the OpenSearch cluster using the distribution generated by the CI/CD pipeline. The runner will ensure that the cluster is fully operational, providing an endpoint (e.g., https://localhost:9200) where smoke tests can be run.
  2. Test Manifest Interpretation: Once the cluster is running, the smoke test runner will load the test manifest provided during execution. This manifest contains a list of components (e.g., OpenSearch core, plugins like alerting or anomaly detection) and the corresponding smoke test specifications. The runner will parse these manifest entries to determine which components to test. For each component listed in the test manifest, the runner will reference the corresponding component spec YAML file (e.g., opensearch.yml, alerting.yml) to extract API paths that need validation. These paths, which are stored in the YAML specification, represent critical OpenSearch operations (e.g., /, /_cat/indices).
  3. API Request Execution: The runner will send requests to these endpoints using requests. It will also handle authentication and security configuration if required (e.g., with or without security plugin). In our example, the request URL will be https://localhost:9200/ & https://localhost:9200/_cat/plugins
  4. OpenAPI Specification Validation: After sending the API requests, the smoke test runner will validate both the requests and the responses against the OpenSearch API specification. This validation ensures that the API responses adhere to the expected structure defined in the opensearch-api-specification.

The runner will import the OpenAPI-Core Python client for validation, which allows us to:

  • Validate requests: Ensures the requests made to the API are formatted correctly and valid, including parameters and headers.
  • Validate responses: Compares the API responses against the expected schema in the OpenSearch API specification. If any deviations are found, the smoke test will flag the response as invalid.
@zelinh zelinh self-assigned this Nov 4, 2024
@zelinh zelinh moved this from 🆕 New to 🏗 In progress in Engineering Effectiveness Board Nov 4, 2024
@github-actions github-actions bot added the untriaged Issues that have not yet been triaged label Nov 4, 2024
@zelinh zelinh added enhancement New Enhancement test and removed untriaged Issues that have not yet been triaged labels Nov 4, 2024
@bshien bshien moved this from 🏗 In progress to 👀 In review in Engineering Effectiveness Board Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New Enhancement test
Projects
Status: 📋 Escalate to LF
Development

No branches or pull requests

1 participant