Merge ci from main (#2584)

* Update README.md nodejs platform support (#2397) * Update README.md Signed-off-by: Avi Fenesh <[email protected]> * Update README.md - lint fix Signed-off-by: Avi Fenesh <[email protected]> * Update README.md ling Signed-off-by: Avi Fenesh <[email protected]> * Update node/README.md Co-authored-by: Yury-Fridlyand <[email protected]> Signed-off-by: Avi Fenesh <[email protected]> --------- Signed-off-by: Avi Fenesh <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]> * CI - Minimal and full CI matrix impl (#2051) * CI - Minimal and full CI matrix impl Signed-off-by: avifenesh <[email protected]> * Fix mypy failing (#2453) --------- Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: avifenesh <[email protected]> * Python: adds JSON.ARRLEN command (#2403) --------- Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: avifenesh <[email protected]> --------- Signed-off-by: avifenesh <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Co-authored-by: Shoham Elias <[email protected]> * CI - Minimal and full CI matrix impl (#2051) * CI - Minimal and full CI matrix impl Signed-off-by: avifenesh <[email protected]> * Fix mypy failing (#2453) --------- Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: avifenesh <[email protected]> * Python: adds JSON.ARRLEN command (#2403) --------- Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: avifenesh <[email protected]> --------- Signed-off-by: avifenesh <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Co-authored-by: Shoham Elias <[email protected]> * CI - Minimal and full CI matrix impl (#2051) * CI - Minimal and full CI matrix impl Signed-off-by: avifenesh <[email protected]> * Fix mypy failing (#2453) --------- Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: avifenesh <[email protected]> * Python: adds JSON.ARRLEN command (#2403) --------- Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: avifenesh <[email protected]> --------- Signed-off-by: avifenesh <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Co-authored-by: Shoham Elias <[email protected]> * Fix for - Minimal and full CI matrix impl #2051 (#2500) * Refactor tests to use async cleanup and improve error handling Signed-off-by: avifenesh <[email protected]> * Enhance Jest configuration and add test setup file; update build scripts and dependencies Signed-off-by: avifenesh <[email protected]> * Update devDependencies in package.json for hybrid-node-tests to latest versions Signed-off-by: avifenesh <[email protected]> * Enhance test utilities and command tests with improved wait logic and version checks Signed-off-by: avifenesh <[email protected]> * Refactor tests to assert expected replica reads are less than or equal to actual reads; update connection handling in utilities and allow unused imports in types Signed-off-by: avifenesh <[email protected]> * Update dependencies and enhance PyO3 bindings; add new features and improve type handling Signed-off-by: avifenesh <[email protected]> * Update GitHub workflows: enhance linting configurations, adjust engine version requirements, and remove obsolete Redis installation workflow Signed-off-by: avifenesh <[email protected]> --------- Signed-off-by: avifenesh <[email protected]> * fixes for CI (#2552) Signed-off-by: avifenesh <[email protected]> * Refactor CI configuration for consistency and clarity Signed-off-by: avifenesh <[email protected]> * Refactor CI configuration for consistency and clarity Signed-off-by: avifenesh <[email protected]> --------- Signed-off-by: Avi Fenesh <[email protected]> Signed-off-by: avifenesh <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Signed-off-by: Shoham Elias <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]> Co-authored-by: Shoham Elias <[email protected]>
valkey-io · Nov 5, 2024 · 2b6578a · 2b6578a
1 parent db7b1bb
commit 2b6578a
Show file tree

Hide file tree

Showing 61 changed files with 1,921 additions and 1,437 deletions.
diff --git a/.github/DEVELOPER.md b/.github/DEVELOPER.md
@@ -0,0 +1,144 @@
+# CI/CD Workflow Guide
+
+### Overview
+
+Our CI/CD pipeline tests and builds our project across multiple languages, versions, and environments. This guide outlines the key components and processes of our workflow.
+
+### Workflow Triggers
+
+-   Pull requests
+-   Pushes to `main` or release branches (PR merges)
+-   Scheduled runs (daily) - starts CI pipelines for all clients
+-   Manual trigger (`workflow_dispatch`) - a developer can start a client's pipeline or the scheduled one to run all pipelines on demand
+
+<img width="437" alt="Job triggers" src="https://github.com/user-attachments/assets/58bf2b76-d778-4e43-8891-5dcbf0ff9b72">
+
+### Test coverage
+
+There are two levels of testing: the basic one and full (_aka_ `full-matrix`).
+Basic amount of testing is executed on every open and merged PR. The full set of tests is executed by the scheduled job.
+A developer can select the level when starting a job, either scheduled or client's pipeline.
+
+<img width="264" alt="Matrices" src="https://github.com/user-attachments/assets/72857f80-078c-4e11-bcc6-75beb0125a9d">
+
+### Language-Specific Workflows
+
+Each language has its own workflow file with similar structure but language-specific steps, for example python.yml for Python, or java.yml for Java.
+
+### Shared Components
+
+#### Matrix Files
+
+While workflows are language-specific, the matrix files are shared across all workflows.
+Workflows are starting by loading the matrix files from the `.github/json_matrices` directory.
+
+-   `engine-matrix.json`: Defines the versions of the engine to test against.
+-   `build-matrix.json`: Defines the host environments for testing.
+-   `supported-languages-versions.json`: Defines the supported versions of languages.
+
+All matrices have a `run` like field which specifies if the configuration should be tested on every workflow run.
+This allows for flexible control over which configurations are tested in different scenarios, optimizing CI/CD performance and resource usage.
+
+#### Engine Matrix (engine-matrix.json)
+
+Defines the versions of Valkey engine to test against:
+
+```json
+[
+    { "type": "valkey", "version": "7.2.5", "run": "always" }
+    // ... other configurations
+]
+```
+
+-   `type`: The type of engine (e.g., Valkey, Redis).
+-   `version`: Specifies the engine version that the workflow should checkout.
+    For example, "7.2.5" represents a release tag, while "7.0" denotes a branch name. The workflow should use this parameter to checkout the specific release version or branch to build the engine with the appropriate version.
+-   `run`: Specifies if the engine version should be tested on every workflow.
+
+#### Build Matrix (build-matrix.json)
+
+Defines the host environments for testing:
+
+```json
+[
+    {
+        "OS": "ubuntu",
+        "RUNNER": "ubuntu-latest",
+        "TARGET": "x86_64-unknown-linux-gnu",
+        "run": ["always", "python", "node", "java"]
+    }
+    // ... other configurations
+]
+```
+
+-   `OS`: The operating system of the host.
+-   `RUNNER`: The GitHub runner to use.
+-   `TARGET`: The target environment as defined in Rust. To see a list of available targets, run `rustup target list`.
+-   `run`: Specifies which language workflows should use this host configuration. The value `always` indicates that the configuration should be used for every workflow trigger.
+
+#### Supported Languages Version (supported-languages-version.json)
+
+Defines the supported versions of languages:
+
+```json
+[
+    {
+        "language": "java",
+        "versions": ["11", "17"],
+        "always-run-versions": ["17"]
+    }
+    // ... other configurations
+]
+```
+
+-   `language`: The language for which the version is supported.
+-   `versions`: The full versions supported of the language which will test against scheduled.
+-   `always-run-versions`: The versions that will always be tested, regardless of the workflow trigger.
+
+#### Triggering Workflows
+
+-   Push to `main` by merging a PR or create a new pull request to run workflows automatically.
+-   Use `workflow_dispatch` for manual triggers, accepting a boolean configuration parameter to run all configurations.
+-   Scheduled runs are triggered daily to ensure regular testing of all configurations.
+
+### Mutual vs. Language-Specific Components
+
+#### Mutual
+
+-   Matrix files - `.github/json_matrices/`
+-   Shared dependencies installation - `.github/workflows/install-shared-dependencies/action.yml`
+-   Rust linters - `.github/workflows/lint-rust/action.yml`
+
+#### Language-Specific
+
+-   Package manager commands
+-   Testing frameworks
+-   Build processes
+
+### Customizing Workflows
+
+Modify `<language>.yml` files to adjust language-specific steps.
+Update matrix files to change tested versions or environments.
+Adjust cron schedules in workflow files for different timing of scheduled runs.
+
+### Workflow Matrices
+
+We use dynamic matrices for our CI/CD workflows, which are created using the `create-test-matrices` action. This action is defined in `.github/workflows/create-test-matrices/action.yml`.
+
+#### How it works
+
+1. The action is called with a `language-name` input and `dispatch-run-full-matrix` input.
+2. It reads the `engine-matrix.json`, `build-matrix.json`, and `supported-languages-version.json` files.
+3. It filters the matrices based on the inputs and the event type.
+4. It generates three matrices:
+    - Engine matrix: Defines the types and versions of the engine to test against, for example Valkey 7.2.5.
+    - Host matrix: Defines the host platforms to run the tests on, for example Ubuntu on ARM64.
+    - Language-version matrix: Defines the supported versions of languages, for example python 3.8.
+
+#### Outputs
+
+-   `engine-matrix-output`: The generated engine matrix.
+-   `host-matrix-output`: The generated host matrix.
+-   `language-version-matrix-output`: The generated language version matrix.
+
+This dynamic matrix generation allows for flexible and efficient CI/CD workflows, adapting the test configurations based on the type of change and the specific language being tested.
diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml
@@ -5,134 +5,134 @@ title: "(topic): (short issue description)"
 labels: [bug, needs-triage]
 assignees: []
 body:
-  - type: textarea
-    id: description
-    attributes:
-      label: Describe the bug
-      description: What is the problem? A clear and concise description of the bug.
-    validations:
-      required: true
+    - type: textarea
+      id: description
+      attributes:
+          label: Describe the bug
+          description: What is the problem? A clear and concise description of the bug.
+      validations:
+          required: true
 
-  - type: textarea
-    id: expected
-    attributes:
-      label: Expected Behavior
-      description: |
-        What did you expect to happen?
-    validations:
-      required: true
+    - type: textarea
+      id: expected
+      attributes:
+          label: Expected Behavior
+          description: |
+              What did you expect to happen?
+      validations:
+          required: true
 
-  - type: textarea
-    id: current
-    attributes:
-      label: Current Behavior
-      description: |
-        What actually happened?
-        
-        Please include full errors, uncaught exceptions, stack traces, and relevant logs.
-        If service responses are relevant, please include wire logs.
-    validations:
-      required: true
+    - type: textarea
+      id: current
+      attributes:
+          label: Current Behavior
+          description: |
+              What actually happened?
 
-  - type: textarea
-    id: reproduction
-    attributes:
-      label: Reproduction Steps
-      description: |
-        Provide a self-contained, concise snippet of code that can be used to reproduce the issue.
-        For more complex issues provide a repo with the smallest sample that reproduces the bug.
-        
-        Avoid including business logic or unrelated code, it makes diagnosis more difficult.
-        The code sample should be an SSCCE. See http://sscce.org/ for details. In short, please provide a code sample that we can copy/paste, run and reproduce.
-    validations:
-      required: true
+              Please include full errors, uncaught exceptions, stack traces, and relevant logs.
+              If service responses are relevant, please include wire logs.
+      validations:
+          required: true
 
-  - type: textarea
-    id: solution
-    attributes:
-      label: Possible Solution
-      description: |
-        Suggest a fix/reason for the bug
-    validations:
-      required: false
+    - type: textarea
+      id: reproduction
+      attributes:
+          label: Reproduction Steps
+          description: |
+              Provide a self-contained, concise snippet of code that can be used to reproduce the issue.
+              For more complex issues provide a repo with the smallest sample that reproduces the bug.
 
-  - type: textarea
-    id: context
-    attributes:
-      label: Additional Information/Context
-      description: |
-        Anything else that might be relevant for troubleshooting this bug. Providing context helps us come up with a solution that is most useful in the real world.
-    validations:
-      required: false
+              Avoid including business logic or unrelated code, it makes diagnosis more difficult.
+              The code sample should be an SSCCE. See http://sscce.org/ for details. In short, please provide a code sample that we can copy/paste, run and reproduce.
+      validations:
+          required: true
 
-  - type: input
-    id: client-version
-    attributes:
-      label: Client version used
-    validations:
-      required: true
+    - type: textarea
+      id: solution
+      attributes:
+          label: Possible Solution
+          description: |
+              Suggest a fix/reason for the bug
+      validations:
+          required: false
 
-  - type: input
-    id: engine-version
-    attributes:
-      label: Engine type and version
-      description: E.g. Valkey 7.0
-    validations:
-      required: true
+    - type: textarea
+      id: context
+      attributes:
+          label: Additional Information/Context
+          description: |
+              Anything else that might be relevant for troubleshooting this bug. Providing context helps us come up with a solution that is most useful in the real world.
+      validations:
+          required: false
 
-  - type: input
-    id: operating-system
-    attributes:
-      label: OS
-    validations:
-      required: true
+    - type: input
+      id: client-version
+      attributes:
+          label: Client version used
+      validations:
+          required: true
 
-  - type: dropdown
-    id: language
-    attributes:
-      label: Language
-      multiple: true
-      options:
-        - TypeScript
-        - Python
-        - Java
-        - Rust
-        - Go
-        - .Net
-    validations:
-      required: true
+    - type: input
+      id: engine-version
+      attributes:
+          label: Engine type and version
+          description: E.g. Valkey 7.0
+      validations:
+          required: true
 
-  - type: input
-    id: language-version
-    attributes:
-      label: Language Version
-      description: E.g. TypeScript (5.2.2) | Python (3.9)
-    validations:
-      required: true
+    - type: input
+      id: operating-system
+      attributes:
+          label: OS
+      validations:
+          required: true
 
-  - type: textarea
-    id: cluster-info
-    attributes:
-      label: Cluster information
-      description: |
-        Cluster information, cluster topology, number of shards, number of replicas, used data types.
-    validations:
-      required: false
+    - type: dropdown
+      id: language
+      attributes:
+          label: Language
+          multiple: true
+          options:
+              - TypeScript
+              - Python
+              - Java
+              - Rust
+              - Go
+              - .Net
+      validations:
+          required: true
 
-  - type: textarea
-    id: logs
-    attributes:
-      label: Logs
-      description: |
-        Client and/or server logs.
-    validations:
-      required: false
+    - type: input
+      id: language-version
+      attributes:
+          label: Language Version
+          description: E.g. TypeScript (5.2.2) | Python (3.9)
+      validations:
+          required: true
 
-  - type: textarea
-    id: other
-    attributes:
-      label: Other information
-      description: |
-        e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. associated pull-request, stackoverflow, etc
-    validations:
-      required: false
+    - type: textarea
+      id: cluster-info
+      attributes:
+          label: Cluster information
+          description: |
+              Cluster information, cluster topology, number of shards, number of replicas, used data types.
+      validations:
+          required: false
+
+    - type: textarea
+      id: logs
+      attributes:
+          label: Logs
+          description: |
+              Client and/or server logs.
+      validations:
+          required: false
+
+    - type: textarea
+      id: other
+      attributes:
+          label: Other information
+          description: |
+              e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. associated pull-request, stackoverflow, etc
+      validations:
+          required: false