Merge pull request #146 from seqeralabs/json_output_to_stdout

feat(SeqeraPlatform): seqerakit can write JSON to stdout for detailed records

README.md

@@ -54,14 +54,17 @@ pip install --upgrade --force-reinstall seqerakit
 ```
 
 ### Local development installation
+
 You can install the development branch of `seqerakit` on your local machine to test feature updates of the tool. Before proceeding, ensure that you have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system.
 
 1. To install directly from pip:
+
 ```bash
 pip install git+https://github.com/seqeralabs/seqera-kit.git@dev
 ```
 
 2. Alternatively, you may clone the repository locally and install manually:
+
 ```bash
 git clone https://github.com/seqeralabs/seqera-kit.git
 cd seqera-kit
@@ -70,6 +73,7 @@ pip install .
 ```
 
 You can verify your installation with:
+
 ```bash
 pip show seqerakit
 ```
@@ -85,11 +89,12 @@ export TOWER_ACCESS_TOKEN=<Your access token>
 ```
 
 For Enterprise installations of Seqera Platform, you will also need to configure the API endpoint that will be used to connect to the Platform. You can do so by exporting the following environment variable:
+
 ```bash
 export TOWER_API_ENDPOINT=<Tower API URL>
 ```
-By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud.
 
+By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud.
 
 ## Usage
 
@@ -100,25 +105,33 @@ seqerakit --info
 ```
 
 Use the `--help` or `-h` parameter to list the available commands and their associated options:
+
 ```bash
 seqerakit --help
 ```
 
 Use `--version` or `-v` to retrieve the current version of your seqerakit installation:
+
 ```bash
 seqerakit --version
 ```
+
 ### Input
+
 `seqerakit` supports input through either file paths to YAMLs or directly from standard input (stdin).
 
 #### Using File Path
+
 ```bash
 seqerakit /path/to/file.yaml
 ```
+
 #### Using stdin
+
 ```console
 $ cat file.yaml | seqerakit -
 ```
+
 See the [Defining your YAML file using CLI options](#defining-your-yaml-file-using-cli-options) section for guidance on formatting your input YAML file(s).
 
 ### Dryrun
@@ -129,6 +142,53 @@ To print the commands that would executed with `tw` when using a YAML file, you
 seqerakit file.yaml --dryrun
 ```
 
+To capture the details of the created resources, you can use the `--json` command-line flag to output the results as JSON to `stdout`. This is equivalent to using the `-o json` flag with the `tw` CLI.
+
+For example:
+
+```bash
+seqerakit -j examples/yaml/e2e/launch.yml
+```
+
+This command internally runs the following `tw` command:
+
+```bash
+INFO:root: Running command: tw -o json launch --name hello --workspace $SEQERA_ORGANIZATION_NAME/$SEQERA_WORKSPACE_NAME hello
+```
+
+The output will look like this:
+
+```json
+{
+  "workflowId": "1wfhRp5ioFIyrs",
+  "workflowUrl": "https://tower.nf/orgs/orgName/workspaces/workspaceName/watch/1wfhRp5ioFIyrs",
+  "workspaceId": 12345678,
+  "workspaceRef": "[orgName / workspaceName]"
+}
+```
+
+This JSON output can be piped into other tools for further processing. Note that logs will still be written to `stderr`, allowing you to monitor the tool's progress in real-time.
+
+If you prefer to suppress the JSON output and focus only on the logs:
+
+```bash
+seqerakit -j examples/yaml/e2e/launch.yml > /dev/null
+```
+
+This will still log:
+
+```bash
+INFO:root: Running command: tw -o json launch --name hello --workspace $SEQERA_ORGANIZATION_NAME/$SEQERA_WORKSPACE_NAME hello
+```
+
+Each execution of the `tw` CLI generates a single JSON object. To combine multiple JSON objects into one, you can use a tool like `jq`:
+
+```bash
+seqerakit -j launch/*.yml | jq --slurp > launched-pipelines.json
+```
+
+This command will merge the individual JSON objects from each `tw` command into a single JSON array and save it to `launched-pipelines.json`.
+
 ### Recursively delete
 
 Instead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag:
@@ -170,6 +230,7 @@ seqerakit hello-world-config.yml --cli="-Djavax.net.ssl.trustStore=/absolute/pat
 <b>Note</b>: Use of `--verbose` option for the `tw` CLI is currently not supported by `seqerakit`. Supplying `--cli="--verbose"` will raise an error.
 
 ## Specify targets
+
 When using a YAML file as input that defines multiple resources, you can use the `--targets` flag to specify which resources to create. This flag takes a comma-separated list of resource names.
 
 For example, given a YAML file that defines the following resources:
@@ -178,28 +239,29 @@ For example, given a YAML file that defines the following resources:
 workspaces:
   - name: 'showcase'
     organization: 'seqerakit_automation'
-...
+---
 compute-envs:
   - name: 'compute-env'
     type: 'aws-batch forge'
     workspace: 'seqerakit/test'
-...
+---
 pipelines:
-  - name: "hello-world-test-seqerakit"
-    url: "https://github.com/nextflow-io/hello"
+  - name: 'hello-world-test-seqerakit'
+    url: 'https://github.com/nextflow-io/hello'
     workspace: 'seqerakit/test'
-    compute-env: "compute-env"
-...
+    compute-env: 'compute-env'
 ```
 
 You can target the creation of `pipelines` only by running:
 
 ```bash
 seqerakit test.yml --targets pipelines
 ```
+
 This will process only the pipelines block from the YAML file and ignore other blocks such as `workspaces` and `compute-envs`.
 
 ### Multiple Targets
+
 You can also specify multiple resources to create by separating them with commas. For example, to create both workspaces and pipelines, run:
 
 ```bash
@@ -240,6 +302,7 @@ params:
 **Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`.
 
 ### 2. `overwrite` Functionality
+
 For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name.
 
 `seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw <subcommand> delete` command before attempting to create it based on the options defined in the YAML file.
@@ -253,22 +316,23 @@ DEBUG:root: The attempted organizations resource already exists. Overwriting.
 DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME
 DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization'
 ```
+
 ### 3. Specifying JSON configuration files with `file-path`
+
 The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file.
 
 An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template:
 
 ```yaml
 compute-envs:
-  - name: 'my_aws_compute_environment'                              # required
-    workspace: 'my_organization/my_workspace'                       # required
-    credentials: 'my_aws_credentials'                               # required
-    wait: 'AVAILABLE'                                               # optional
-    file-path: './compute-envs/my_aws_compute_environment.json'     # required
+  - name: 'my_aws_compute_environment' # required
+    workspace: 'my_organization/my_workspace' # required
+    credentials: 'my_aws_credentials' # required
+    wait: 'AVAILABLE' # optional
+    file-path: './compute-envs/my_aws_compute_environment.json' # required
     overwrite: True
 ```
 
-
 ## Quick start
 
 You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform.
@@ -281,10 +345,10 @@ You will need to have an account on Seqera Platform (see [Plans and pricing](htt
 
    ```yaml # noqa
    launch:
-     - name: 'hello-world'                              # Workflow name
-       workspace: '<YOUR_WORKSPACE>'                    # Workspace name
-       compute-env: '<YOUR_COMPUTE_ENVIRONMENT>'        # Compute environment
-       revision: 'master'                               # Pipeline revision
+     - name: 'hello-world' # Workflow name
+       workspace: '<YOUR_WORKSPACE>' # Workspace name
+       compute-env: '<YOUR_COMPUTE_ENVIRONMENT>' # Compute environment
+       revision: 'master' # Pipeline revision
        pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL
    ```
 
@@ -340,6 +404,7 @@ Options:
       --revision=<revision>                A valid repository commit Id, tag or branch name.
   ...
 ```
+
 2. Define Key-Value Pairs in YAML
 
 Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance:
@@ -364,12 +429,12 @@ In this example:
 - The corresponding values are user-defined
 
 ### Best Practices:
+
 - Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting.
 - Use quotes around strings that contain special characters or spaces.
 - When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above.
 - For complex configurations, refer to the [Templates](./templates/) provided in this repository.
 
-
 ## Templates
 
 We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization:

examples/python/launch_hello_world.py

@@ -28,5 +28,4 @@
     "--wait",
     "SUBMITTED",
     "https://github.com/nextflow-io/hello",
-    to_json=True,
 )

seqerakit/cli.py

@@ -47,6 +47,9 @@ def parse_args(args=None):
         action="store_true",
         help="Display Seqera Platform information and exit.",
     )
+    general.add_argument(
+        "-j", "--json", action="store_true", help="Output JSON format in stdout."
+    )
     general.add_argument(
         "--dryrun",
         "-d",
@@ -152,9 +155,12 @@ def main(args=None):
     options = parse_args(args if args is not None else sys.argv[1:])
     logging.basicConfig(level=getattr(logging, options.log_level.upper()))
 
-    # Parse CLI arguments into a list and create a Seqera Platform instance
+    # Parse CLI arguments into a list
     cli_args_list = options.cli_args.split() if options.cli_args else []
-    sp = seqeraplatform.SeqeraPlatform(cli_args=cli_args_list, dryrun=options.dryrun)
+
+    sp = seqeraplatform.SeqeraPlatform(
+        cli_args=cli_args_list, dryrun=options.dryrun, json=options.json
+    )
 
     # If the info flag is set, run 'tw info'
     if options.info:

seqerakit/computeenvs.py

@@ -49,4 +49,4 @@ def export_ce(self, name, *args, **kwargs):
         ]
 
         # Pass the built command to the base class method in SeqeraPlatform
-        return self._tw_run(command, *args, **kwargs, to_json=True)
+        return self._tw_run(command, *args, **kwargs)

seqerakit/seqeraplatform.py

@@ -43,20 +43,21 @@ def __call__(self, *args, **kwargs):
             return self.tw_instance._tw_run(command, **kwargs)
 
     # Constructs a new SeqeraPlatform instance
-    def __init__(self, cli_args=None, dryrun=False, print_stdout=True):
+    def __init__(self, cli_args=None, dryrun=False, print_stdout=True, json=False):
         if cli_args and "--verbose" in cli_args:
             raise ValueError(
                 "--verbose is not supported as a CLI argument to seqerakit."
             )
         self.cli_args = cli_args or []
         self.dryrun = dryrun
         self.print_stdout = print_stdout
+        self.json = json
         self._suppress_output = False
 
     def _construct_command(self, cmd, *args, **kwargs):
         command = ["tw"] + self.cli_args
 
-        if kwargs.get("to_json"):
+        if self.json:
             command.extend(["-o", "json"])
 
         command.extend(cmd)
@@ -112,13 +113,21 @@ def _execute_command(self, full_cmd, to_json=False, print_stdout=True):
             print_stdout if print_stdout is not None else self.print_stdout
         ) and not self._suppress_output
 
-        if should_print:
+        # Do not print output in logging if self.json is enabled
+        if should_print and not self.json:
             logging.info(f" Command output: {stdout}")
 
         if "ERROR: " in stdout or process.returncode != 0:
             self._handle_command_errors(stdout)
 
-        return json.loads(stdout) if to_json else stdout
+        if self.json or to_json:
+            out = json.loads(stdout)
+            print(json.dumps(out))
+        else:
+            out = stdout
+            print(stdout)
+
+        return out
 
     def _handle_command_errors(self, stdout):
         # Check for specific tw cli error patterns and raise custom exceptions
@@ -139,7 +148,7 @@ def _tw_run(self, cmd, *args, **kwargs):
         full_cmd = self._construct_command(cmd, *args, **kwargs)
         if not full_cmd or self.dryrun:
             logging.info(f"DRYRUN: Running command {full_cmd}")
-            return
+            return None
         return self._execute_command(full_cmd, kwargs.get("to_json"), print_stdout)
 
     @contextmanager