Add CLI command to dump inputs/outputs of `CalcJob`/`WorkChain` #6276

- Reverted formatting changes in `cmd_process` - Removed unneccessary comments - Removed `:type` if type annotations present - `no_node_inputs` -> `include_inputs` to be consistent with `include_attributes`/`include_extras` - Don't set default path (rather than pwd) and check for None

- Remove `verdi calcjob dump` endpoint (not sure about this, might add again) - Only call `process_dump` -> Adapted the function for this - Removed conditional in `generate_node_input_label`

This option allows dumping of a CalcJob (or of a simple WorkChain that only calls a single CalcJob) in a flat directory, without creating the hierarchy of `raw_inputs`, `raw_outputs`, and `node_inputs`. This might be useful for cases where AiiDA is only used to run a calculation and dump the results in a specific custom path, where the custom path is dictated by the other code that calls AiiDA to submit calculations (e.g. what we are currently working on aiida-koopmans).

Different calcjob_dump io functions run through. Also added tests for the flat option for these functions. Still need to update tests for arithmetic_add, and process_dump functions, both for io and for multiply_add.

- Moved both `genereate_..._io` functions to the end of the file - Extended `generate_workchain_io` fixture to allow adding multiple `calcjob_nodes` to test that the flat dumping breaks in that case - Currently, when dumping the MultiplyAddWorkchain flat, the `source_file` of the `multiply` step is missing -> Still need to figure that one out

To be consistent with other commands in `cmd_process`. And with that `process_dump` and `calcjob_dump` in `processes.py` to `process_node_dump` and `calcjob_node_dump`.

Now in function `generate_calcjob_io_dump_paths`. Takes care of handling the `flat` argument and the naming of the `raw_inputs`, `raw_outputs`, and `node_inputs` subdirectories.

Changed `--flat` option to still create subdirectories for the individual steps of the WorkChain. Instead, just the subdirectories per CalcJob are removed. Generalized the dumping of outputs that it doesn't only dump `retrieved` -> With this, it's dumping a whole range of aiida nodes, basically all the parsed outputs, which are mainly numpy arrays dumped as `.npy` files. Add an option to enable this, as it might not be necessary to dump all of those. Currently, I just defined a global variable in the file, but this will eventually become a class attribute of the ProcessDumper class.

To avoid having to pass all the arguments `include_node_inputs`, `include_attributes/extras`, `overwrite`, `flat`, `all_aiida_nodes` through the different functions, everything related to the dumping is now compiled in the `ProcessDumper` class, which defines the main entry-point method `dump`. For nested workflows, this is recursively called (as before). Once `CalculationFunction` nodes are reached, their content is dumped via `dump_calculation_node`. The helper functions to create and validate labels and paths of nested subdirectories are also methods of the `ProcessDumper`. Introduced the `parent_process` class attribute which is dynamically generated from the parent_node, and which is used to generate the main README, which is only created when the dumping is done via the `verdi` CLI. For the other functions, this concept does not make sense, due to the recursion, so the respective `process_node`s (which are changing during the recursion) are always passed as arguments. Next steps: - Update tests to actually test the new implementations - Update docstrings - Add section to `How to work with data` section of the docs - If the `OverridableOptions` are only used here, they can also just be defined as normal `click` options (however, we can also start thinking about the `verdi archive dump` functionality that we should start implementing soon)

for more information, see https://pre-commit.ci

Moved the recursive logic out of the top-level `dump` function instead into `_workflow_dump`. In addition, moved the default path creation and validation into the top-level `dump` function and out of the `cmd_process.py` file. The following entities are now dumped for each child `CalculationNode` reached during the dumping: - `CalculationNode` repository -> `inputs` - `CalculationNode` retrieved output -> `outputs` - `CalculationNode` input nodes -> `node_inputs` - `CalculationNode` output nodes (apart from `retrieved`) -> `node_outputs` By default, everything apart from the `node_outputs` is dumped, as to avoid too many non-`SinglefileData` or `FolderData` nodes to be written to disk. The `--all-aiida-nodes` option is instead removed. The number of files might still grow large for complex workchains, e.g. `SelfConsistentHubbardWorkchain` or `EquationOfStateWorkChain`. In addition, set `_generate_default_dump_path`, `_generate_readme`, and `_generate_child_node_label` as `staticmethod`s, as they logically belong to the class, but don't access any of its attributes. The former two are further only called in the top-level `dump` method. Other methods like `_validate_make_dump_path` still access class attributes like `overwrite` or `flat`, so they remain normal class methods.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CLI command to dump inputs/outputs of `CalcJob`/`WorkChain` #6276

Add CLI command to dump inputs/outputs of `CalcJob`/`WorkChain` #6276

Commits on May 27, 2024

Add CLI command to dump inputs/outputs of CalcJob/WorkChain #6276

Add CLI command to dump inputs/outputs of CalcJob/WorkChain #6276

Commits on May 27, 2024

Add CLI command to dump inputs/outputs of `CalcJob`/`WorkChain` #6276

Add CLI command to dump inputs/outputs of `CalcJob`/`WorkChain` #6276