Optionally persist intermediates for reduce

[shreyashankar]

Add persist_intermediates feature to ReduceOperation

This PR closes #13.

Changes

• Users can now specify "persist_intermediates": True in the reduce operation config.
• When enabled, intermediate fold outputs are saved to a special key "_{op_name}_intermediates" in the result.

New Functionality

• The persist_intermediates option allows users to track the progression of the reduce operation.
• Intermediate results are stored as a list of dictionaries, each containing:
  • iter: The iteration number
  • intermediate: The intermediate result
  • scratchpad: Any additional information from the fold operation

Testing

• Added a new test case test_reduce_operation_persist_intermediates to verify the functionality.
• The test ensures that:
  • Intermediates are correctly persisted when the option is enabled
  • The structure of the intermediates is as expected
  • Intermediates are stored in the correct order

Usage Example

reduce_config = {
"name": "example_reduce",
"type": "reduce",
"reduce_key": "group",
"persist_intermediates": True,
# ... other config options ...
}

With this configuration, the results will include a key _example_reduce_intermediates containing the list of intermediate results.

Notes

• This feature is optional and does not affect the normal operation of the ReduceOperation when not enabled.
• The persisted intermediates can be useful for debugging, auditing, or analyzing the reduce process in detail.