Validate datasets versions

[ElenaKhaustova]

Description

Solves #4327

Merge before #4329

Development notes

Added _validate_versions function to ensure all datasets in a catalog adhere to a versioning scheme - we allow single load version per dataset in the catalog and one save version for all datasets in the catalog. The function automatically updates the provided load versions based on the versions specified for the individual datasets. It also ensures all versioned datasets in the catalog share the same save version. If a conflict arises, a VersionAlreadyExistsError is raised.

Validation is applied to both DataCatalog and KedroDataCatalog when a catalog is created or the dataset is added.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[merelcht]

LGTM, and all makes sense 👍

One thing I was wondering is if this could somehow break existing workflows that use the DataCatalog?

[ElenaKhaustova]

LGTM, and all makes sense 👍

One thing I was wondering is if this could somehow break existing workflows that use the DataCatalog?

That's a good question. Technically, we always required DataCatalog with versioned datasets to have only one save version. But we never validated it, and this requirement can be bypassed. So we consider this change as a fix—that's why we applied it for both old and new catalogs. We also think this shouldn't be the common case, but even if some workflows break, that's because people used versioning not as expected.

The alternative is to apply these changes just to KedroDataCatalog.

[merelcht]

That's a good question. Technically, we always required DataCatalog with versioned datasets to have only one save version. But we never validated it, and this requirement can be bypassed. So we consider this change as a fix—that's why we applied it for both old and new catalogs. We also think this shouldn't be the common case, but even if some workflows break, that's because people used versioning not as expected.

The alternative is to apply these changes just to KedroDataCatalog.

Yes so technically speaking this is breaking for people who have been passing instantiated dataset objects to catalog constructor with different save versions. It's a very rare thing to do and I agree this is actually improving the expected behaviour. So let's just keep it like this, but then at least if someone does complain we know what's going on.

[idanov]

LGTM 👍