Enable Data Operations for Updates

[cristianvasquez]

Description

The objective of this task is to implement a mechanism that allows users to query data sources and schedule new transformation jobs based on those queries.

Features

• Select Sources: Users should be able to create and define queries to select data sources according to various criteria, facilitating targeted data processing.
• Schedule New Jobs: The sources identified through the queries can be utilized to initiate new transformation jobs, ensuring efficient handling of relevant data.
• Prevent Concurrent Processing: Implement locking mechanisms during transformations to prevent concurrent processing and race conditions. Locks should be released once the transformation is complete or has failed, allowing for further actions.

Metadata Requirements

Ensure that the operational metadata has sufficient granularity and that the queries are expressive enough to support Data Operations effectively.

User Interface Proposal

Propose an interface or API that enables users to edit and inspect the results of the queries for scheduling new jobs. This interaction should be user-friendly and accessible through a UI or a Jupyter notebook, designed for users with basic Python knowledge.

Deliverables

• Examples of Selection Queries: Provide examples of queries that can be used to select notices and batches.

• Demonstrator: Create a demonstrator that showcases a common operation, illustrating how to select and schedule sources for transformation.

Constraints

• During transformations, ensure that notices are locked to prevent concurrent processing and race conditions.
• Clean up any obsolete or unused components in the codebase.