[FEATURE] [BREAKING] Transition Streaming Framework and Plugin to Scala #61
Assignees
Labels
code/new-feature
New feature or request
Comments
This was referenced Oct 25, 2024
s-vitaliy
added a commit
that referenced
this issue
Nov 4, 2024
## Scope Part of #61 This PR adds the method that generates backfill queries for the MS SQL source
s-vitaliy
added a commit
that referenced
this issue
Nov 4, 2024
## Scope Part of #61 This PR adds the method that generates backfill queries for the MS SQL source
s-vitaliy
added a commit
that referenced
this issue
Nov 4, 2024
## Scope Part of #61 This PR implements the backfill behavior for MSSQL server client, where the client queries all data from the data source and returns this data as a LazyList. NOTE: the `LazyList` class is used because the `Stream` class is deprecated
4 tasks
s-vitaliy
added a commit
that referenced
this issue
Nov 4, 2024
* Add the backfill implementation for MSSQL client ## Scope Part of #61 This PR implements the backfill behavior for MSSQL server client, where the client queries all data from the data source and returns this data as a LazyList. NOTE: the `LazyList` class is used because the `Stream` class is deprecated * Fix unit tests * Update framework/arcane-framework/src/main/scala/models/DataColumn.scala Co-authored-by: George Zubrienko <[email protected]> * Review fixes --------- Co-authored-by: George Zubrienko <[email protected]>
4 tasks
s-vitaliy
added a commit
that referenced
this issue
Nov 6, 2024
Part of #61 This PR Changes the initial version of ChangeTracking stream to Option[Long] from Long
4 tasks
s-vitaliy
added a commit
that referenced
this issue
Nov 6, 2024
* Change type of the initial version type Part of #61 This PR Changes the initial version of ChangeTracking stream to Option[Long] from Long * Self-review fixes
george-zubrienko
changed the title
s-vitaliy
added a commit
that referenced
this issue
Nov 8, 2024
#61 This PR fixes the issue which can occur when the method `getLatestVersion` of the `HasVersion` typeclass peeks the first element of the `VersionedBatch`. In this case the stateful `ResultSet` inside this class moves forward and the first row of the set is being lost. To avoid this issue, the `CanPeekHead` trait was added. The implementation of this trait saves the first row if the `ResultSet` if the head of the list was requested.
4 tasks
s-vitaliy
added a commit
that referenced
this issue
Nov 8, 2024
* Fix issue when VersionedResult can lose first row of the data #61 This PR fixes the issue which can occur when the method `getLatestVersion` of the `HasVersion` typeclass peeks the first element of the `VersionedBatch`. In this case the stateful `ResultSet` inside this class moves forward and the first row of the set is being lost. To avoid this issue, the `CanPeekHead` trait was added. The implementation of this trait saves the first row if the `ResultSet` if the head of the list was requested. * Build fix * Test fix
s-vitaliy
added a commit
that referenced
this issue
Nov 8, 2024
Part of #61 This PR adds test that validates the ability of the MS SQL streaming graph read deletions from the database and adds missing documentation for the expected behavior. Original test case in Arcane.Framework.Net can be found [here](https://github.com/SneaksAndData/arcane-framework/blob/8ffdc749f044d544605a2bcd7faee8d84b5e59f6/test/Sources/SqlServerChangeTrackingSourceTests.cs#L75)
s-vitaliy
added a commit
that referenced
this issue
Nov 8, 2024
* Fix issue when VersionedResult can lose first row of the data #61 This PR fixes the issue which can occur when the method `getLatestVersion` of the `HasVersion` typeclass peeks the first element of the `VersionedBatch`. In this case the stateful `ResultSet` inside this class moves forward and the first row of the set is being lost. To avoid this issue, the `CanPeekHead` trait was added. The implementation of this trait saves the first row if the `ResultSet` if the head of the list was requested. * Build fix * Test fix * Port unit tests from the .Net version of Arcane Framework part 1 Part of #61 This PR adds test that validates the ability of the MS SQL streaming graph read deletions from the database and adds missing documentation for the expected behavior. Original test case in Arcane.Framework.Net can be found [here](https://github.com/SneaksAndData/arcane-framework/blob/8ffdc749f044d544605a2bcd7faee8d84b5e59f6/test/Sources/SqlServerChangeTrackingSourceTests.cs#L75)
s-vitaliy
added a commit
that referenced
this issue
Nov 11, 2024
Part of #61. Implement the DI using ZIO and add the interfaces for the basic services
s-vitaliy
added a commit
that referenced
this issue
Nov 11, 2024
Part of #61. Implement the DI using ZIO and add the interfaces for the basic services
s-vitaliy
added a commit
that referenced
this issue
Nov 11, 2024
Part of #61. Implemented: - Graph processor that reads lazy results and creates a grouped chunk of DataRow for the Sink - Added Settings models: `GroupingSettings`, `VersionedDataGraphBuilderSettings`
4 tasks
s-vitaliy
added a commit
that referenced
this issue
Nov 12, 2024
* Implement DI and basic interfaces Part of #61. Implement the DI using ZIO and add the interfaces for the basic services * Implement grouping and look back interval Part of #61. Implemented: - Graph processor that reads lazy results and creates a grouped chunk of DataRow for the Sink - Added Settings models: `GroupingSettings`, `VersionedDataGraphBuilderSettings`
This was referenced Nov 13, 2024
Merged
This was referenced Nov 25, 2024
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
.NET has served its purpose and with the new design on the road that allows having both producer and consumer inside Arcane runner, we need Iceberg API access, which is not a thing in .NET. Thus we now depart from MS world and re-write streaming code in ZIO
Checklist
getChangesmethod actually moves forward in version history. see also: Implement getChanges method for Arcane SQL server #75 (comment)VersionedBatchtype should be abstractQueryRunnerclass should be deletedConversiontypeclass as it is implemented hereThe text was updated successfully, but these errors were encountered: