4.4.1

Key Highlights

• Added two new codecs STRING_BLOB and ASCII_BLOB to allow migration from TEXT and ASCII fields to BLOB fields. These codecs can also be used to convert BLOB to TEXT or ASCII, but in such cases the BLOB value must be TEXT based in nature & fit within the applicable limits.

Jar file can also be found in the packages section here.

4.4.0

Key Highlights

• Added property spark.cdm.connect.origin.tls.isAstra and spark.cdm.connect.target.tls.isAstra to allow connecting to Astra DB without using SCB. This may be needed for enterprises that may find credentials packaged within SCB as a security risk [while actually it is not a real concern as they're protected with access tokens; having access to just one of them won't grant access to the Astra DB cluster]. TLS properties can now be passed as params OR wrapper scripts (not included) could be used to pull sensitive credentials from a vault service in real-time & pass them to CDM.
• Switched to using Apache Cassandra® 5.0 docker image for testing
• Introduces smoke testing of vector CQL data type

Jar file can also be found in the packages section here.

4.3.10

Key Highlights

• Added a new property spark.cdm.trackRun.runId to support a custom unique identifier for the current run. This can be used by wrapper scripts to pass a known runId and then use it to query the cdm_run_info and cdm_run_details tables.

Jar file can also be found in the packages section here.

4.3.9

Key Highlights

• Added new status value of DIFF_CORRECTED on cdm_run_details table to specifically mark partitions that were corrected during the CDM validation run.
• Upgraded Validation job to skip partitions with DIFF_CORRECTED status on rerun with a previous runId.

Jar file can also be found in the packages section here.

4.3.8

Key Highlights

• Upgraded spark.cdm.trackRun feature to include status on cdm_run_info table.
• Code fixes to handle rerun of previous run which may have exited before being correctly initialized.

Jar file can also be found in the packages section here.

4.3.7

Key Highlights

• Added property spark.cdm.transform.custom.ttl to allow a custom constant value to be set for TTL instead of using the values from origin rows.
• Repo wide code formating & imports organization

Jar file can also be found in the packages section here.

4.3.6

Key Highlights

• Added overwrite option to conditionally check or skip Validation when it has a non-null value in target for the spark.cdm.feature.extractJson feature.

Jar file can also be found in the packages section here.

4.3.5

Key Highlights

• Added feature spark.cdm.feature.extractJson which allows you to extract a json value from a column with json content in an Origin table and map it to a column in the Target table.
• Upgraded to use Spark 3.5.2.

Jar file can also be found in the packages section here.

4.3.4

Key Highlights

• Minor bug fix (Use origin keyspace/table value when target value is empty/missing) to the regression introduced in 4.3.3.

Jar file can also be found in the packages section here.

4.3.3

Key Highlights

• Removed partitions file feature (which was deprecated in the prior versions).
• Minor bug fix (Enable tracking when only previousRunId provided, but trackRun not set to true).
• Upgraded Spark Cassandra Connector (SCC) version to 3.5.1.

Jar file can also be found in the packages section here.