Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major refactor of code to make it work in Spark Cluster mode #323

Merged
merged 5 commits into from
Oct 25, 2024

Conversation

pravinbhat
Copy link
Collaborator

@pravinbhat pravinbhat commented Oct 24, 2024

What this PR does: Major refactor of code to make it work in Spark Cluster mode

  • Makes CDM work when deployed on a Spark Cluster (does not report metrics counts correctly in this mode)
  • Helps make CDM more performant for large migration efforts (multi-terabytes, several billions of rows) using Spark Cluster (instead of individual VMs)
  • No feature changes (other than removing unused Guardrail feature within Migrate and DiffData)
  • Fully backward compatible (other than the changes to Guardrail feature mentioned above)

Note: This PR/release will have a bug (only in cluster mode) that it does not report metrics counts when deployed on Spark Cluster, however it does the underlying tasks (Migration, DiffData, GuardrailCheck) as expected. When deployed as a Spark job on a single VM (not cluster mode), everything (including the metrics) will work as expected. So no impact to existing customer who use the tool in the single VM fashion

Checklist:

  • Automated Tests added/updated
  • Documentation added/updated
  • CLA Signed: DataStax CLA

@msmygit msmygit enabled auto-merge (squash) October 25, 2024 12:54
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@msmygit msmygit merged commit d3a0d05 into main Oct 25, 2024
9 checks passed
@msmygit msmygit deleted the feature/spark-cluster-mode branch October 25, 2024 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants