DataStax CDM 5.1.0 origin DB connection issue.

[Diksha-Tewari-1212]

We found an issue with CDM 5.1.0 jar when trying to submit a spark job with origin DB as Cassasndra and target DB as astra (please refer configuration parameters as below) over Dataproc Serverless batch we get following error.

Configuration parameters for Spark Job:

gcloud dataproc batches submit spark
--region=us-east1
--jars=gs://<GCP BUCKET PATH>/cassandra-data-migrator-5.1.0.jar
--class=com.datastax.cdm.job.Migrate
--properties
^#^spark.executor.memory=4g#
spark.executor.cores=4#
spark.executor.instances=2#
spark.cdm.perfops.batchSize=1#
spark.cdm.perfops.numParts=10#
spark.cdm.perfops.ratelimit.origin=15000#
spark.cdm.trackRun=true#
spark.cdm.trackRun.runId=2204#
spark.cdm.schema.origin.keyspaceTable=<SOURCE_KEYSPACE>.forecasting_features#
spark.cdm.connect.origin.username=<ORIGIN USERNAME>#
spark.cdm.connect.origin.password=<ORIGIN PASSWORD>#
spark.cdm.connect.origin.host=<ORIGIN HOST FQDN/ IP>#
spark.cdm.connect.origin.port=9042#
spark.cdm.schema.origin.column.writetime.names=feature#
spark.cdm.feature.constantColumns.names=tenant_uuid#
spark.cdm.feature.constantColumns.types=uuid#
spark.cdm.feature.constantColumns.values=e72a66e5-e185-4111-8f73-4c1d60b2d8a7#
spark.cdm.schema.target.keyspaceTable=<TARGET_KEYSPACE>.forecasting_features#
spark.cdm.connect.target.username=<TARGET USERNAME>#
spark.cdm.connect.target.password=<TARGET PASSWORD>#
spark.cdm.connect.target.scb=secure-connect-kpi-keng03-dev-diksha-astra-db-05004-mykronos.zip    
--files=gs://<GCP BUCKET PATH>/secure-connect-kpi-keng03-dev-diksha-astra-db-05004-mykronos.zip,gs://<GCP BUCKET PATH>/identity.jks,gs://<GCP BUCKET PATH>/trustStore.jks
--service-account=<GCP SECRET ACCOUNT NAME> 
--tags=<NETWORK_TAGS>
--subnet=<SUBNET>
--batch=batch-cdm-dse-22-11-008
--version=2.2

Error Logs:

Caused by: com.datastax.oss.driver.api.core.AllNodesFailedException: Could not reach any contact point, make sure you've provided valid addresses (showing first 3 nodes, use getAllErrors() for more): Node(endPoint=65f333dc-991b-4a9a-877c-21e250a2d67c-us-east1.db.int.dev.mykronos.com/<unresolved>:29042:00958fd0-af98-391f-9f51-7e3fe763da5c, hostId=null, hashCode=3e8d557f): [com.datastax.oss.driver.api.core.auth.AuthenticationException: Authentication error on node 65f333dc-991b-4a9a-877c-21e250a2d67c-us-east1.db.int.dev.mykronos.com/<unresolved>:29042:00958fd0-af98-391f-9f51-7e3fe763da5c: server replied with 'We recently improved your database security. To find out more and reconnect, see https://docs.datastax.com/en/astra/docs/manage-application-tokens.html' to AuthResponse request], Node(endPoint=65f333dc-991b-4a9a-877c-21e250a2d67c-us-east1.db.int.dev.mykronos.com/<unresolved>:29042:b4f6d9ed-0f1d-3f7a-82f1-a4a7ea4f84d4, hostId=null, hashCode=753592c3): [com.datastax.oss.driver.api.core.auth.AuthenticationException: Authentication error on node 65f333dc-991b-4a9a-877c-21e250a2d67c-us-east1.db.int.dev.mykronos.com/<unresolved>:29042:b4f6d9ed-0f1d-3f7a-82f1-a4a7ea4f84d4: server replied with 'We recently improved your database security. To find out more and reconnect, see https://docs.datastax.com/en/astra/docs/manage-application-tokens.html' to AuthResponse request], Node(endPoint=65f333dc-991b-4a9a-877c-21e250a2d67c-us-east1.db.int.dev.mykronos.com/<unresolved>:29042:678e6a1d-e5b1-3cba-b949-7bfaf823957f, hostId=null, hashCode=1ba02720): [com.datastax.oss.driver.api.core.auth.AuthenticationException: Authentication error on node 65f333dc-991b-4a9a-877c-21e250a2d67c-us-east1.db.int.dev.mykronos.com/<unresolved>:29042:678e6a1d-e5b1-3cba-b949-7bfaf823957f: server replied with 'We recently improved your database security. To find out more and reconnect, see https://docs.datastax.com/en/astra/docs/manage-application-tokens.html' to AuthResponse request]
    at com.datastax.oss.driver.api.core.AllNodesFailedException.copy(AllNodesFailedException.java:143)
    at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:151)
    at com.datastax.oss.driver.api.core.session.SessionBuilder.build(SessionBuilder.java:837)
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createSession(CassandraConnectionFactory.scala:143)
    at com.datastax.spark.connector.cql.CassandraConnector$.createSession(CassandraConnector.scala:167)
    ... 25 more
    Suppressed: com.datastax.oss.driver.api.core.auth.AuthenticationException: Authentication error on node

[msmygit]

Thank you for reporting this bug @Diksha-Tewari-1212 and many congratulations for opening your 1st ever GitHub issue 🎉. Asking the obvious question here to rule them out:

1. Did we ensure we're indeed using the correct auth credentials along with the secure-connect-bundle (SCB) for the region that you're attempting to use (I see this is a single-region at GCP us-east1 Astra DB target cluster, so there shouldn't be any confusion, but still it is good to check if we aren't using an SCB from a different cluster)?
2. Have you enabled private link or IP access list that would prevent the machine where CDM is hosted attempting to connect to this target cluster?
3. Also, would you be open to trying with the latest patch release version of CDM, which is 5.1.2 and see if you're facing the same behavior?