RMM Failure - using SPARK Rapid with python

[DrVajiha]

I'm using an Ubuntu system with the GPU- NVIDIA Quadro K1200 which has 4GB memory, 32GB RAM.
Installed - CUDF 22.10.0 jar file, Rapid 4 Spark 2.12-22.10.0 jar file, CUDA Toolkit 11.8.0 Linux version.
I'm running only single server with 1 GPU, Master is localhost
SPARK version 3.2.2, pyspark version 3.2.2, Java8 version,

I'm trying to run python code through spark submit & Python idle. I'm getting error (RMM Failure) in spark config. Kindly help me to resolve this error. Kindly give suggestions where I'm doing wrong?

Python Code which I'm running:
spark = SparkSession.builder.appName('Spark.com').config("master","local").config("num-executors", "1")\        .config("spark.executor.memory","12g").config("spark.driver.memory","6g").config("spark.executor.resource.gpu.discoveryScript",". /home/mwadmin/.local/lib/python3.8/site-packages/pyspark/getGpusResources.sh")
        .config("spark.executor.cores","1").config("spark.rapids.sql.concurrentGpuTasks","1")
        .config("driver-memory", "10g").config("spark.rapids.memory.pinnedPool.size","1G ")
        .config("spark.sql.files.maxPartitionBytes","512m").config("spark.rapids.sql.enabled",True)
        .config("spark.plugins","com.nvidia.spark.SQLPlugin").config("spark.task.resource.gpu.amount","0.5")
        .config("spark.executor.resource.gpu.amount","1").config("spark.rapids.sql.concurrentGpuTasks", "1").getOrCreate()

Error when running code through spark-submit:
spark-submit /home/mwadmin/Documents/test.py
22/11/30 14:59:32 WARN Utils: Your hostname, mwadmin-HP-Z440-Workstation resolves to a loopback address: 127.0.1.1; using ... instead (on interface eno1)
22/11/30 14:59:32 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/11/30 14:59:32 INFO SparkContext: Running Spark version 3.2.2
22/11/30 14:59:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/11/30 14:59:33 INFO ResourceUtils: ==============================================================
22/11/30 14:59:33 INFO ResourceUtils: No custom resources configured for spark.driver.
22/11/30 14:59:33 INFO ResourceUtils: ==============================================================
22/11/30 14:59:33 INFO SparkContext: Submitted application: Spark.com
22/11/30 14:59:33 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: , gpu -> name: gpu, amount: 1, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0, gpu -> name: gpu, amount: 0.5)
22/11/30 14:59:33 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
22/11/30 14:59:33 WARN ResourceUtils: The configuration of resource: gpu (exec = 1, task = 0.5/2, runnable tasks = 2) will result in wasted resources due to resource cpus limiting the number of runnable tasks per executor to: 1. Please adjust your configuration.
22/11/30 14:59:33 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/11/30 14:59:33 INFO SecurityManager: Changing view acls to: mwadmin
22/11/30 14:59:33 INFO SecurityManager: Changing modify acls to: mwadmin
22/11/30 14:59:33 INFO SecurityManager: Changing view acls groups to:
22/11/30 14:59:33 INFO SecurityManager: Changing modify acls groups to:
22/11/30 14:59:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mwadmin); groups with view permissions: Set(); users with modify permissions: Set(mwadmin); groups with modify permissions: Set()
22/11/30 14:59:33 INFO Utils: Successfully started service 'sparkDriver' on port 45883.
22/11/30 14:59:33 INFO SparkEnv: Registering MapOutputTracker
22/11/30 14:59:33 INFO SparkEnv: Registering BlockManagerMaster
22/11/30 14:59:33 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/11/30 14:59:33 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/11/30 14:59:33 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/11/30 14:59:33 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-647d2c2a-72e4-402d-aeff-d7460726eb6d
22/11/30 14:59:33 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB
22/11/30 14:59:33 INFO SparkEnv: Registering OutputCommitCoordinator
22/11/30 14:59:33 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/11/30 14:59:33 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at htttp://localhost:4040
22/11/30 14:59:33 INFO ShimLoader: Loading shim for Spark version: 3.2.2
22/11/30 14:59:33 INFO ShimLoader: Complete Spark build info: 3.2.2, https://github.com/apache/spark, HEAD, 78a5825fe266c0884d2dd18cbca9625fa258d7f7, 2022-07-11T15:44:21Z
22/11/30 14:59:33 INFO ShimLoader: findURLClassLoader found a URLClassLoader org.apache.spark.util.MutableURLClassLoader@1530c739
22/11/30 14:59:33 INFO ShimLoader: Updating spark classloader org.apache.spark.util.MutableURLClassLoader@1530c739 with the URLs: jar:file:/home/mwadmin/spark-3.2.2-bin-hadoop3.2/jars/rapids-4-spark_2.12-22.10.0.jar!/spark3xx-common/, jar:file:/home/mwadmin/spark-3.2.2-bin-hadoop3.2/jars/rapids-4-spark_2.12-22.10.0.jar!/spark322/
22/11/30 14:59:33 INFO ShimLoader: Spark classLoader org.apache.spark.util.MutableURLClassLoader@1530c739 updated successfully
22/11/30 14:59:33 INFO ShimLoader: Updating spark classloader org.apache.spark.util.MutableURLClassLoader@1530c739 with the URLs: jar:file:/home/mwadmin/spark-3.2.2-bin-hadoop3.2/jars/rapids-4-spark_2.12-22.10.0.jar!/spark3xx-common/, jar:file:/home/mwadmin/spark-3.2.2-bin-hadoop3.2/jars/rapids-4-spark_2.12-22.10.0.jar!/spark322/
22/11/30 14:59:33 INFO ShimLoader: Spark classLoader org.apache.spark.util.MutableURLClassLoader@1530c739 updated successfully
22/11/30 14:59:33 INFO RapidsPluginUtils: RAPIDS Accelerator build: {version=22.10.0, user=, url=https://github.com/NVIDIA/spark-rapids.git, date=2022-10-17T11:25:41Z, revision=c75a2eafc9ce9fb3e6ab75c6677d97bf681bff50, cudf_version=22.10.0, branch=HEAD}
22/11/30 14:59:33 INFO RapidsPluginUtils: RAPIDS Accelerator JNI build: {version=22.10.0, user=, url=https://github.com/NVIDIA/spark-rapids-jni.git, date=2022-10-14T05:19:41Z, revision=b2c02b61afe1747f3741d6c5e2064edb8da51b32, branch=HEAD}
22/11/30 14:59:33 INFO RapidsPluginUtils: cudf build: {version=22.10.0, user=, date=2022-10-14T01:51:22Z, revision=8ffe375d85f8fd0f98e0052f36ccd820a669d0ab, branch=HEAD}
22/11/30 14:59:33 WARN RapidsPluginUtils: RAPIDS Accelerator 22.10.0 using cudf 22.10.0.
22/11/30 14:59:33 WARN RapidsPluginUtils: spark.rapids.sql.multiThreadedRead.numThreads is set to 20.
22/11/30 14:59:33 WARN RapidsPluginUtils: RAPIDS Accelerator is enabled, to disable GPU support set spark.rapids.sql.enabled to false.
22/11/30 14:59:33 WARN RapidsPluginUtils: spark.rapids.sql.explain is set to NOT_ON_GPU. Set it to 'NONE' to suppress the diagnostics logging about the query placement on the GPU.
22/11/30 14:59:33 INFO DriverPluginContainer: Initialized driver component for plugin com.nvidia.spark.SQLPlugin.
22/11/30 14:59:33 WARN ResourceUtils: The configuration of resource: gpu (exec = 1, task = 0.5/2, runnable tasks = 2) will result in wasted resources due to resource cpus limiting the number of runnable tasks per executor to: 1. Please adjust your configuration.
22/11/30 14:59:34 INFO Executor: Starting executor ID driver on host ...
22/11/30 14:59:34 INFO RapidsExecutorPlugin: RAPIDS Accelerator build: {version=22.10.0, user=, url=https://github.com/NVIDIA/spark-rapids.git, date=2022-10-17T11:25:41Z, revision=c75a2eafc9ce9fb3e6ab75c6677d97bf681bff50, cudf_version=22.10.0, branch=HEAD}
22/11/30 14:59:34 INFO RapidsExecutorPlugin: cudf build: {version=22.10.0, user=, date=2022-10-14T01:51:22Z, revision=8ffe375d85f8fd0f98e0052f36ccd820a669d0ab, branch=HEAD}
22/11/30 14:59:34 INFO RapidsExecutorPlugin: Initializing memory from Executor Plugin
22/11/30 14:59:47 INFO Executor: Told to re-register on heartbeat
22/11/30 14:59:47 INFO BlockManager: BlockManager null re-registering with master
22/11/30 14:59:48 INFO BlockManagerMaster: Registering BlockManager null
22/11/30 14:59:48 ERROR Inbox: Ignoring error
java.lang.NullPointerException
at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:534)
at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:117)
at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
22/11/30 14:59:48 WARN Executor: Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:103)
at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:87)
at org.apache.spark.storage.BlockManagerMaster.registerBlockManager(BlockManagerMaster.scala:78)
at org.apache.spark.storage.BlockManager.reregister(BlockManager.scala:626)
at org.apache.spark.executor.Executor.reportHeartBeat(Executor.scala:1009)
at org.apache.spark.executor.Executor.$anonfun$heartbeater$1(Executor.scala:212)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2048)
at org.apache.spark.Heartbeater$$anon$1.run(Heartbeater.scala:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException
at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:534)
at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:117)
at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
22/11/30 14:59:52 INFO GpuDeviceManager: Initializing RMM ASYNC pool size = 3137.0625 MB on gpuId 0
22/11/30 14:59:52 INFO GpuDeviceManager: Using per-thread default stream
22/11/30 14:59:52 ERROR RapidsExecutorPlugin: Exception in the executor plugin, shutting down!
ai.rapids.cudf.CudfException: RMM failure at: /home/jenkins/agent/workspace/jenkins-cudf-release-39-cuda11/cpp/build/_deps/rmm-src/include/rmm/mr/device/cuda_async_memory_resource.hpp:90: cudaMallocAsync not supported with this CUDA driver/runtime version
at ai.rapids.cudf.Rmm.initializeInternal(Native Method)
at ai.rapids.cudf.Rmm.initialize(Rmm.java:119)
at com.nvidia.spark.rapids.GpuDeviceManager$.initializeRmm(GpuDeviceManager.scala:296)
at com.nvidia.spark.rapids.GpuDeviceManager$.initializeMemory(GpuDeviceManager.scala:328)
at com.nvidia.spark.rapids.GpuDeviceManager$.initializeGpuAndMemory(GpuDeviceManager.scala:137)
at com.nvidia.spark.rapids.RapidsExecutorPlugin.init(Plugin.scala:258)
at org.apache.spark.internal.plugin.ExecutorPluginContainer.$anonfun$executorPlugins$1(PluginContainer.scala:125)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at org.apache.spark.internal.plugin.ExecutorPluginContainer.(PluginContainer.scala:113)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:211)
at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:199)
at org.apache.spark.executor.Executor.$anonfun$plugins$1(Executor.scala:253)
at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:231)
at org.apache.spark.executor.Executor.(Executor.scala:253)
at org.apache.spark.scheduler.local.LocalEndpoint.(LocalSchedulerBackend.scala:64)
at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:132)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220)
at org.apache.spark.SparkContext.(SparkContext.scala:581)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)
22/11/30 14:59:52 INFO DiskBlockManager: Shutdown hook called
22/11/30 14:59:52 INFO ShutdownHookManager: Shutdown hook called
22/11/30 14:59:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-58488513-7d53-42f2-8bc4-cdcb34b5cf49
22/11/30 14:59:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-24b8e0ea-43d4-430a-9756-b1e84ceaa1ff/userFiles-5ce7f28f-16db-48fd-94bd-e9ef563c01f1
22/11/30 14:59:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-24b8e0ea-43d4-430a-9756-b1e84ceaa1ff

Error When running code through Python IDLE:
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

[abellina]

Unfortunately this GPU (Quadro K1200) is compute capability 5.0 and it is not supported for RAPIDS-backed libraries. RAPIDS is documented to support devices at 6.0+ capability (Pascal or above) https://rapids.ai/start.html, so devices with 5.0 capability are not going to work.

[revans2]

I filed #7208 because we should really throw a much cleaner error message instead.

[DrVajiha]

Thanks @abellina for the reply, I have tried GPU above 6.0+ compute capability and it works.

[abellina]

@DrVajiha that is good to hear. Feel free to create other discussions as you have questions or file an issue if you run into something while you experiment.