You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Nov 14, 2022. It is now read-only.
We try to run K-means algorithm by using PMEM-SPILL on spark-3.0.1 while meet the issue like following picture:
PMEM-SPILL overwrite the source code of spark("src/main/scala/org/apache/spark/internal/config/package.scala") and add some configs for PMEM
While the package.scala of spark-3.0.0 is different with the one of spark-3.0.1 and the issue is caused by the package.scala of OAP-1.1 PMEM-SPILL not contains EXECUTOR_ALLOW_SPARK_CONTEXT.
So how can we decouple package.scala of spark? If we cannot decouple this file, we need to adapt to all some versions of spark.
The text was updated successfully, but these errors were encountered:
@haojinIntel Now, the pmem-spill project has not been adapted to spark 3.0.1 and only supports Spark 3.0.0. Compared with Spark3.0.0, the configuration items in package.scala of Spark3.0.1 are slightly changed. Similarly, Spark 3.1.1 also has the same problem. I submitted a PR to make pmem-spill support Spark 3.1.1, please refer to #33 .If you have any questions, welcome to discuss and communicate
We try to run K-means algorithm by using PMEM-SPILL on spark-3.0.1 while meet the issue like following picture:
PMEM-SPILL overwrite the source code of spark("src/main/scala/org/apache/spark/internal/config/package.scala") and add some configs for PMEM
While the package.scala of spark-3.0.0 is different with the one of spark-3.0.1 and the issue is caused by the package.scala of OAP-1.1 PMEM-SPILL not contains EXECUTOR_ALLOW_SPARK_CONTEXT.
So how can we decouple package.scala of spark? If we cannot decouple this file, we need to adapt to all some versions of spark.
The text was updated successfully, but these errors were encountered: