JPMML-SparkML

Java library and command-line application for converting Apache Spark ML pipelines to PMML.

Table of Contents

• Features
  • Overview
  • Supported libraries
• Prerequisites
• Installation
  • Library
  • Example application
• Usage
  • Library
  • Example application
• Documentation
• License
• Additional information

Features

Overview

• Functionality:
  • Thorough collection, analysis and encoding of feature information:
    • Names.
    • Data and operational types.
    • Valid, invalid and missing value spaces.
  • Pipeline extensions:
    • Pruning.
    • Model verification.
  • Conversion options.
• Extensibility:
  • Rich Java APIs for developing custom converters.
  • Automatic discovery and registration of custom converters based on META-INF/sparkml2pmml.properties resource files.
  • Direct interfacing with other JPMML conversion libraries such as JPMML-LightGBM and JPMML-XGBoost.
• Production quality:
  • Complete test coverage.
  • Fully compliant with the JPMML-Evaluator library.

Supported libraries

Apache Spark ML

Examples: main.py

• Feature extractors, transformers and selectors:
  • feature.Binarizer
  • feature.Bucketizer
  • feature.ChiSqSelectorModel (the result of fitting a feature.ChiSqSelector)
  • feature.ColumnPruner
  • feature.CountVectorizerModel (the result of fitting a feature.CountVectorizer)
  • feature.IDFModel (the result of fitting a feature.IDF)
  • feature.ImputerModel (the result of fitting a feature.Imputer)
  • feature.IndexToString
  • feature.Interaction
  • feature.MaxAbsScalerModel (the result of fitting a feature.MaxAbsScaler)
  • feature.MinMaxScalerModel (the result of fitting a feature.MinMaxScaler)
  • feature.NGram
  • feature.OneHotEncoderModel (the result of fitting a feature.OneHotEncoder)
  • feature.PCAModel (the result of fitting a feature.PCA)
  • feature.QuantileDiscretizer
  • feature.RegexTokenizer
  • feature.RFormulaModel (the result of fitting a feature.RFormula)
  • feature.SQLTransformer
    • Subqueries.
    • Control flow expressions case when and if.
    • Arithmetic operators +, -, * and /.
    • Comparison operators <, <=, ==, >= and >.
    • Logical operators and, or and not.
    • Math functions abs, ceil, exp, expm1, floor, hypot, ln, log10, log1p, pow and rint.
    • Trigonometric functions sin, asin, sinh, cos, acos, cosh, tan, atan, tanh.
    • Aggregation functions greatest and least.
    • RegExp functions regexp_replace and rlike.
    • String functions char_length, character_length, concat, lcase, length, lower, replace, substring, trim, ucase and upper.
    • Type cast functions boolean, cast, double, int and string.
    • Value functions in, isnan, isnull, isnotnull, negative and positive.
  • feature.StandardScalerModel (the result of fitting a feature.StandardScaler)
  • feature.StopWordsRemover
  • feature.StringIndexerModel (the result of fitting a feature.StringIndexer)
  • feature.Tokenizer
  • feature.VectorAssembler
  • feature.VectorAttributeRewriter
  • feature.VectorIndexerModel (the result of fitting a feature.VectorIndexer)
  • feature.VectorSizeHint
  • feature.VectorSlicer
• Prediction models:
  • classification.DecisionTreeClassificationModel
  • classification.GBTClassificationModel
  • classification.LinearSVCModel
  • classification.LogisticRegressionModel
  • classification.MultilayerPerceptronClassificationModel
  • classification.NaiveBayesModel
  • classification.RandomForestClassificationModel
  • clustering.KMeansModel
  • fpm.FPGrowthModel
  • regression.DecisionTreeRegressionModel
  • regression.GBTRegressionModel
  • regression.GeneralizedLinearRegressionModel
  • regression.LinearRegressionModel
  • regression.RandomForestRegressionModel
• Prediction model chains:
  • PipelineModel
  • Referencing the prediction column (HasPredictionCol#getPredictionCol()) of earlier clustering, classification and regression models.
  • Referencing the predicted probabilities column (HasProbabilityCol#getProbabilityCol()) of earlier classification models.
• Hyperparameter selectors and tuners:
  • tuning.CrossValidatorModel
  • tuning.TrainValidationSplitModel

JPMML-SparkML

• Feature transformers:
  • org.jpmml.sparkml.feature.InvalidCategoryTransformer
  • org.jpmml.sparkml.feature.SparseToDenseTransformer

LightGBM

Examples: LightGBMAuditNA.scala, LightGBMAutoNA.scaka, etc.

• Prediction models:
  • com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassificationModel
  • com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressionModel

XGBoost

Examples: XGBoostAuditNA.scala, XGBoostAutoNA.scala, etc.

• Prediction models:
  • ml.dmlc.xgboost4j.scala.spark.XGBoostClassificationModel
  • ml.dmlc.xgboost4j.scala.spark.XGBoostRegressionModel

Prerequisites

• Apache Spark 3.0.X, 3.1.X, 3.2.X, 3.3.X, 3.4.X or 3.5.X.

Installation

Library

JPMML-SparkML library JAR file (together with accompanying Java source and Javadocs JAR files) is released via Maven Central Repository.

The current version is 2.5.1 (20 June, 2024).

<dependency>
	<groupId>org.jpmml</groupId>
	<artifactId>pmml-sparkml</artifactId>
	<version>2.5.1</version>
</dependency>

Compatibility matrix

Active development branches:

Apache Spark version	JPMML-SparkML branch
3.0.X	2.0.X
3.1.X	2.1.X
3.2.X	2.2.X
3.3.X	2.3.X
3.4.X	2.4.X
3.5.X	master

Archived development branches:

Apache Spark version	JPMML-SparkML branch
1.5.X and 1.6.X	1.0.X
2.0.X	1.1.X
2.1.X	1.2.X
2.2.X	1.3.X
2.3.X	1.4.X
2.4.X	1.5.X
3.0.X	1.6.X
3.1.X	1.7.X
3.2.X	1.8.X

Example application

Enter the project root directory and build using Apache Maven:

mvn clean install

The build produces two JAR files:

• pmml-sparkml/target/pmml-sparkml-2.5-SNAPSHOT.jar - Library JAR file.
• pmml-sparkml-exampletarget/pmml-sparkml-example-executable-2.5-SNAPSHOT.jar - Example application JAR file.

Usage

Library

Fitting a Spark ML pipeline:

import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.classification.DecisionTreeClassifier
import org.apache.spark.ml.feature.RFormula

val irisData = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("Iris.csv")
val irisSchema = irisData.schema

val rFormula = new RFormula().setFormula("Species ~ .")
val dtClassifier = new DecisionTreeClassifier().setLabelCol(rFormula.getLabelCol).setFeaturesCol(rFormula.getFeaturesCol)
val pipeline = new Pipeline().setStages(Array(rFormula, dtClassifier))

val pipelineModel = pipeline.fit(irisData)

Converting the fitted Spark ML pipeline to an in-memory PMML class model object:

import org.jpmml.sparkml.PMMLBuilder

val pmml = new PMMLBuilder(irisSchema, pipelineModel).build()

The representation of individual Spark ML pipeline stages can be customized via conversion options:

import org.jpmml.sparkml.PMMLBuilder
import org.jpmml.sparkml.model.HasTreeOptions

val dtClassifierModel = pipelineModel.stages(1)

val pmml = new PMMLBuilder(irisSchema, pipelineModel).putOption(dtClassifierModel, HasTreeOptions.OPTION_COMPACT, false).putOption(dtClassifierModel, HasTreeOptions.OPTION_ESTIMATE_FEATURE_IMPORTANCES, true).build()

Viewing the in-memory PMML class model object:

import javax.xml.transform.stream.StreamResult
import org.jpmml.model.JAXBUtil

JAXBUtil.marshalPMML(pmml, new StreamResult(System.out))

Example application

The example application JAR file contains an executable class org.jpmml.sparkml.example.Main, which can be used to convert a pair of serialized org.apache.spark.sql.types.StructType and org.apache.spark.ml.PipelineModel objects to PMML.

The example application JAR file does not include Apache Spark runtime libraries. Therefore, this executable class must be executed using Apache Spark's spark-submit helper script.

For example, converting a pair of Spark ML schema and pipeline serialization files pmml-sparkml/src/test/resources/schema/Iris.json and pmml-sparkml/src/test/resources/pipeline/DecisionTreeIris.zip, respectively, to a PMML file DecisionTreeIris.pmml:

spark-submit --master local --class org.jpmml.sparkml.example.Main pmml-sparkml-example/target/pmml-sparkml-example-executable-2.5-SNAPSHOT.jar --schema-input pmml-sparkml/src/test/resources/schema/Iris.json --pipeline-input pmml-sparkml/src/test/resources/pipeline/DecisionTreeIris.zip --pmml-output DecisionTreeIris.pmml

Getting help:

spark-submit --master local --class org.jpmml.sparkml.example.Main pmml-sparkml-example/target/pmml-sparkml-example-executable-2.5-SNAPSHOT.jar --help

Documentation

• Training PySpark LightGBM pipelines
• Converting logistic regression models to PMML documents
• Deploying Apache Spark ML pipeline models on Openscoring REST web service
• Converting Apache Spark ML pipeline models to PMML documents

License

JPMML-SparkML is licensed under the terms and conditions of the GNU Affero General Public License, Version 3.0.

If you would like to use JPMML-SparkML in a proprietary software project, then it is possible to enter into a licensing agreement which makes JPMML-SparkML available under the terms and conditions of the BSD 3-Clause License instead.

Additional information

JPMML-SparkML is developed and maintained by Openscoring Ltd, Estonia.

Interested in using Java PMML API software in your company? Please contact [email protected]