Remove Python3.7 support in merlin-sdk (#464)

<!--  Thanks for sending a pull request!  Here are some tips for you:

1. Run unit tests and ensure that they are passing
2. If your change introduces any API changes, make sure to update the
e2e tests
3. Make sure documentation is updated for your PR!

-->

**What this PR does / why we need it**:
<!-- Explain here the context and why you're making the change. What is
the problem you're trying to solve. --->

Python 3.7 has reached its end of life
([ref](https://devguide.python.org/versions/)). This MR removes support
for Python 3.7 from the SDK. For the time being the Pyfunc model service
and batch job apps will still support Python 3.7 - this is achieved by
pinning their dependency on the `merlin-sdk`. A future MR will also
remove support for Python 3.7 from these components.

Related PR for Turing: caraml-dev/turing#362

## Summary of changes
* `python/sdk/setup.py` - Bump up Python requirement to a minimum of
`3.8`
* `python/pyfunc-server/requirements.txt`,
`python/batch-predictor/requirements.txt` - Pin `merlin-sdk` dependency
at `0.33.0` which is the latest SDK release that supports Python 3.7.
Corresponding changes in `python/pyfunc-server/setup.py` and
`python/batch-predictor/setup.py`.
* Update documentation and samples which use Python 3.7 to a different
version

**Does this PR introduce a user-facing change?**:
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required. Enter your extended release note in
the block below.
If the PR requires additional action from users switching to the new
release, include the string "action required".

For more information about release notes, see kubernetes' guide here:
http://git.k8s.io/community/contributors/guide/release-notes.md
-->

```release-note
Remove support for Python 3.7 from merlin-sdk
```

**Checklist**

- [ ] Added unit test, integration, and/or e2e tests
- [ ] Tested locally
- [x] Updated documentation
- [ ] Update Swagger spec if the PR introduce API changes
- [ ] Regenerated Golang and Python client if the PR introduce API
changes

.github/workflows/merlin.yml

@@ -42,7 +42,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.7", "3.8", "3.9", "3.10"] #TODO: Remove Python 3.7 support
     env:
       PIPENV_DEFAULT_PYTHON_VERSION: ${{ matrix.python-version }}
     steps:
@@ -74,7 +74,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.7", "3.8", "3.9", "3.10"] #TODO: Remove Python 3.7 support
     env:
       PIPENV_DEFAULT_PYTHON_VERSION: ${{ matrix.python-version }}
     steps:
@@ -106,7 +106,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.8", "3.9", "3.10"]
     env:
       PIPENV_DEFAULT_PYTHON_VERSION: ${{ matrix.python-version }}
     steps:

.github/workflows/release.yml

@@ -24,7 +24,7 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: 3.7
+          python-version: 3.8
       - name: Install dependencies
         working-directory: ./python/sdk
         run: |
@@ -46,7 +46,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.8", "3.9", "3.10"]
     needs:
       - publish-python-sdk
     steps:

.readthedocs.yml

@@ -11,6 +11,6 @@ sphinx:
 
 # Optionally set the version of Python and requirements required to build your docs
 python:
-  version: 3.7
+  version: 3.8
   install:
     - requirements: python/sdk/docs/requirements_docs.txt

api/api/versions_api.go

@@ -27,7 +27,7 @@ import (
 	"github.com/caraml-dev/merlin/utils"
 )
 
-const DEFAULT_PYTHON_VERSION = "3.7.*"
+const DEFAULT_PYTHON_VERSION = "3.8.*"
 
 type VersionsController struct {
 	*AppContext

config.yaml

@@ -26,7 +26,7 @@ ImageBuilderConfig:
       CPU: 2
       Memory: 1Gi
   PredictionJobBaseImages:
-    "3.7.*":
+    "3.8.*":
       ImageName: ghcr.io/caraml-dev/merlin-pyspark-base:v0.7.0
       DockerfilePath:  docker/app.Dockerfile
       BuildContextURI: ""

docs/reference/e2e-test.md

@@ -11,7 +11,7 @@ Run E2E test as part of github actions workflow. This E2E test will be triggered
 - Pull merlin repository
 - Pull mlp repository
 - Setup go
-- Setup python 3.7
+- Setup python 3.8
 - Setup cluster
 - Setup mlp namespace
 - Deploy mlp

docs/user-guide/batch_prediction.md

@@ -136,7 +136,7 @@ Following error is thrown during batch prediction execution
                              "values")
 E       AttributeError: Can only use .dt accessor with datetimelike values
 
-../../../.local/share/virtualenvs/merlin-pyspark-n4ybPFnE/lib/python3.7/site-packages/pandas/core/indexes/accessors.py:324: AttributeError
+../../../.local/share/virtualenvs/merlin-pyspark-n4ybPFnE/lib/python3.8/site-packages/pandas/core/indexes/accessors.py:324: AttributeError
 
 Assertion failed
 ```

examples/batch/env.yaml

@@ -2,6 +2,6 @@ dependencies:
 - pip:
     - joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
     - numpy>=1.19.5
-    - scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+    - scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
     - xgboost==1.6.2
     - mlflow==1.23.0

examples/batch/requirements.txt

@@ -1,6 +1,6 @@
 joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
 numpy>=1.19.5
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 xgboost==1.6.2
 mlflow==1.23.0
 merlin-sdk

examples/custom-model/http_json/requirements.txt

@@ -1,3 +1,3 @@
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 xgboost==1.6.2
 merlin-sdk

examples/metrics/env.yaml

@@ -1,2 +1,2 @@
 dependencies:
-  - python=3.7
+  - python=3.8

examples/model-endpoint/requirements.txt

@@ -1,4 +1,4 @@
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 merlin-sdk
 joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
 cloudpickle==2.0.0

examples/pyfunc-upi/env.yaml

@@ -1,5 +1,5 @@
 dependencies:
-  - python=3.7
+  - python=3.8
   - pip:
     - numpy
     - xgboost==1.6.2

examples/pyfunc-upi/requirements.txt

@@ -1,4 +1,4 @@
 xgboost==1.6.2
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 merlin-sdk>=0.22.2rc4
 numpy

examples/pyfunc/env.yaml

@@ -1,7 +1,7 @@
 dependencies:
-  - python=3.7
+  - python=3.8
   - pip:
     - joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
     - numpy
-    - scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+    - scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
     - xgboost==1.6.2

examples/pyfunc/requirements.txt

@@ -1,4 +1,4 @@
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
 xgboost==1.6.2
 merlin-sdk

examples/resource-request-gpu/requirements.txt

@@ -1,4 +1,4 @@
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 merlin-sdk
 joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
 cloudpickle==2.0.0

examples/resource-request/requirements.txt

@@ -1,4 +1,4 @@
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 merlin-sdk
 joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
 cloudpickle==2.0.0

examples/sklearn/requirements.txt

@@ -1,4 +1,4 @@
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 merlin-sdk
 joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
 cloudpickle==2.0.0

examples/transformer/custom-transformer/env.yaml

@@ -1,5 +1,5 @@
 dependencies:
-  - python=3.7
+  - python=3.8
   - pip:
     - torch==1.3.1
     - torchvision==0.4.2

examples/transformer/feast-enricher-transformer/env.yaml

@@ -1,2 +1,2 @@
 dependencies:
-  - python=3.7
+  - python=3.8

examples/transformer/standard-transformer/env.yaml

@@ -1,2 +1,2 @@
 dependencies:
-  - python=3.7
+  - python=3.8

examples/transformer/upi/env.yaml

@@ -1,2 +1,2 @@
 dependencies:
-  - python=3.7
+  - python=3.8

examples/xgboost/requirements.txt

@@ -1,4 +1,4 @@
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation-learn
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 xgboost==1.6.2
 merlin-sdk
 cloudpickle==2.0.0

python/batch-predictor/requirements.txt

@@ -4,4 +4,5 @@ mlflow>=1.2.0,<=1.23.0
 cloudpickle==2.0.0
 pyarrow>=0.14.1,<=9.0.0
 protobuf>=3.0,<4.0.0
-file:${SDK_PATH}#egg=merlin-sdk
+#TODO: Update merlin-sdk dep to: file:${SDK_PATH}#egg=merlin-sdk
+merlin-sdk==0.33.0 # Pin to the version that supports Python 3.7.

python/batch-predictor/requirements_test.txt

@@ -2,7 +2,7 @@ pytest
 pytest-cov
 mypy
 google-cloud-bigquery
-scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
 joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
 mypy-protobuf>=1.19
 types-PyYAML

python/batch-predictor/setup.py

@@ -21,11 +21,12 @@
 with open('requirements.txt') as f:
     REQUIRE = f.read().splitlines()
 
-merlin_path = os.path.join(os.getcwd(), "../sdk")
-merlin_sdk_package = "merlin-sdk"
-for index, item in enumerate(REQUIRE):
-    if merlin_sdk_package in item:
-        REQUIRE[index] = f"{merlin_sdk_package} @ file://localhost/{merlin_path}#egg={merlin_sdk_package}"
+# TODO: Uncomment below lines after Pyfunc server stops supporting Python 3.7
+# merlin_path = os.path.join(os.getcwd(), "../sdk")
+# merlin_sdk_package = "merlin-sdk"
+# for index, item in enumerate(REQUIRE):
+#     if merlin_sdk_package in item:
+#         REQUIRE[index] = f"{merlin_sdk_package} @ file://localhost/{merlin_path}#egg={merlin_sdk_package}"
 
 setup(
     name='merlin-pyspark-app',

python/batch-predictor/test-model/conda.yaml

@@ -1,4 +1,4 @@
 dependencies:
 - pip:
-  - scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+  - scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
   - joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version

python/pyfunc-scaffolding/{{cookiecutter.model_slug}}/env/conda.yaml

@@ -1,2 +1,2 @@
 dependencies:
-  - python=3.7
+  - python=3.8

python/pyfunc-server/pyfuncserver/utils/contants.py

@@ -11,14 +11,14 @@
 conda env.yaml that you pass to log_pyfunc method, e.g.:
 
 dependencies:
-- python=3.7
+- python=3.8
 - pip:
   - my-missing-package==1.0
   
 OR if the package is not available in PyPI distribution
 
 dependencies:
-- python=3.7
+- python=3.8
 - my-missing-package=1.0
 
 See: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file

python/pyfunc-server/requirements.txt

@@ -11,4 +11,5 @@ grpcio<1.49.0
 grpcio-reflection<1.49.0
 grpcio-tools<1.49.0
 grpcio-health-checking<1.49.0
-file:${SDK_PATH}#egg=merlin-sdk
+#TODO: Update merlin-sdk dep to: file:${SDK_PATH}#egg=merlin-sdk
+merlin-sdk==0.33.0 # Pin to the version that supports Python 3.7.

python/pyfunc-server/setup.py

@@ -30,11 +30,13 @@
 
 # replace merlin relative path in requirements.txt into absolute path
 # setuptools could not install relative path requirements
-merlin_path = os.path.join(os.getcwd(), "../sdk")
-merlin_sdk_package = "merlin-sdk"
-for index, item in enumerate(REQUIRE):
-    if merlin_sdk_package in item:
-        REQUIRE[index] = f"{merlin_sdk_package} @ file://localhost/{merlin_path}#egg={merlin_sdk_package}"
+
+# TODO: Uncomment below lines after Pyfunc server stops supporting Python 3.7
+# merlin_path = os.path.join(os.getcwd(), "../sdk")
+# merlin_sdk_package = "merlin-sdk"
+# for index, item in enumerate(REQUIRE):
+#     if merlin_sdk_package in item:
+#         REQUIRE[index] = f"{merlin_sdk_package} @ file://localhost/{merlin_path}#egg={merlin_sdk_package}"
 
 setup(
     name='pyfuncserver',

python/sdk/Dockerfile

@@ -1,4 +1,4 @@
-ARG PYTHON_VERSION=3.7
+ARG PYTHON_VERSION=3.8
 FROM python:${PYTHON_VERSION}-slim-buster
 
 LABEL org.opencontainers.image.source https://github.com/caraml-dev/merlin

python/sdk/setup.py

@@ -53,7 +53,7 @@
     "pytest",
     "recursive-diff>=1.0.0",
     "requests",
-    "scikit-learn==1.0.2",  # >=1.1.2 upon python 3.7 deprecation
+    "scikit-learn==1.0.2", #TODO: >=1.1.2 upon python 3.7 deprecation
     "types-python-dateutil",
     "types-PyYAML",
     "types-six",
@@ -76,7 +76,7 @@
     setup_requires=["setuptools_scm"],
     tests_require=TEST_REQUIRES,
     extras_require={'test': TEST_REQUIRES},
-    python_requires='>=3.7,<3.11',
+    python_requires='>=3.8,<3.11',
     long_description=open("README.md").read(),
     long_description_content_type='text/markdown',
     entry_points='''

python/sdk/test/batch/model/env.yaml

@@ -1,7 +1,7 @@
 dependencies:
   - pip:
       - mlflow
-      - scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+      - scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
       - joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
       - numpy>=1.19.5,<1.23.4 # https://github.com/Azure/MachineLearningNotebooks/issues/1314
       - pandas==1.3.5

python/sdk/test/invalid-models/pyfunc-model-invalid/MLmodel

@@ -4,5 +4,5 @@ flavors:
     env: conda.yaml
     loader_module: mlflow.pyfunc.model
     python_model: python_model.pkl
-    python_version: 3.7.5
+    python_version: 3.8.5
 utc_time_created: '2020-01-20 09:53:55.355722'

python/sdk/test/invalid-models/pyfunc-model-invalid/conda.yaml

@@ -1,7 +1,7 @@
 channels:
 - defaults
 dependencies:
-- python=3.7.5
+- python=3.8.5
 - pip:
   - mlflow
   - cloudpickle==2.0.0

python/sdk/test/pyfunc/env.yaml

@@ -3,7 +3,7 @@ dependencies:
   - pip:
     - joblib>=0.13.0,<1.2.0  # >=1.2.0 upon upgrade of kserve's version
     - numpy
-    - scikit-learn==1.0.2  # >=1.1.2 upon python 3.7 deprecation
+    - scikit-learn==1.0.2  #TODO: >=1.1.2 upon python 3.7 deprecation
     - xgboost==1.6.2
     - pytest
     - pytest-xdist==1.34.0

scripts/e2e/values-e2e.yaml

@@ -27,14 +27,14 @@ imageBuilder:
   k8sConfig: {}
   builderConfig:
     BaseImages:
-      3.7.*:
-        ImageName: pyfunc-py37:v0.31.3
+      3.8.*:
+        ImageName: pyfunc-py38:v0.31.3
         DockerfilePath: "docker/Dockerfile"
         BuildContextURI: "git://github.com/gojek/merlin.git#refs/tags/v0.31.3"
         MainAppPath: /merlin-spark-app/main.py
     PredictionJobBaseImages:
-      3.7.*:
-        ImageName: pyspark-py37:v0.31.3
+      3.8.*:
+        ImageName: pyspark-py38:v0.31.3
         DockerfilePath: "docker/app.Dockerfile"
         BuildContextURI: "git://github.com/gojek/merlin.git#refs/tags/v0.31.3"
         MainAppPath: /merlin-spark-app/main.py