preload pyfunc v2 model to avoid pydantic schema generation error

[khorshuheng]

Description

Due to a bug that was resolved in https://github.com/mlflow/mlflow/pull/8722/files , the current version of Mlflow used by Merlin can potentially introduce some side effect by removing package from sys.modules.

One such side effect occurs when the user's Mlflow Pyfunc code happens to contain a subdirectory named typing . When appending the Pyfunc code to sys.path, Mlflow also unintentionally remove the typing standard module due to name clash, which clear some module level cache.

We have recently migrated from swagger to OpenAPI for code generation, and as a result, the Merlin Pyfunc model (in merlin.model) is now depending on Pydantic model. The removal of typing standard module from sys.modules affect pydantic ability to resolve some field types correctly, which results in schema generation error.

Ideally, we should upgrade Mlflow version to resolve this. But the fix is only introduce is version >2.0, which makes migration difficult. Hence, the alternative is to import Merlin pyfunc model prior to calling pyfunc.load_model. Importing the Pyfunc model allows pydantic to cache the schema for the class, which allows Pyfunc model to be instantiated correctly even though the typing modules is removed from sys.modules.

Modifications

Import Pyfunc models prior to calling pyfunc.load_model.

Tests

Checklist

• Added PR label
• Added unit test, integration, and/or e2e tests
• Tested locally
• Updated documentation
• Update Swagger spec if the PR introduce API changes
• Regenerated Golang and Python client if the PR introduces API changes

Release Notes

[ghost]

👇 Click on the image for a new way to code review

Legend

[leonlnj]

Wow thanks for catching this obscure one.

[khorshuheng]

Update: while this resolves the issue of image building, it still doesn't prevent the batch prediction job spark executor from failing.