Skip to content

Commit

Permalink
Address review
Browse files Browse the repository at this point in the history
  • Loading branch information
Arief Rahmansyah committed Jan 15, 2024
1 parent c94fd32 commit b46f8fd
Show file tree
Hide file tree
Showing 11 changed files with 205 additions and 99 deletions.
189 changes: 174 additions & 15 deletions examples/pyfunc/Pyfunc.ipynb
View file

Large diffs are not rendered by default.

Binary file modified examples/pyfunc/sklearn-model/model_2.joblib
View file
Binary file not shown.
Binary file modified examples/pyfunc/xgboost-model/model_1.bst
View file
Binary file not shown.
67 changes: 24 additions & 43 deletions python/pyfunc-server/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,35 +5,37 @@ It leverages mlflow.pyfunc model for model loading.

## Usage

### HTTP Server
### HTTP Server

Run following command to load sample `echo-model` model and start HTTP server:

```bash
PROMETHEUS_MULTIPROC_DIR=prometheus \
python -m pyfuncserver --model_dir echo-model/model
```

This will start http server at port 8080 which you can test using curl command

```bash
curl localhost:8080/v1/models/model-1:predict -H "Content-Type: application/json" -d '{}'
```

### UPI V1 Server

Run following command to load sample `echo-model` model and start UPI v1 server:

```bash
PROMETHEUS_MULTIPROC_DIR=prometheus \
CARAML_PROTOCOL=UPI_V1 \
WORKERS=2 python -m pyfuncserver --model_dir echo-model/model
```


Since UPI v1 interface is gRPC then you can use grpcurl to send request

```bash
grpcurl -plaintext -d '{}' localhost:9000 caraml.upi.v1.UniversalPredictionService/PredictValues
```


## Development

Requirements:
Expand All @@ -54,48 +56,29 @@ make test
```

To run benchmark
```bash
make benchmark
```

## Building Docker Image

To create docker image locally you'll need to first download model artifact.

```bash
gsutil cp -r gs://bucket-name/mlflow/11/68eb8538374c4053b3ecad99a44170bd/artifacts/model .
```

Build the docker image

```bash
docker build -t mymodel:latest -f docker/local.Dockerfile .
```

And run the model service

```bash
docker run -e MODEL_NAME=model -p 8080:8080 mymodel:latest
make benchmark
```

## Configuration

Pyfunc server can be configured via following environment variables

| Environment Variable | Description |
| ------------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| CARAML_PROTOCOL | Protocol to be used, the valid values are `HTTP_JSON` and `UPI_V1` |
| CARAML_HTTP_PORT | Pyfunc server will start http server listening to this port when `CARAML_PROTOCOL` = `HTTP_JSON` |
| CARAML_GRPC_PORT | Pyfunc server will start grpc server listening to this port when `CARAML_PROTOCOL` = `UPI_V1` |
| CARAML_MODEL_NAME | Model name |
| CARAML_MODEL_VERSION | Model version |
| CARAML_MODEL_FULL_NAME | Model full name in the format of `${CARAML_MODEL_NAME}-${CARAML_MODEL_FULL_NAME}` |
| WORKERS | Number of Python processes that will be created to allow multi processing (default = 1) |
| LOG_LEVEL | Log level, valid values are `INFO`, `ERROR`, `DEBUG`, `WARN`, `CRITICAL` (default='INFO') |
| GRPC_OPTIONS | GRPC options to configure UPI server as json string. The possible options can be found in [grpc_types.h](https://github.com/grpc/grpc/blob/v1.46.x/include/grpc/impl/codegen/grpc_types.h). Example: '{"grpc.max_concurrent_streams":100}' |
| GRPC_CONCURRENCY | Size of grpc handler threadpool per worker (default = 10) |
| PUSHGATEWAY_ENABLED | Enable pushing metrics to prometheus push gateway, only available when `CARAML_PROTOCOL` is set to `UPI_V1` (default = false) |
| PUSHGATEWAY_URL | Url of the prometheus push gateway (default = localhost:9091) |
| Environment Variable | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| CARAML_PROTOCOL | Protocol to be used, the valid values are `HTTP_JSON` and `UPI_V1` |
| CARAML_HTTP_PORT | Pyfunc server will start http server listening to this port when `CARAML_PROTOCOL` = `HTTP_JSON` |
| CARAML_GRPC_PORT | Pyfunc server will start grpc server listening to this port when `CARAML_PROTOCOL` = `UPI_V1` |
| CARAML_MODEL_NAME | Model name |
| CARAML_MODEL_VERSION | Model version |
| CARAML_MODEL_FULL_NAME | Model full name in the format of `${CARAML_MODEL_NAME}-${CARAML_MODEL_FULL_NAME}` |
| WORKERS | Number of Python processes that will be created to allow multi processing (default = 1) |
| LOG_LEVEL | Log level, valid values are `INFO`, `ERROR`, `DEBUG`, `WARN`, `CRITICAL` (default='INFO') |
| GRPC_OPTIONS | GRPC options to configure UPI server as json string. The possible options can be found in [grpc_types.h](https://github.com/grpc/grpc/blob/v1.46.x/include/grpc/impl/codegen/grpc_types.h). Example: '{"grpc.max_concurrent_streams":100}' |
| GRPC_CONCURRENCY | Size of grpc handler threadpool per worker (default = 10) |
| PUSHGATEWAY_ENABLED | Enable pushing metrics to prometheus push gateway, only available when `CARAML_PROTOCOL` is set to `UPI_V1` (default = false) |
| PUSHGATEWAY_URL | Url of the prometheus push gateway (default = localhost:9091) |
| PUSHGATEWAY_PUSH_INTERVAL_SEC | Interval in seconds for pushing metrics to prometheus push gateway (default = 30) |

## Directory Structure
Expand All @@ -104,10 +87,8 @@ Pyfunc server can be configured via following environment variables
├── benchmark <- Benchmarking artifacts
├── docker <- Dockerfiles and environment files
├── Dockerfile <- Dockerfile that will be used by kaniko to build user image in the cluster
├── base.Dockerfile <- Base docker image that will be used by `Dockerfile` and `local.Dockerfile`
├── local.Dockerfile <- Dockerfile that can be used to perform local testing
├── envXY.yaml <- Conda environment for python version X.Y that will be created within `base.Dockerfile`
├── echo-model <- Simple model for testing
├── base.Dockerfile <- Base docker image that will be used by `Dockerfile`
├── examples <- Examples of PyFunc models implementation
├── test <- Test package
├── pyfuncserver <- Source code of this workflow
│ ├── __main__.py <- Entry point of pyfuncserver
Expand All @@ -120,10 +101,10 @@ Pyfunc server can be configured via following environment variables
│ └── rest <- Server implementation for HTTP_JSON protocol
│ └── upi <- Server implementation for UPI_V1 protocol
├── .gitignore
├── Makefile <- Makefile
├── Makefile <- Makefile
├── README.md <- The top-level README for developers using this project.
├── requirements.txt <- pyfuncserver dependencies
├── setup.py <- setup.py
├── run.sh <- Script to activate `merlin-model` environment and run pyfuncserver when `docker run` is invoked
```
```
36 changes: 0 additions & 36 deletions python/pyfunc-server/docker/local.Dockerfile
View file

This file was deleted.

0 ...n/pyfunc-server/examples/iris/__init__.py → ...unc-server/examples/iris_http/__init__.py
View file
File renamed without changes.
0 python/pyfunc-server/examples/iris/env.yaml → ...pyfunc-server/examples/iris_http/env.yaml
View file
File renamed without changes.
4 changes: 2 additions & 2 deletions python/pyfunc-server/examples/iris/iris.py → ...nc-server/examples/iris_http/iris_http.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ def train_models():
model_instance=IrisModel(),
conda_env="env.yaml",
artifacts={
"xgb_model": "models/model_1.bst",
"sklearn_model": "models/model_2.joblib",
"xgb_model": XGB_PATH,
"sklearn_model": SKLEARN_PATH,
},
)
0 ...c-server/examples/iris/models/model_1.bst → ...ver/examples/iris_http/models/model_1.bst
View file
File renamed without changes.
Binary file renamed ...erver/examples/iris/models/model_2.joblib → .../examples/iris_http/models/model_2.joblib
View file
Binary file not shown.
8 changes: 5 additions & 3 deletions python/sdk/merlin/pyfunc.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@

import docker
import grpc
import mlflow
import numpy
import pandas
from caraml.upi.v1 import upi_pb2
Expand All @@ -17,6 +16,8 @@
from merlin.version import VERSION
from mlflow.pyfunc import PythonModel

import mlflow

PYFUNC_EXTRA_ARGS_KEY = "__EXTRA_ARGS__"
PYFUNC_MODEL_INPUT_KEY = "__INPUT__"
PYFUNC_PROTOCOL_KEY = "__PROTOCOL__"
Expand Down Expand Up @@ -468,7 +469,7 @@ def run_pyfunc_local_server(
pyfunc_base_image: str = None,
port: int = 8080,
env_vars: Dict[str, str] = None,
protocol: Protocol = None,
protocol: Protocol = Protocol.HTTP_JSON,
debug: bool = False,
):
if pyfunc_base_image is None:
Expand Down Expand Up @@ -557,7 +558,7 @@ def _run_container(
model_full_name,
port,
env_vars: Dict[str, str] = None,
protocol: Protocol = None,
protocol: Protocol = Protocol.HTTP_JSON,
):
docker_client = docker.from_env()

Expand All @@ -580,6 +581,7 @@ def _run_container(

ports = {"8080/tcp": port}
if protocol == Protocol.UPI_V1:
env_vars["CARAML_PROTOCOL"] = protocol.value
ports = {"9000/tcp": port}

try:
Expand Down

0 comments on commit b46f8fd

Please sign in to comment.