MapReduce ML Model Deployment

Deploying an ML model in a MapReduce job.

Deploying an ML model in a MapReduce job.

This code is used in this blog post.

Installation

The makefile included with this project contains targets that help to automate several tasks.

To download the source code execute this command:

git clone https://github.com/schmidtbri/map-reduce-ml-model-deployment

Then create a virtual environment and activate it:

# go into the project directory
cd map-reduce-ml-model-deployment

make venv

source venv/bin/activate

Install the dependencies:

make dependencies

Running the unit tests

To run the unit test suite execute these commands:

# first install the test dependencies
make test-dependencies

# run the test suite
make test

Running the Job

To start the job execute these commands:

export PYTHONPATH=./
export APP_SETTINGS=ProdConfig
python model_mapreduce_job/ml_model_map_reduce_job.py \
    --model_qualified_name iris_model ./data/input.ldjson > \
    data/output.ldjson