Amazon SageMaker Operators for Kubernetes are operators that can be used to train machine learning models, optimize hyperparameters for a given model, run batch transform jobs over existing models, and set up inference endpoints. With these operators, users can manage their jobs in Amazon SageMaker from their Kubernetes cluster in Amazon Elastic Kubernetes Service EKS.
First, you must install the operators. After installation is complete, create a TrainingJob YAML specification by following one of the samples, like samples/xgboost-mnist-trainingjob.yaml. Then, use kubectl to create and monitor the progress of your job:
$ kubectl apply -f xgboost-mnist-trainingjob.yaml
trainingjob.sagemaker.aws.amazon.com/xgboost-mnist created
$ kubectl get trainingjob
NAME STATUS SECONDARY-STATUS CREATION-TIME SAGEMAKER-JOB-NAME
xgboost-mnist InProgress Starting 2019-11-26T23:38:11Z xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06Once the job starts training, you can use a kubectl plugin to stream training logs:
$ kubectl get trainingjob
NAME STATUS SECONDARY-STATUS CREATION-TIME SAGEMAKER-JOB-NAME
xgboost-mnist InProgress Training 2019-11-26T23:38:11Z xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06
$ kubectl smlogs trainingjob xgboost-mnist | head -n 5
"xgboost-mnist" has SageMaker TrainingJobName "xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06" in region "us-east-2", status "InProgress" and secondary status "Training"
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.449 -0800 PST Arguments: train
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.449 -0800 PST [2019-11-26:23:41:10:INFO] Running standalone xgboost training.
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.45 -0800 PST [2019-11-26:23:41:10:INFO] File size need to be processed in the node: 1122.95mb. Available memory size in the node: 8501.08mb
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.45 -0800 PST [2019-11-26:23:41:10:INFO] Determined delimiter of CSV input is ','
xgboost-mnist-cf1e16fb10a511eaaa450a350733ba06/algo-1-1574811611 2019-11-26 15:41:13.45 -0800 PST [23:41:10] S3DistributionType set as FullyReplicatedThe Amazon SageMaker Operators for Kubernetes enable management of SageMaker TrainingJobs, HyperParameterTuningJobs, BatchTransformJobs and HostingDeployments (Endpoints). Create and monitor them using the same kubectl tool as above.
To install the operators onto your Kubernetes cluster, follow our User Guide.
To make a YAML spec, follow one of the below examples as a guide. Replace values like RoleARN, S3 input buckets and S3 output buckets with values that correspond to your account.
amazon-sagemaker-operator-for-k8s is an open source project. See CONTRIBUTING for details.
This project is distributed under the Apache License, Version 2.0, see LICENSE and NOTICE for more information.