Presentation
Swift Machine Learning Model Serving Scheduling: A Region Based Reinforcement Learning Approach
SessionCloud Scheduling
Event Type
Paper
TP
Algorithms
Clouds and Distributed Computing
Machine Learning
Performance
Resiliency
Scheduling
TimeTuesday, 19 November 20192pm - 2:30pm
Location405-406-407
DescriptionThe success of machine learning has prospered Machine-Learning-as-a-Service (MLaaS) -- deploying trained machine learning (ML) models in cloud to provide low latency inference services at scale. To meet latency Service-Level-Objective (SLO), judicious parallelization at both request and operation levels is utterly important. However, existing ML systems (e.g., Tensorflow) and cloud ML serving platforms (e.g., SageMaker) are SLO-agnostic and rely on users to manually configure the parallelism. To provide low latency ML serving, this paper proposes a swift machine learning serving scheduling framework with a novel Region-based Reinforcement Learning (RRL) approach. RRL can efficiently identify the optimal parallelism configuration under different workloads by estimating performance of similar configurations with that of the known ones. We both theoretically and experimentally show that the RRL approach can outperform state-of-the-art approaches by finding near optimal solutions over 8 times faster while reducing inference latency up to 79.0% and reducing SLO violation up to 49.9%.
Download PDF
Archive
