Dexter: A Performance-Cost Efficient Resource Allocation Manager for Serverless Data Analytics

Leveraging serverless platforms for the efficient execution of distributed data analytics frameworks, such as Apache Spark [3], has gained substantial interest since early 2022. The elasticity, free-of-management, and on-demand scalability offered by serverless have motivated the effort in deploying distributed data analytics applications to serverless platforms. However, auto-scaling resources for such complex workloads so that we can fully benefit from the resource elasticity of serverless is a non-trivial challenge. Mis-configuration can result in severe performance and cost issues arising from resource under- and over-provisioning. In this paper, we present Dexter, a robust resource allocation manager continuously monitoring the execution of applications and dynamically allocating resources at a fine-grained level to guarantee performance-cost efficiency (optimizing total runtime cost). Dexter is novel in combining predictive and reactive strategies that fully leverage the elasticity of serverless to enhance the performance-cost efficiency for workflow executions. Unlike black-box ML models, Dexter quickly reaches a sufficiently good solution, prioritizing simplicity, generality, and ease of understanding. Our experimental evaluation shows that, compared with the default serverless Spark resource allocation that dynamically requests exponentially more executors to accommodate pending tasks, our solution achieves a significant cost reduction of up to 4.65×, while improving resource efficiency up to 3.50×. Dexter also enables a substantial resource saving, demanding up to 5.75× fewer resources.