An Adaptive Efï¬?ciency-Fairness Meta-scheduler for Data-Intensive Computing
In data-intensive cluster computing platforms such as Hadoop YARN, efï¬?ciency and fairness are two important factors for system design and optimizations. Previous studies are either for efï¬?ciency or for fairness solely, without considering the tradeoff between efï¬?ciency and fairness. Recent studies observe that there is a tradeoff between efï¬?ciency and fairness because of resource contention between users/jobs. By leveraging the existing schedulers, a meta-scheduler is able to dynamically choose one of them for job/task scheduling at runtime. In this paper, we propose a meta-scheduler called FLEX to realize the tradeoff between system efï¬?ciency and fairness in Hadoop YARN. FLEX combines multiple existing schedulers into a single aggregated view without any modiï¬?cation on the original schedulers. Equipped with these candidate schedulers, FLEX utilizes machine learning approach to adaptively choose the most proper scheduler according to the characteristic of current running workload and user-deï¬?ned SLA (Service Level Agreement). We implement FLEX in Hadoop YARN. We conduct experiments with real deployment in a local cluster and perform simulation studies with production traces. Experimental results show that the FLEX outperforms the state-of-the-art approach in two aspects: 1) Given a predeï¬?ned threshold on the fairness loss, the FLEX reduces the makespan by up to 22% and 24% in real deployment and the large-scale simulation, respectively; 2) Given the predeï¬?ned threshold on the makespan reduction, the FLEX reduces the fairness loss by up to 75% and 73% in real deployment and the large-scale simulation, respectively.
meta-scheduling, efï¬?ciency-fairness tradeoff, data-intensive, Hadoop YARN