BE/BTech & ME/MTech Final Year Projects for Computer Science | Information Technology | ECE Engineer | IEEE Projects Topics, PHD Projects Reports, Ideas and Download | Sai Info Solution | Nashik |Pune |Mumbai
director@saiinfo settings_phone02536644344 settings_phone+919270574718 +919096813348 settings_phone+917447889268
logo


SAI INFO SOLUTION


Diploma | BE |B.Tech |ME | M.Tech |PHD

Project Development and Training

Search Project by Domainwise


An Adaptive Ef?ciency-Fairness Meta-scheduler for Data-Intensive Computing


3D Reconstruction in Canonical

Scalable and Secure Big Data I
Abstract


In data-intensive cluster computing platforms such as Hadoop YARN, ef?ciency and fairness are two important factors for system design and optimizations. Previous studies are either for ef?ciency or for fairness solely, without considering the tradeoff between ef?ciency and fairness. Recent studies observe that there is a tradeoff between ef?ciency and fairness because of resource contention between users/jobs. By leveraging the existing schedulers, a meta-scheduler is able to dynamically choose one of them for job/task scheduling at runtime. In this paper, we propose a meta-scheduler called FLEX to realize the tradeoff between system ef?ciency and fairness in Hadoop YARN. FLEX combines multiple existing schedulers into a single aggregated view without any modi?cation on the original schedulers. Equipped with these candidate schedulers, FLEX utilizes machine learning approach to adaptively choose the most proper scheduler according to the characteristic of current running workload and user-de?ned SLA (Service Level Agreement). We implement FLEX in Hadoop YARN. We conduct experiments with real deployment in a local cluster and perform simulation studies with production traces. Experimental results show that the FLEX outperforms the state-of-the-art approach in two aspects: 1) Given a prede?ned threshold on the fairness loss, the FLEX reduces the makespan by up to 22% and 24% in real deployment and the large-scale simulation, respectively; 2) Given the prede?ned threshold on the makespan reduction, the FLEX reduces the fairness loss by up to 75% and 73% in real deployment and the large-scale simulation, respectively.

KeyWords
meta-scheduling, ef?ciency-fairness tradeoff, data-intensive, Hadoop YARN



Share
Share via WhatsApp
BE/BTech & ME/MTech Final Year Projects for Computer Science | Information Technology | ECE Engineer | IEEE Projects Topics, PHD Projects Reports, Ideas and Download | Sai Info Solution | Nashik |Pune |Mumbai
Call us : 09096813348 / 02536644344
Mail ID : developer.saiinfo@gmail.com
Skype ID : saiinfosolutionnashik