DtCraft A High-performance Distributed Execution Engine at Scale
Recent years have seen rapid growth in datadriven distributed systems such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C++-based distributed execution engine to streamline the development of high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown the promising performance from single multi-core machines to clusters of computers. In a particular semiconductor design problem, we achieved 30Ã? speedup with 40 nodes and 15Ã? less development efforts over hand-crafted implementation.