Skip to main content

Riffle: optimized shuffle service for large-scale data analytics

Author(s): Zhang, Haoyu; Cho, Brian; Seyfe, Ergin; Ching, Avery; Freedman, Michael J

To refer to this page use:
Abstract: The rapidly growing size of data and complexity of analytics present new challenges for large-scale data processing systems. Modern systems keep data partitions in memory for pipelined operators, and persist data across stages with wide dependencies on disks for fault tolerance. While processing can often scale well by splitting jobs into smaller tasks for better parallelism, all-to-all data transfer---called shuffle operations---become the scaling bottleneck when running many small tasks in multi-stage data analytics jobs. Our key observation is that this bottleneck is due to the superlinear increase in disk I/O operations as data volume increases. We present Riffle, an optimized shuffle service for big-data analytics frameworks that significantly improves I/O efficiency and scales to process petabytes of data. To do so, Riffle efficiently merges fragmented intermediate shuffle files into larger block files, and thus converts small, random disk I/O requests into large, sequential ones. Riffle further improves performance and fault tolerance by mixing both merged and unmerged block files to minimize merge operation overhead. Using Riffle, Facebook production jobs on Spark clusters with over 1,000 executors experience up to a 10x reduction in the number of shuffle I/O requests and 40% improvement in the end-to-end job completion time.
Publication Date: Apr-2018
Citation: Zhang, Haoyu, Brian Cho, Ergin Seyfe, Avery Ching, and Michael J. Freedman. "Riffle: optimized shuffle service for large-scale data analytics." In Proceedings of the Thirteenth EuroSys Conference (2018): pp. 1-15. doi:10.1145/3190508.3190534
DOI: 10.1145/3190508.3190534
Pages: 1 - 15
Type of Material: Conference Article
Journal/Proceeding Title: Proceedings of the Thirteenth EuroSys Conference
Version: Final published version. This is an open access article.

Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.