Skip to main content

Harnessing the Power of Many: Extensible Toolkit for Scalable Ensemble Applications

Author(s): Balasubramanian, Vivek; Turilli, Matteo; Hu, Weiming; Lefebvre, Matthieu; Lei, Wenjie; et al

To refer to this page use:
Abstract: Many scientific problems require multiple distinct computational tasks to be executed in order to achieve a desired solution. We introduce the Ensemble Toolkit (EnTK) to address the challenges of scale, diversity and reliability they pose. We describe the design and implementation of EnTK, characterize its performance and integrate it with two exemplar use cases: seismic inversion and adaptive analog ensembles. We perform nine experiments, characterizing EnTK overheads, strong and weak scalability, and the performance of the two use case imple-mentations, at scale and on production infrastructures. We show how EnTK meets the following general requirements: (i) imple-menting dedicated abstractions to support the description and execution of ensemble applications; (ii) support for execution on heterogeneous computing infrastructures; (iii) efficient scalability up to O(10 4 ) tasks; and (iv) task-level fault tolerance. We discuss novel computational capabilities that EnTK enables and the scientific advantages arising thereof. We propose EnTK as an important addition to the suite of tools in support of production scientific computing.
Publication Date: 6-Aug-2018
Citation: Balasubramanian, Vivek, Matteo Turilli, Weiming Hu, Matthieu Lefebvre, Wenjie Lei, Ryan Modrak, Guido Cervone, Jeroen Tromp, and Shantenu Jha. "Harnessing the power of many: Extensible toolkit for scalable ensemble applications." In 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS), IEEE (2018): 536-545. doi:10.1109/IPDPS.2018.00063.
DOI: doi:10.1109/IPDPS.2018.00063
Pages: 536 - 545
Type of Material: Journal Article
Journal/Proceeding Title: 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Version: Author's manuscript

Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.