Chunkflow: hybrid cloud processing of large 3D images by convolutional nets

Abstract

Automated microscopes with both high resolution and large field of view are generating terascale and even petascale 3D images. A local cluster might not have enough computational resources to process them in reasonable time, but public cloud platforms can provide computational resources on demand. Convolutional networks have become the state-of-the-art approach for 3D biological image analysis1,2 , and cloud processing by 3D convolutional nets has been used for processing independent small image stacks3–5 . However, cloud computing tools to perform distributed processing of terascale or petascale 3D images by convolutional nets are lacking. Here, we report chunkflow, a framework for distributing computational tasks over both cloud and local computational resources, including both GPUs and CPUs with multiple deep-learning framework back ends, to maximize efficiency, increase flexibility and reduce cost.