Skip to main content

Hardware-in-the-Loop End-to-End Optimization of Camera Image Processing Pipelines

Author(s): Mosleh, Ali; Sharma, Avinash; Onzon, Emmanuel; Mannan, Fahim; Robidoux, Nicolas; et al

To refer to this page use:
Abstract: Commodity imaging systems rely on hardware image signal processing (ISP) pipelines. These low-level pipelines consist of a sequence of processing blocks that, depending on their hyperparameters, reconstruct a color image from RAW sensor measurements. Hardware ISP hyperparameters have a complex interaction with the output image, and therefore with the downstream application ingesting these images. Traditionally, ISPs are manually tuned in isolation by imaging experts without an end-to-end objective. Very recently, ISPs have been optimized with 1st-order methods that require differentiable approximations of the hardware ISP. Departing from such approximations, we present a hardware-in-the-loop method that directly optimizes hardware image processing pipelines for end-to-end domain-specific losses by solving a nonlinear multi-objective optimization problem with a novel 0th-order stochastic solver directly interfaced with the hardware ISP. We validate the proposed method with recent hardware ISPs and 2D object detection, segmentation, and human viewing as end-to-end downstream tasks. For automotive 2D object detection, the proposed method outperforms manual expert tuning by 30% mean average precision (mAP) and recent methods using ISP approximations by 18% mAP.
Publication Date: 2020
Citation: Mosleh, Ali, Avinash Sharma, Emmanuel Onzon, Fahim Mannan, Nicolas Robidoux, and Felix Heide. "Hardware-in-the-Loop End-to-End Optimization of Camera Image Processing Pipelines." In IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020): pp. 7526-7535. doi:10.1109/CVPR42600.2020.00755
DOI: 10.1109/CVPR42600.2020.00755.
ISSN: 1063-6919
EISSN: 2575-7075
Pages: 7526 - 7535
Type of Material: Conference Article
Journal/Proceeding Title: IEEE/CVF Conference on Computer Vision and Pattern Recognition
Version: Author's manuscript

Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.