To refer to this page use:
|Abstract:||Commodity imaging systems rely on hardware image signal processing (ISP) pipelines. These low-level pipelines consist of a sequence of processing blocks that, depending on their hyperparameters, reconstruct a color image from RAW sensor measurements. Hardware ISP hyperparameters have a complex interaction with the output image, and therefore with the downstream application ingesting these images. Traditionally, ISPs are manually tuned in isolation by imaging experts without an end-to-end objective. Very recently, ISPs have been optimized with 1st-order methods that require differentiable approximations of the hardware ISP. Departing from such approximations, we present a hardware-in-the-loop method that directly optimizes hardware image processing pipelines for end-to-end domain-specific losses by solving a nonlinear multi-objective optimization problem with a novel 0th-order stochastic solver directly interfaced with the hardware ISP. We validate the proposed method with recent hardware ISPs and 2D object detection, segmentation, and human viewing as end-to-end downstream tasks. For automotive 2D object detection, the proposed method outperforms manual expert tuning by 30% mean average precision (mAP) and recent methods using ISP approximations by 18% mAP.|
|Citation:||Mosleh, Ali, Avinash Sharma, Emmanuel Onzon, Fahim Mannan, Nicolas Robidoux, and Felix Heide. "Hardware-in-the-Loop End-to-End Optimization of Camera Image Processing Pipelines." In IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020): pp. 7526-7535. doi:10.1109/CVPR42600.2020.00755|
|Pages:||7526 - 7535|
|Type of Material:||Conference Article|
|Journal/Proceeding Title:||IEEE/CVF Conference on Computer Vision and Pattern Recognition|
Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.