Skip to main content

Hardware-in-the-Loop End-to-End Optimization of Camera Image Processing Pipelines

Author(s): Mosleh, Ali; Sharma, Avinash; Onzon, Emmanuel; Mannan, Fahim; Robidoux, Nicolas; et al

To refer to this page use:
Full metadata record
DC FieldValueLanguage
dc.contributor.authorMosleh, Ali-
dc.contributor.authorSharma, Avinash-
dc.contributor.authorOnzon, Emmanuel-
dc.contributor.authorMannan, Fahim-
dc.contributor.authorRobidoux, Nicolas-
dc.contributor.authorHeide, Felix-
dc.identifier.citationMosleh, Ali, Avinash Sharma, Emmanuel Onzon, Fahim Mannan, Nicolas Robidoux, and Felix Heide. "Hardware-in-the-Loop End-to-End Optimization of Camera Image Processing Pipelines." In IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020): pp. 7526-7535. doi:10.1109/CVPR42600.2020.00755en_US
dc.description.abstractCommodity imaging systems rely on hardware image signal processing (ISP) pipelines. These low-level pipelines consist of a sequence of processing blocks that, depending on their hyperparameters, reconstruct a color image from RAW sensor measurements. Hardware ISP hyperparameters have a complex interaction with the output image, and therefore with the downstream application ingesting these images. Traditionally, ISPs are manually tuned in isolation by imaging experts without an end-to-end objective. Very recently, ISPs have been optimized with 1st-order methods that require differentiable approximations of the hardware ISP. Departing from such approximations, we present a hardware-in-the-loop method that directly optimizes hardware image processing pipelines for end-to-end domain-specific losses by solving a nonlinear multi-objective optimization problem with a novel 0th-order stochastic solver directly interfaced with the hardware ISP. We validate the proposed method with recent hardware ISPs and 2D object detection, segmentation, and human viewing as end-to-end downstream tasks. For automotive 2D object detection, the proposed method outperforms manual expert tuning by 30% mean average precision (mAP) and recent methods using ISP approximations by 18% mAP.en_US
dc.format.extent7526 - 7535en_US
dc.relation.ispartofIEEE/CVF Conference on Computer Vision and Pattern Recognitionen_US
dc.rightsAuthor's manuscripten_US
dc.titleHardware-in-the-Loop End-to-End Optimization of Camera Image Processing Pipelinesen_US
dc.typeConference Articleen_US

Files in This Item:
File Description SizeFormat 
HardwareInLoopOptimizCameraImageProcessPipelines.pdf9.76 MBAdobe PDFView/Download

Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.