[{"data":1,"prerenderedAt":281},["ShallowReactive",2],{"content-query-YkFx13lhKB":3},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"cover":11,"type":12,"category":13,"body":14,"_type":275,"_id":276,"_source":277,"_file":278,"_stem":279,"_extension":280},"/technology-blogs/en/3409","en",false,"","Idea Sharing | High-Quality Specular Highlight Removal Using MindSpore","Leveraging physics-based modeling and deep learning techniques to progressively refine highlight removal, producing a specular-free image with consistent color tone to the original input.","2024-05-31","https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/11/28/a4693a6e9c424b44a30c3f331725113d.png","technology-blogs","Practices",{"type":15,"children":16,"toc":272},"root",[17,25,35,40,48,53,61,72,80,89,94,102,110,115,120,128,133,138,146,151,158,163,168,173,178,183,190,195,203,208,215,220,225,232,237,244,249,254,262,267],{"type":18,"tag":19,"props":20,"children":22},"element","h1",{"id":21},"idea-sharing-high-quality-specular-highlight-removal-using-mindspore",[23],{"type":24,"value":8},"text",{"type":18,"tag":26,"props":27,"children":28},"p",{},[29],{"type":18,"tag":30,"props":31,"children":32},"strong",{},[33],{"type":24,"value":34},"Paper Title",{"type":18,"tag":26,"props":36,"children":37},{},[38],{"type":24,"value":39},"Towards High-Quality Specular Highlight Removal by Leveraging Large-scale Synthetic Data",{"type":18,"tag":26,"props":41,"children":42},{},[43],{"type":18,"tag":30,"props":44,"children":45},{},[46],{"type":24,"value":47},"Source",{"type":18,"tag":26,"props":49,"children":50},{},[51],{"type":24,"value":52},"ICCV2023",{"type":18,"tag":26,"props":54,"children":55},{},[56],{"type":18,"tag":30,"props":57,"children":58},{},[59],{"type":24,"value":60},"Paper URL",{"type":18,"tag":26,"props":62,"children":63},{},[64],{"type":18,"tag":65,"props":66,"children":70},"a",{"href":67,"rel":68},"https://arxiv.org/pdf/2309.06302.pdf",[69],"nofollow",[71],{"type":24,"value":67},{"type":18,"tag":26,"props":73,"children":74},{},[75],{"type":18,"tag":30,"props":76,"children":77},{},[78],{"type":24,"value":79},"Code URL",{"type":18,"tag":26,"props":81,"children":82},{},[83],{"type":18,"tag":65,"props":84,"children":87},{"href":85,"rel":86},"https://github.com/nauyihsnehs/TSHRNet-MindSpore",[69],[88],{"type":24,"value":85},{"type":18,"tag":26,"props":90,"children":91},{},[92],{"type":24,"value":93},"The MindSpore community supports analysis on top-level conference papers and promotes original AI achievements. In this blog, I'd like to share the paper of the team led by Pro. Xiao Chunxia, School of Computer Science, Wuhan University.",{"type":18,"tag":26,"props":95,"children":96},{},[97],{"type":18,"tag":98,"props":99,"children":101},"img",{"alt":7,"src":100},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/10/11/c99c8105c10847d79b24cf8e9810b358.png",[],{"type":18,"tag":26,"props":103,"children":104},{},[105],{"type":18,"tag":30,"props":106,"children":107},{},[108],{"type":24,"value":109},"01 Research Background",{"type":18,"tag":26,"props":111,"children":112},{},[113],{"type":24,"value":114},"Specular highlights are a common occurrence in the real world, but they can significantly impact the quality of images for photography. Their presence often obscures important visual details, such as skin texture in portraits or intricate patterns in documents. Removing these highlights from a single image not only restores visual content but also enhances its overall perception. This process has wide-reaching implications for various applications, including recoloring, light source estimation, specular object recognition, and intrinsic image decomposition, highlighting its significance in computer vision and computer graphics.",{"type":18,"tag":26,"props":116,"children":117},{},[118],{"type":24,"value":119},"Existing solutions typically fall into two categories: traditional methods based on intensity and chromaticity analysis, and more recent deep learning-based approaches. Traditional methods often struggle to accurately interpret the semantic context of a scene, leading to unsatisfactory results characterized by visual artifacts and loss of detail. While deep learning-based techniques have shown promise, they still face challenges in effectively removing specular highlights under complex lighting conditions and with varying object materials.",{"type":18,"tag":26,"props":121,"children":122},{},[123],{"type":18,"tag":30,"props":124,"children":125},{},[126],{"type":24,"value":127},"02 Team Introduction",{"type":18,"tag":26,"props":129,"children":130},{},[131],{"type":24,"value":132},"Fu Gang, the first author of this paper, graduated in 2022 with a PhD from the School of Computer Science at Wuhan University under the supervision of Professor Xiao Chunxia. His research focuses on lighting processing and editing within the fields of computer vision and graphics. He has investigated several key subproblems including specular highlight detection and removal, intrinsic image decomposition, shadow removal, and relighting.",{"type":18,"tag":26,"props":134,"children":135},{},[136],{"type":24,"value":137},"Xiao Chunxia, correspondent author of the paper, positions as a professor of the School of Computer Science at Wuhan University. He is honored as distinguished talent by the Ministry of Education, and mainly engaged in research in computer graphics, virtual reality, augmented reality, and computer vision. He has published more than 160 papers, including over 80 papers in international authoritative or SCI academic journals such as TOG, TPAMI, IJCV, and TVCG, and over 30 papers at top academic conferences such as CVPR, ICCV, ECCV and AAAI. More over, he has been granted a number of national patents and software copyrights, awarded with several national natural science awards and presided in multiple national major science projects.",{"type":18,"tag":26,"props":139,"children":140},{},[141],{"type":18,"tag":30,"props":142,"children":143},{},[144],{"type":24,"value":145},"03 Introduction to the Paper",{"type":18,"tag":26,"props":147,"children":148},{},[149],{"type":24,"value":150},"This paper presents a novel three-stage framework for handling specular highlights in images. The framework leverages physics-based modeling and deep learning techniques to progressively refine highlight removal, ultimately producing a specular-free image with consistent color tone to the original input.",{"type":18,"tag":26,"props":152,"children":153},{},[154],{"type":18,"tag":98,"props":155,"children":157},{"alt":7,"src":156},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/10/11/d6eb103cb5ff40d681159c533bc254d3.png",[],{"type":18,"tag":26,"props":159,"children":160},{},[161],{"type":24,"value":162},"Figure 1: Three-stage specular highlight removal framework",{"type":18,"tag":26,"props":164,"children":165},{},[166],{"type":24,"value":167},"Stage 1: Physics-based specular highlight removal (PSHR). The input image is first decomposed into three fundamental components: albedo, shading, and specular residue. This decomposition stems from a physical image formation model, which assumes that an image comprises these three components. Two encoder-decoder networks are employed to estimate albedo and shading, respectively. A preliminary specular-free image is then generated by multiplying these two estimations.",{"type":18,"tag":26,"props":169,"children":170},{},[171],{"type":24,"value":172},"Stage 2: Specular-free refinement (SR). With the preliminary specular-free image, this stage utilizes another encoder-decoder network to further optimize the image, improving visual artifacts such as color distortions. This stage focuses on enhancing detail preservation and natural appearance to optimize the final image quality.",{"type":18,"tag":26,"props":174,"children":175},{},[176],{"type":24,"value":177},"Stage 3: Tone correction (TC). After refinement, the overall tone of the image might deviate from the original input. To address this discrepancy, this stage adjusts the tone of the refined image to align more closely with the original. This correction is achieved through another encoder-decoder network that takes the input image, specular residue map, and the refined specular-free image as input. The network fine-tunes the tone of the final output to ensure visual consistency.",{"type":18,"tag":26,"props":179,"children":180},{},[181],{"type":24,"value":182},"To facilitate the training and evaluation of this framework, a large-scale synthetic dataset was constructed, as depicted in Figure 2. This dataset encompasses a variety of objects and illumination conditions. Each data sample comprises albedo, shading, specular residue, and a ground-truth specular-free image. By leveraging this dataset, the network can effectively learn to remove specular highlights while preserving image quality in a supervised manner.",{"type":18,"tag":26,"props":184,"children":185},{},[186],{"type":18,"tag":98,"props":187,"children":189},{"alt":7,"src":188},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/10/11/fd698572142f45a591bb9abad783654c.png",[],{"type":18,"tag":26,"props":191,"children":192},{},[193],{"type":24,"value":194},"Figure 2 Example image groups in the dataset of this paper (a) Input image. (b) Albedo (c) Shading (d) Specular residue (e) Ground truth (f) Tone correction version of (e)",{"type":18,"tag":26,"props":196,"children":197},{},[198],{"type":18,"tag":30,"props":199,"children":200},{},[201],{"type":24,"value":202},"04 Experiment Results",{"type":18,"tag":26,"props":204,"children":205},{},[206],{"type":24,"value":207},"This study implements the proposed specular highlight removal network using MindSpore and evaluates its effectiveness on three datasets: SSHIQ, PSD, and the proposed SSHR dataset. The team compared the three-stage framework with four state-of-the-art traditional methods and two deep learning-based methods.",{"type":18,"tag":26,"props":209,"children":210},{},[211],{"type":18,"tag":98,"props":212,"children":214},{"alt":7,"src":213},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/10/11/76460550606140a6ae9bc513f62bdc21.png",[],{"type":18,"tag":26,"props":216,"children":217},{},[218],{"type":24,"value":219},"Table 1 Quantitative comparison results on SSSH, SHIQ, and PSD datasets",{"type":18,"tag":26,"props":221,"children":222},{},[223],{"type":24,"value":224},"In Table 1, the best results are in bold and the second-best results are underlined. The method of this paper achieves superior PSNR and SSIM values, demonstrating its advantage over existing approaches.",{"type":18,"tag":26,"props":226,"children":227},{},[228],{"type":18,"tag":98,"props":229,"children":231},{"alt":7,"src":230},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/10/11/e202defa22ae40a4972bcacee3ebcbb5.png",[],{"type":18,"tag":26,"props":233,"children":234},{},[235],{"type":24,"value":236},"Figure 3 Visual comparisons on synthetic images",{"type":18,"tag":26,"props":238,"children":239},{},[240],{"type":18,"tag":98,"props":241,"children":243},{"alt":7,"src":242},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/10/11/cfe06bf94e4f441d9ac4b9674884c2e3.png",[],{"type":18,"tag":26,"props":245,"children":246},{},[247],{"type":24,"value":248},"Figure 4 Visual comparisons on real images",{"type":18,"tag":26,"props":250,"children":251},{},[252],{"type":24,"value":253},"Figures 3 and 4 present visual comparisons on synthetic and real images, respectively. Existing traditional methods often introduce various visual artifacts, such as color distortions and black blocks. Other deep learning-based methods sometimes exhibit color deviation artifacts or generate implausible texture details. In contrast, the method of this paper produces high-quality specular highlight removal results without noticeable visual artifacts, a feat unachievable by existing methods.",{"type":18,"tag":26,"props":255,"children":256},{},[257],{"type":18,"tag":30,"props":258,"children":259},{},[260],{"type":24,"value":261},"05 Summary and Prospects",{"type":18,"tag":26,"props":263,"children":264},{},[265],{"type":24,"value":266},"This paper presents a three-stage method for specular highlight removal in object images, achieving high-quality, natural-looking results by progressively eliminating various visual artifacts. A key contribution is the introduction of a large-scale synthetic object image dataset, facilitating both network training and quantitative evaluation. Extensive experiments, including quantitative comparisons, visual analysis, and user studies, demonstrate the superiority of our method over existing approaches.",{"type":18,"tag":26,"props":268,"children":269},{},[270],{"type":24,"value":271},"The power and flexibility of the MindSpore framework proved invaluable, enabling the training and deployment of this complex deep learning model and yielding a model with excellent generalization capabilities. As MindSpore continues to evolve and its community expands, we anticipate an even greater impact on the field of AI. We encourage all MindSpore developers to actively contribute to the community, fostering collaboration and driving technological advancements through open source initiatives.",{"title":7,"searchDepth":273,"depth":273,"links":274},4,[],"markdown","content:technology-blogs:en:3409.md","content","technology-blogs/en/3409.md","technology-blogs/en/3409","md",1776506111082]