[{"data":1,"prerenderedAt":378},["ShallowReactive",2],{"content-query-q89pFwIUOo":3},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"cover":11,"type":12,"body":13,"_type":372,"_id":373,"_source":374,"_file":375,"_stem":376,"_extension":377},"/technology-blogs/en/3022","en",false,"","MindSpore-powered Joint Correcting and Refinement for Balanced Low-Light Image Enhancement","Author: Li Ruifeng | Source: Zhihu","2024-02-02","https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/03/15/a8fdc8a90d264fb5817ecaa2b38fa58c.png","technology-blogs",{"type":14,"children":15,"toc":369},"root",[16,24,30,35,40,45,50,61,66,75,88,101,116,121,126,131,145,157,168,182,187,192,200,205,210,215,222,227,241,246,253,258,263,270,275,294,301,306,311,318,323,328,335,340,345,359,364],{"type":17,"tag":18,"props":19,"children":21},"element","h1",{"id":20},"mindspore-powered-joint-correcting-and-refinement-for-balanced-low-light-image-enhancement",[22],{"type":23,"value":8},"text",{"type":17,"tag":25,"props":26,"children":27},"p",{},[28],{"type":23,"value":29},"Paper Title",{"type":17,"tag":25,"props":31,"children":32},{},[33],{"type":23,"value":34},"Joint Correcting and Refinement for Balanced Low-Light Image Enhancement",{"type":17,"tag":25,"props":36,"children":37},{},[38],{"type":23,"value":39},"Paper Source",{"type":17,"tag":25,"props":41,"children":42},{},[43],{"type":23,"value":44},"IEEE Transactions on Multimedia",{"type":17,"tag":25,"props":46,"children":47},{},[48],{"type":23,"value":49},"Paper URL",{"type":17,"tag":25,"props":51,"children":52},{},[53],{"type":17,"tag":54,"props":55,"children":59},"a",{"href":56,"rel":57},"https://arxiv.org/abs/2309.16128",[58],"nofollow",[60],{"type":23,"value":56},{"type":17,"tag":25,"props":62,"children":63},{},[64],{"type":23,"value":65},"Code URL",{"type":17,"tag":25,"props":67,"children":68},{},[69],{"type":17,"tag":54,"props":70,"children":73},{"href":71,"rel":72},"https://github.com/Daniel00008/JCRNet-mindspore",[58],[74],{"type":23,"value":71},{"type":17,"tag":25,"props":76,"children":77},{},[78,80,86],{"type":23,"value":79},"As an open-source AI framework, MindSpore supports ultra-large-scale AI pre-training and brings excellent experience of device-edge-cloud synergy, simplified development, ultimate performance, and security and reliability for researchers and developers. Since it was open sourced on March 28th, 2020, MindSpore has been downloaded for more than 6.57 million times. It has also been the subject of thousands of papers presented at premier AI conferences. Furthermore, it has a large community of developers and has been introduced in over 290 top universities and 5000 commercial applications. Being widely used in scenarios such as AI computing centers, finance, smart manufacturing, cloud, wireless, datacom, energy, \"1+8+",{"type":17,"tag":81,"props":82,"children":83},"em",{},[84],{"type":23,"value":85},"N",{"type":23,"value":87},"\" consumers, and smart automobiles, MindSpore has emerged as the leading open-source software on Gitee. The MindSpore community extends warm welcome to everyone who wishes to contribute to the development of the open-source community, suites, OpenMind projects, industry innovations and applications, algorithm innovations, academic cooperation, AI-themed book writing, and study cases in the cloud, device, edge, and security fields.",{"type":17,"tag":25,"props":89,"children":90},{},[91,93,99],{"type":23,"value":92},"Thanks to the support from scientific, industry, and academic circles, MindSpore-based papers accounted for 7% of all papers about AI frameworks in 2023, ranking No. 2 globally for two consecutive years. The MindSpore community is thrilled to share and interpret top-level conference papers and is looking forward to collaborating with experts from industries, academia, and research institutions, so as to yield proprietary AI outcomes and innovate AI applications. In this blog, I'd like to share the paper of the team led by Prof. ",{"type":17,"tag":94,"props":95,"children":96},"strong",{},[97],{"type":23,"value":98},"Han Yahong",{"type":23,"value":100},", College of Intelligence and Computing at Tianjin University.",{"type":17,"tag":25,"props":102,"children":103},{},[104,109,111],{"type":17,"tag":94,"props":105,"children":106},{},[107],{"type":23,"value":108},"01",{"type":23,"value":110}," ",{"type":17,"tag":94,"props":112,"children":113},{},[114],{"type":23,"value":115},"Research Background",{"type":17,"tag":25,"props":117,"children":118},{},[119],{"type":23,"value":120},"Low-light image enhancement, as an important preprocessing stage, enhances the feasibility of image analysis and understanding tasks. In recent years, thanks to the influence of deep learning, the low-light image enhancement technology has made significant progress. However, most existing models extract deep features from the entire image, ignoring the image's local details and local brightness ranges, which makes it difficult for the features to be fully utilized. To address this issue, some models also consider illumination balance, but do not consider comprehensive transformation of details and specific brightness adjustments in images, resulting in poor adjustment capabilities and unnatural image enhancement.",{"type":17,"tag":25,"props":122,"children":123},{},[124],{"type":23,"value":125},"Even though current low-light image enhancement techniques have shown some success, it is crucial to consider the balance between brightness, color, and illumination in practical applications. Neglecting this balance can significantly impact human visual perception and the performance of advanced visual models. Through exploring appropriate collaborative enhancement mechanisms, we can achieve more efficient and practical low-light image enhancement.",{"type":17,"tag":25,"props":127,"children":128},{},[129],{"type":23,"value":130},"The primary objective of this paper is to address the challenge of low-light image enhancement in extreme environments, particularly in low-light scenes during nighttime, by achieving a balance between image brightness, color, and illumination. The focus is on the practical application scenarios where the proposed approach can be effectively utilized. Existing methods often focus on a single aspect of images without considering this balance, resulting in problems such as color distortion and overexposure. This seriously affects human visual perception and the performance of advanced visual models. Therefore, this paper proposes a novel synergistic structure that can more effectively balance brightness, color, and illumination. To be more concise, the proposed joint correcting and refinement network (JCRNet) consists of three stages: feature extraction, joint refinement, and illumination adjustment. Its purpose is to balance the enhancement of brightness, color, and illumination. The algorithms presented in this paper can be implemented according to the cases and code in the MindSpore official documentation, and experiments and analysis on different low-light image enhancement data have verified the effectiveness of the proposed method.",{"type":17,"tag":25,"props":132,"children":133},{},[134,139,140],{"type":17,"tag":94,"props":135,"children":136},{},[137],{"type":23,"value":138},"02",{"type":23,"value":110},{"type":17,"tag":94,"props":141,"children":142},{},[143],{"type":23,"value":144},"Team Introduction",{"type":17,"tag":25,"props":146,"children":147},{},[148,150,155],{"type":23,"value":149},"The first author of this paper, ",{"type":17,"tag":94,"props":151,"children":152},{},[153],{"type":23,"value":154},"Yu Nana",{"type":23,"value":156},", is currently a doctoral student in the College of Intelligence and Computing at Tianjin University (2022-present), with a main research focus on computer vision and image processing. Her supervisor is Prof. Han Yahong.",{"type":17,"tag":25,"props":158,"children":159},{},[160,162,166],{"type":23,"value":161},"The correspondent author of this paper, ",{"type":17,"tag":94,"props":163,"children":164},{},[165],{"type":23,"value":98},{"type":23,"value":167},", is a professor and doctoral supervisor in the College of Intelligence and Computing at Tianjin University. His research interests include multimedia analysis, computer vision, and machine learning. In March 2012, he earned his PhD degree from the School of Computer Science at Zhejiang University. Following graduation, he became an associate professor at Tianjin University, where he was later promoted to full professor in 2016. In 2021, he was appointed as a 'Talented Professor' with long-term employment at Tianjin University. He has been recognized with numerous awards for his outstanding papers and achievements. In recent years, he has published over 60 papers in prestigious journals such as TPAMI and top academic conferences such as NeurIPS, with a focus on cross-media content understanding and inference, adversarial vision, and adversarial robustness. Additionally, he has undertaken research tasks for national key research and development projects, key natural fund projects, science and technology innovation special zones, and basic strengthening projects.",{"type":17,"tag":25,"props":169,"children":170},{},[171,176,177],{"type":17,"tag":94,"props":172,"children":173},{},[174],{"type":23,"value":175},"03",{"type":23,"value":110},{"type":17,"tag":94,"props":178,"children":179},{},[180],{"type":23,"value":181},"Introduction to the Paper",{"type":17,"tag":25,"props":183,"children":184},{},[185],{"type":23,"value":186},"This paper introduces research on a low-light image enhancement technology, which plays a crucial role in the image preprocessing stage of foundation model-guided generalized visual scenes. It can effectively improve the practicality and generalization of models in different downstream applications such as scene analysis, object detection, and image segmentation.",{"type":17,"tag":25,"props":188,"children":189},{},[190],{"type":23,"value":191},"Most of the existing low-light image enhancement methods do not pay attention to local enhancement of images. As a result, some areas of the enhanced images are still underexposed. Moreover, some methods fail to pay attention to the local exposure balance of images, resulting in overexposure in areas that were originally well-exposed. The low-light image shown in Figure 1 contains both underexposed and overexposed areas, so illumination balance must be considered during the enhancement process. In addition, if the colors of the image are not taken into account, the enhanced image will exhibit dim colors and distortions such as halos. From the above analysis, it can be seen that existing methods have improved the quality of low-light images in certain aspects, but still lack a proper collaborative mechanism in terms of brightness, color, and illumination.",{"type":17,"tag":25,"props":193,"children":194},{},[195],{"type":17,"tag":196,"props":197,"children":199},"img",{"alt":7,"src":198},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/03/15/f2fce89fe2754b3e8059d3fcec36c649.png",[],{"type":17,"tag":25,"props":201,"children":202},{},[203],{"type":23,"value":204},"Figure 1: Enhancement effects of existing methods in local brightness, illumination, and color (line charts show quantitative evaluation results of different methods.)",{"type":17,"tag":25,"props":206,"children":207},{},[208],{"type":23,"value":209},"This paper proposes a low-light image enhancement network that maintains the balance of image brightness, color and illumination in response to the preceding issue. In contrast with existing methods, this paper first uses the attention and local supervision mechanisms to extract more comprehensive local information. This is beneficial for improving a model's learning ability for brightness, color, and illumination. In addition, cross-stage feature transmission and spatial feature transformation can restore more details, which helps improve the fidelity of color information. Finally, inspired by the theory of back projection, this paper uses brightening and darkening operations to dynamically adjust the brightness range of an image. This approach prevents overexposure of the enhanced image and enables gradual learning of residual information. Therefore, for low-light image enhancement, it is important to have a joint correcting and refinement mechanism, which can improve brightness while maintaining color fidelity and illumination balance.",{"type":17,"tag":25,"props":211,"children":212},{},[213],{"type":23,"value":214},"As shown in the network structure in Figure 2, the core of the method in this paper is a synergistic three-stage network. In the feature extraction stage, the residual channel attention and encoder-decoder blocks are used to extract main features, and the self-supervision block is used to achieve adaptive propagation of useful features. In the joint refinement stage, cross-stage feature transmission and RGB image color correction are used to enhance image details and correct distortions. In the illumination adjustment stage, by using the theory of back projection, a model can actively learn the residual between a normal light image and the predicted image, and thus adaptively adjust the illumination balance of the enhanced image.",{"type":17,"tag":25,"props":216,"children":217},{},[218],{"type":17,"tag":196,"props":219,"children":221},{"alt":7,"src":220},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/03/15/28d749660cea4d759d558dd90a633c3e.png",[],{"type":17,"tag":25,"props":223,"children":224},{},[225],{"type":23,"value":226},"Figure 2: Architecture of the low-light image enhancement network (JCRNet) proposed in this paper",{"type":17,"tag":25,"props":228,"children":229},{},[230,235,236],{"type":17,"tag":94,"props":231,"children":232},{},[233],{"type":23,"value":234},"04",{"type":23,"value":110},{"type":17,"tag":94,"props":237,"children":238},{},[239],{"type":23,"value":240},"Experimental Result",{"type":17,"tag":25,"props":242,"children":243},{},[244],{"type":23,"value":245},"To verify the effectiveness of the proposed JCRNet, the authors compared it with 20 advanced low-light image enhancement methods on nine widely used low-light enhancement datasets (LOL, COCO, MIT, VV, MEF, NPE, DICM, ExDark, LIME), and further analyzed the experimental results.",{"type":17,"tag":25,"props":247,"children":248},{},[249],{"type":17,"tag":196,"props":250,"children":252},{"alt":7,"src":251},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/03/15/2737093990fa4973944703439f614ffc.png",[],{"type":17,"tag":25,"props":254,"children":255},{},[256],{"type":23,"value":257},"Table 1: Quantitative comparison results of evaluation metrics on three reference datasets",{"type":17,"tag":25,"props":259,"children":260},{},[261],{"type":23,"value":262},"Table 1 visually displays the quantitative results of the proposed low-light image enhancement method on three widely used datasets, where the best, second-best, and third-best performances are marked in red, blue, and green, respectively. The method proposed in this paper is superior to most methods in most metrics on these three datasets. For example, on the LOL dataset, compared to the second-best method, the proposed method improves the PSNR by 0.7dB compared to the existing best-performing method. Similar gains can be observed in other metrics.",{"type":17,"tag":25,"props":264,"children":265},{},[266],{"type":17,"tag":196,"props":267,"children":269},{"alt":7,"src":268},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/03/15/cc3da7cfd897486aaa1f08608bd95179.png",[],{"type":17,"tag":25,"props":271,"children":272},{},[273],{"type":23,"value":274},"Table 2: Quantitative comparison results of evaluation metrics on six non-reference datasets",{"type":17,"tag":25,"props":276,"children":277},{},[278,280,285,287,292],{"type":23,"value":279},"Table 2 presents the quantitative results of the proposed method on six non-reference datasets, where ",{"type":17,"tag":94,"props":281,"children":282},{},[283],{"type":23,"value":284},"T",{"type":23,"value":286}," denotes conventional methods and ",{"type":17,"tag":94,"props":288,"children":289},{},[290],{"type":23,"value":291},"DL",{"type":23,"value":293}," denotes deep learning methods. It can be observed from the table that the model trained based on the MindSpore framework in this paper exhibits performance improvement on most datasets.",{"type":17,"tag":25,"props":295,"children":296},{},[297],{"type":17,"tag":196,"props":298,"children":300},{"alt":7,"src":299},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/03/15/546b14c151094b818bc12852b55e5cba.png",[],{"type":17,"tag":25,"props":302,"children":303},{},[304],{"type":23,"value":305},"Figure 3: Visualized display of LOL dataset enhancement results, with a red box highlighting the enlarged portion of each image",{"type":17,"tag":25,"props":307,"children":308},{},[309],{"type":23,"value":310},"Figure 3 shows the enhancement results of the LOL dataset in indoor low-light scenes. The proposed method produces images with relatively natural overall brightness and colors after enhancement. However, other methods' enhancement results exhibit issues of overall darkening or overexposure, as well as noises in certain areas due to excessive brightening or poor model generalization. Additionally, the authors have magnified some details of the enhancement results of all methods. This indicates that the proposed method remains competitive in terms of details.",{"type":17,"tag":25,"props":312,"children":313},{},[314],{"type":17,"tag":196,"props":315,"children":317},{"alt":7,"src":316},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/03/15/ebfad79d83af4556ba4e3812a6d68a06.png",[],{"type":17,"tag":25,"props":319,"children":320},{},[321],{"type":23,"value":322},"Figure 4: Visualized display of VV dataset enhancement results, with a red box highlighting the enlarged portion of each image",{"type":17,"tag":25,"props":324,"children":325},{},[326],{"type":23,"value":327},"Figure 4 provides the enhancement results of the VV dataset in indoor low-light scenes. Overall, most of the enhancement results are overexposed, but only KinD, EnGAN, DDNet, and the method proposed in this paper do not lead to overexposure.",{"type":17,"tag":25,"props":329,"children":330},{},[331],{"type":17,"tag":196,"props":332,"children":334},{"alt":7,"src":333},"https://obs-mindspore-file.obs.cn-north-4.myhuaweicloud.com/file/2024/03/15/4f89d6edb2da47818d82c4d888e81434.png",[],{"type":17,"tag":25,"props":336,"children":337},{},[338],{"type":23,"value":339},"Figure 5: Visualized display of MEF dataset enhancement results, with a red box highlighting the enlarged portion of each image",{"type":17,"tag":25,"props":341,"children":342},{},[343],{"type":23,"value":344},"Figure 5 shows examples of low-light image enhancement on the MEF dataset. It is evident from the comparison that the method proposed in this paper produces overall brighter results and recovers more color and detail information.",{"type":17,"tag":25,"props":346,"children":347},{},[348,353,354],{"type":17,"tag":94,"props":349,"children":350},{},[351],{"type":23,"value":352},"05",{"type":23,"value":110},{"type":17,"tag":94,"props":355,"children":356},{},[357],{"type":23,"value":358},"Summary and Prospects",{"type":17,"tag":25,"props":360,"children":361},{},[362],{"type":23,"value":363},"This paper mainly introduces JCRNet, a balanced low-light image enhancement network that addresses the challenge of enhancing images in extreme low-light environments, such as nighttime scenes. JCRNet leverages joint correcting and refinement techniques to achieve a more effective balance of brightness, color, and illumination, resulting in enhanced images with more comprehensive details. This network further promotes color correction and feature refinement through cross-stage feature transmission and spatial feature transformation. It has been extensively validated through experiments on nine different low-light enhancement datasets. Comprehensive experiments have shown that the proposed method outperforms existing models in addressing color distortion and exposure imbalance in low-light images. Additionally, the proposed model has been demonstrated to be effective in different types of low-light images and has shown practicality in saliency detection tasks.",{"type":17,"tag":25,"props":365,"children":366},{},[367],{"type":23,"value":368},"When using MindSpore to implement the algorithms in this paper, the authors find that a large number of tutorial documents and samples from the MindSpore open source community can assist them in getting started and completing the design and implementation of the algorithms. Additionally, documents about API comparison with other deep learning frameworks (such as PyTorch and TensorFlow) can also help guide the model porting. They hope to actively encourage and attract more developers and researchers to participate in the construction of the MindSpore community and provide more examples of code, case analysis, and solutions.",{"title":7,"searchDepth":370,"depth":370,"links":371},4,[],"markdown","content:technology-blogs:en:3022.md","content","technology-blogs/en/3022.md","technology-blogs/en/3022","md",1776506109468]