# 一种结合自注意力和门控机制的图像超分辨率重建算法An Image Super-Resolution Reconstruction Algorithm Combining Self-Attention and Gating Mechanism

Abstract: Image super resolution reconstruction aims to reconstruct a low resolution image into a clearer high-resolution image. The super resolution reconstruction algorithm is helpful to improve the image quality and can recover the missing texture and detail information as accurately as possible. It has important scientific significance and application value in the field of image processing. In order to further improve the quality of image reconstruction, this paper combines the sparse representation and deep learning algorithm. The reconstruction of the sparse representation model is used to get the high resolution image as input of deep learning model, and on the basis of introducing the VDSR network since attention mechanism and gating mechanism, models can be dynamically in the process of training to learn the importance of different characteristics. Thus, the pixel size and characteristics of granularity further enrich the characteristics of the image. We carried out a large number of experiments on the public super-fractional reconstruction data sets, such as Set5, SET14, B100 and Urban100. The results show that the multi-granularity feature extraction reconstruction algorithm proposed in this paper can obtain better reconstruction details and higher PSNR/SSIM values compared with the existing reconstruction methods.

1. 引言

2. 本文算法描述

Figure 1. Algorithm structure diagram in this paper

2.1. 图像输入

Figure 2. The image input was compared with the result of bicubic interpolation reconstruction using sparse representation

2.2. 特征提取

2.2.1. 自注意力机制特征提取

${S}_{j,i}=\frac{\mathrm{exp}\left({F}_{i},{G}_{j}\right)}{\underset{i=1}{\overset{N}{\sum }}\mathrm{exp}\left({F}_{i},{G}_{j}\right)}$ (1)

Figure 3. The autoattention mechanism and gating mechanism were used to extract the feature map

$y=S\left(X\right)\otimes H\left(X\right)$ (2)

2.2.2. 门控机制特征提取

$a=\sigma \left(\mathrm{tanh}\left({W}_{1}*{f}_{1}+{W}_{2}*{f}_{2}\right)\right)$ (3)

$F=a*{f}_{1}+\left(1-a\right)*{f}_{2}$ (4)

2.3. 重建图像

3. 实验结果及分析

3.1. 训练过程

3.2. 实验结果比较与分析

Table 1. Different SR algorithms use different amplification factor evaluation indexes on four benchmark data sets

Figure 4. A visual comparison of models with magnification factor of 4 in Set14 “Zebra”

Figure 5. Visual Comparison of different models with magnification factor of 4 in Urban “IMG043”

Figure 6. Visual comparison of SET14 “Comic”, B100 “8023” and Urban100 “Image034” with magnification factor of 4

4. 结论

NOTES

*通讯作者。

[1] Dai, S., Han, M., Xu, W., Wu, Y. and Gong, Y. (2007) Soft Edge Smoothness Prior for Alpha Channel Super Resolu-tion. Proceedings of the IEEE Conference on Computer Vision and Pattern Classification (CVPR), Minneapolis, 17-22 June 2007, 1-8.
https://doi.org/10.1109/CVPR.2007.383028

[2] Sun, J., Xu, Z. and Shum, H. (2008) Image Su-per-Resolution Using Gradient Profile Prior. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, 24-26 June 2008, 1-8.

[3] Yang, J.C., Wright, J., Huang, T., et al. (2010) Image Su-per-Resolution via Sparse Representation. IEEE Transactions on Image Processing, 19, 2861-2873.
https://doi.org/10.1109/TIP.2010.2050625

[4] Yeganli, F., Nazzal, M., Unal, M. and Ozkaramanli, H. (2014) Image Super Resolution Viasparse Representation over Coupled Dictionary Learning Based on Patch Sharpness. Euro-pean Modelling Symposium, Prague, 21-23 October 2014, 203-208.
https://doi.org/10.1109/EMS.2014.67

[5] Zhang, Y., Wu, W., Dai, Y., Yang, X., Yan, B. and Lu, W. (2013) Re-mote Sensing Images Super-Resolution Based on Sparse Dictionaries and Residual Dictionaries. IEEE 11th International Conference on Dependable, Autonomic and Secure Computing, Chengdu, 21-22 December 2013, 318-323.
https://doi.org/10.1109/DASC.2013.82

[6] Fu, C.-H., Chen, H., Zhang, H. and Chan, Y.-L. (2014) Single Image Super Resolution Based on Sparse Representation and Adaptive Dictionary Selection. 19th International Conference on Digital Signal Processing, Hong Kong, 20-23 August 2014, 449-453.

[7] Timofte, R., De Smet V. and Van Gool, L. (2014) A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution. In: Proceedings of the Asian Con-ference on Computer Vision (ACCV), Springer, Berlin, 111-126.
https://doi.org/10.1007/978-3-319-16817-3_8

[8] Zeiler, M.D. and Fergus, R. (2014) Visualizing and Under-standing Convolutional Networks. In: Proceedings of the European Conference on Computer Vision (ECCV), Springer, Berlin, 818-833.
https://doi.org/10.1007/978-3-319-10590-1_53

[9] Radford, A., Metz, L. and Chintala, S. (2015) Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. ICLR 2016, Computer Sci-ence.

[10] Dong, C., Loy, C.C., He, K. and Tang, X. (2015) Image Super Resolution Using Deep Convolutional Net-works. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38, 295-307.
https://doi.org/10.1109/TPAMI.2015.2439281

[11] Dong, C., Chen, C.L. and Tang, X. (2016) Accelerating the Super Resolution Convolutional Neural Network. In: European Conference on Computer Vision, Springer, Cham, 391-407.
https://doi.org/10.1007/978-3-319-46475-6_25

[12] Kim, J., Kwon, L.J. and Mu, L.K. (2016) Accurate Image Super Resolution Using Very Deep Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 27-30 June 2016, 1646-1654.
https://doi.org/10.1109/CVPR.2016.182

[13] He, K., Zhang, X., Ren, S. and Sun, J. (2016) Deep Residual Learn-ing for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, 27-30 June 2016, 770-778.
https://doi.org/10.1109/CVPR.2016.90

[14] Zhang, H., Goodfellow, I., Metaxas, D. and Odena, A. (2018) Self-Attention Generative Adversarial Networks.

[15] Zhang, Y., Li, K., Li, K., Zhong, B. and Fu, Y. (2019) Residual Non-Local Attention Networks for Image Restoration.

[16] Yi, P., Wang, Z.Y., Jiang, K., Jiang, J.J. and Ma, J.Y. (2019) Progressive Fusion Video Super Resolution Network via Exploiting Non Local Spatiotemporal Correlations. Proceed-ings of the IEEE International Conference on Computer Vision (ICCV), Seoul, 27 October-2 November 2019, 3106-3115.
https://doi.org/10.1109/ICCV.2019.00320

[17] Luong, M.T., Pham, H. and Manning, C.D. (2015) Ef-fective Approaches to Attention Based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, September 2015, 1412-1421.
https://doi.org/10.18653/v1/D15-1166

[18] Lai, W.-S., Huang, J.-B., Ahuja, N. and Yang, M.-H. (2017) Deep La-placian Pyramid Networks for Fast and Accurate Super Resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 21-26 July 2017, 5835-5843.
https://doi.org/10.1109/CVPR.2017.618

Top