The Influence of the Amount of Parameters in Different Layers on the Performance of Deep Learning Models
Abstract: In recent years, deep learning has been widely used in many pattern recognition tasks including image classification and speech recognition due to its excellent performance. But a general rule for the structure design is lacked. We explored the influence of the amount of parameters in different layers of two deep learning models, convolutional neural network (CNN) and recurrent convolutional neural network (RCNN). Experiments on three benchmark datasets, CIFAR-10, CIFAR-100 and SVHN showed that when the total number of parameters was fixed, increasing the number of parameters in higher layers could boost the performance of the models while increasing the number of parameters in lower layers could be harmful to the performance of the models. Based on this simple rule, we obtained the state-of-the-art classification accuracy on CIFAR-100 and SVHN with single models.
文章引用: 岳喜斌 , 胡晓林 , 唐亮 (2015) 深度学习模型各层参数数目对于性能的影响。 计算机科学与应用， 5， 445-453. doi: 10.12677/CSA.2015.512056
Lecun, Y., Boser, B.E., Denker, J.S., et al. (1989) Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1, 541-551.
 Lecun, Y., Boser, B.E., Denker, J.S., et al. (1990) Handwritten Digit Recognition with a Back-Propagation Network. Advances in Neural Information Processing Systems, 396-404.
 Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012) ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems, 25.
 Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A. (2014) Going Deeper with Convolutions.
 Chatfield, K., Simonyan, K., Vedaldi, A., et al. (2014) Return of the Devil in the Details: Delving Deep into Convolutional Nets.
 葛詹尼加, 等, 著. 周晓林, 高国定, 等, 译. 认知神经科学: 关于心智的生物学[M]. 北京: 中国轻工业出版社, 2011: 141-177.
Babadi, B. and Sompolinsky, H. (2014) Sparseness and Expansion in Sensory Representations. Neuron, 83, 1213-1226.
 Liang, M. and Hu, X. (2015) Recurrent Convolutional Neural Network for Object Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni-tion.
 Dayan, P. and Abbott, L.F. (2001) Theoretical Neuroscience. MIT Press, Cambridge.
 Liang, M., Hu, X. and Zhang, B. (2015) Convolutional Neural Networks with Intra-Layer Recurrent Connections for Scene Labeling. Advances in Neural Information Processing (NIPS), 7-12 December 2015, Montréal.
 Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A.C. and Bengio, Y. (2013) Maxout Networks. Proceedings of the 30th In-ternational Conference on Machine Learning (ICML), 1319-1327.
 Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012) Imagenet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems (NIPS), 1097-1105.
 Lin, M., Chen, Q. and Yan, S. (2014) Network in Network. International Conference on Learning Representations (ICLR).
 Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z. and Tu, Z. (2014) Deeply Supervised Nets. Advances in Neural Information Processing Systems (NIPS), Deep Learning and Representation Learning Workshop.
 Xu, B., et al. (2015) Empirical Evaluation of Rectified Activations in Convolutional Network.
 Greff, S.K. and Schmidhuber, J. (2015) Highway Networks. Advances in Neural Information Processing (NIPS), Montréal, 7-12 December 2015.