随着下一代无线通信技术的发展，Femtocell的概念应运而生。Femtocell是一种可以改善室内覆盖的家庭基站，其主要致力于增加室内覆盖，改善系统性能，减少Macrocell网络通信负担。然而，在Femtocell与Macrocell共存的双层网络中存在同信道干扰问题，这会降低系统性能。本文首先分析了分布式Q学习算法在Femtocell与Macrocell构成的两层网络下行链路功率控制中的应用，在此基础上仿真并分析了独立Q学习(Independent Q-learning，IQL)算法和合作Q学习(Cooperation Q-learning，CQL)算法，并对CQL进行了改进，保证在满足基站发射功率和容量性能的情况下回报函数值是正值，其他情况下是负值，经matlab仿真验证，这可提高系统容量效率和公平性。
Power Control for Femtocell Two-Tier Networks Based on Distributed Q-Learning
Abstract: With the development of the next generation wireless communications technology, the concept of Femtocell came into being. Femtocell is a kind of home base station which can expand coverage indoor, improve system performance and decrease macrocell communication overhead. However, the existing co-channel interference of the network between macrocells and femtocells can dramatically degrade the overall performance of the network. For this reason, we did a research and wrote the paper. First we analyze a two-tier network of distributed Q-learning algorithm in Femtocell and Macrocell downlink power control, then simulate and analyze the independent Q-learning algorithm and cooperation Q-learning algorithm based on that. In addition, we optimize the cooperation Q-learning algorithm so that it can ensure that the value of return function is positive in the case of satisfying the base station transmit power and capacity performance, and is negative in other cases. This improvement can indeed increase the efficiency and fairness of the system capacity after the confirmation of matlab simulation.
文章引用: 宁海芹 , 潘沛生 (2013) 基于分布式Q学习的Femtocell两层网络功率控制方法。 无线通信， 3， 1-6. doi: 10.12677/HJWC.2013.31001
 J. Zhang, G. de la Roche. Femtocells: Technologies and deployment. Hoboken: John Wiley and Sons, Ltd., 2010.
 S. A. Grandhi, J. Zander. Constrained power control in cellular radio systems. 1994 IEEE 44th Vehicular Technology Con- ference, 1994, 2: 824-828.
 C. J. C. H. Watldns. Learning from delayed rewards. PhD Thesis, Cambridge: University of Cambridge, 1989.
 S. Mahadevan, J. Connell. Automatic programming of behavior- based robots using reinforcement learning. Proceedings of the 1991 National Conference on AI, 1991: 768-773.
 R. S. Sutton, A. G. Barto. Reinforcement learning: An intro- duction. Cambridge: MIT Press, 1998.
 C. J. C. H. Watkins, P. Dayan. Technical note Q-learning. Journal of Machine Learning, 1992, 8(3-4): 279-292.
 J. R. Kok, N. Vlassis. Collaborative multiagent reinforcement learning by payoffpropagation. The Journal of Machine Learning Research, 2006, 7: 1789-1828. ttp://portal.acm.org/citation.cfm?id=1248547.124861
 M. Ahmadabadi, M. Asadpour. Expertness based cooperative Q-learning. Systems, Man, and Cybernetics, Part B: IEEE Transactions on Cybernetics, 2002, 32(1): 66-76.
 A. Galindo-Serrano, L. Giupponi. Distributed Q-learning for aggregated interference control in cognitive radio networks. IEEE Transactions on Vehicular Technology, 2010, 59(4): 1823- 1834.
 A. Galindo, L. Giupponi. Decentralized Q-learning for aggregated interference control in completely and partially observable cognitive radio networks. 2010 7th IEEE Proceed- ings of the Consumer Communications and Networking Conference (CCNC), Las Vegas, 2010: 1-6.
 A. Galindo-Serrano, L. Giupponi and M. Dohler. Cognition and docition in OFDMA-based Femtocell networks. 2010 IEEE Global Telecommunications Conference, Miami, 2010: 1-6.
 R. Jain, D.-M. Chiu and W. Hawe. A quantitative measure of fairness and discrimination for resource allocation in shared computer systems. Maynard: Eastern Research Laboratory, Digital Equipment Corporation, 1998.