Evolutionary Reinforcement Learning of Neural Network Controller for Pendulum Task by Evolution Strategy
Keywords:
Evolutionary algorithm, Evolution strategy, Neural network, Neuroevolution, Reinforcement learningAbstract
Reinforcement learning of neural networks requires gradient-free algorithms because labeled training data are not available. Evolutionary algorithms are applicable to the reinforcement learning because the algorithms do not rely on gradients. To successfully train neural networks by evolutionary algorithms, we need to carefully choose appropriate algorithms because many algorithm variations are available. The author experimentally evaluates Evolution Strategy, an instance of evolutionary algorithms, for the reinforcement learning of neural networks. A pendulum control task is adopted in this work. Experimental results revealed that ES could successfully train an MLP so that the trained MLP could make and keep the pendulum upright quickly, if the MLP was equipped with sufficient hidden units. For the task adopted in this work, 16 units are the best among 8, 16 and 32 units in terms of the task performance and the computational efficiency. Besides, the results revealed that exploration contributes more for ES to search for better solutions than exploitation.
References
T. Bäck, H.P. Schwefel, “An Overview of Evolutionary Algorithms for Parameter Optimization,” Evolutionary Computation, Vol.1, No.1, pp.1-23, 1993.
D.B. Fogel, “An Introduction to Simulated Evolutionary Optimization,” IEEE Transactions on Neural Networks, Vol.5, No.1, pp.3-14, 1994.
T. Bäck, “Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms,” Oxford University Press, 1996.
A.E. Eiben, R. Hinterding, Z. Michalewicz, “Parameter Control in Evolutionary Algorithms,” IEEE Transactions on Evolutionary Computation, Vol.3, No.2, pp.124-141, 1999.
A.E. Eiben, J.E. Smith, “Introduction to Evolutionary Computing (2nd ed.),” Springer, 2015.
C.J.C.H. Watkins, “Learning from Delayed Rewards,” PhD Thesis, Cambridge University, 1989.
C.J.C.H. Watkins, P. Dayan, “Q-Learning,” Machine Learning, Vol.8, No.3, pp.279-292, 1992.
R.S. Sutton, A.G. Barto, “Reinforcement Learning: An Introduction (2nd ed.),” MIT Press, 2018.
H.P. Schwefel, “Evolution Strategies: A Family of Non-Linear Optimization Techniques based on Imitating Some Principles of Organic Evolution,” Annals of Operations Research, Vol.1, pp.165-167, 1984.
H.G. Beyer, H.P. Schwefel, “Evolution Strategies: A Compre-hensive Introduction,” Journal Natural Computing, Vol.1, No.1, pp.3-52, 2002.
D.E. Goldberg, J.H. Holland, “Genetic Algorithms and Machine Learning,” Machine Learning, Vol.3, No.2, pp.95-99, 1988.
J.H. Holland, “Genetic Algorithms,” Scientific American, Vol.267, No.1, pp.66-73, 1992.
M. Mitchell, “An Introduction to Genetic Algorithms,” MIT Press, 1998.
K. Sastry, D. Goldberg, G. Kendall, “Genetic Algorithms,” Search Methodologies, Springer, pp.97-125, 2005.
R. Storn, K. Price, “Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces,” Journal of Global Optimization, Vol.11, pp.341-359, 1997.
K. Price, R.M. Storn, J.A. Lampinen, “Differential Evolution: A Practical Approach to Global Optimization,” Springer Science & Business Media, 2006.
S. Das, P.N. Suganthan, “Differential Evolution: A Survey of the State-of-the-art,” IEEE transactions on evolutionary computation, Vol.15, No.1, pp.4-31, 2010.
D.E. Rumelhart, G.E. Hinton, R.J. Williams. “Learning Internal Representations by Error Propagation,” in D.E. Rumelhart, J.L. McClelland, and the PDP research group (editors), “Parallel Distributed Processing: Explorations in the Microstructure of Cognition,” Vol.1: Foundation. MIT Press, 1986.
R. Collobert, S. Bengio, “Links Between Perceptrons, MLPs and SVMs,” Proc. of the Twenty-First International Conference on Machine Learning (ICML’04), ACM, 2004.
X. Yao, Y. Liu, “A New Evolutionary System for Evolving Arti?cial Neural Networks,” IEEE Transactions on Neural Networks, Vol.8, No.3, pp.694-713, 1997.
N.T. Siebel, G. Sommer, “Evolutionary Reinforcement Learning of Artificial Neural Networks,” Internatinal Journal of Hybrid Intelligent Systems, Vol.4, No.3, pp.171-183. 2007.
K. Chellapilla, D.B. Fogel, “Evolving Neural Networks to Play Checkers Without Relying on Expert Knowledge,” IEEE Transactions on Neural Networks, Vol.10, No.6, pp.1382-1391, 1999.
L. Cardamone, D. Loiacono and P. L. Lanzi, “Evolving Competitive Car Controllers for Racing Games with Neuro-evolution,” Proc. of 11th Annual Conference on Genetic and Evolutinary Computation, pp.1179-1186, 2009.
S. Risi, J. Togelius, “Neuroevolution in Games: State of the Art and Open Challenges”, IEEE Transactions on Computational Intelligence and AI in Games, Vol.9, No.1, pp.25-41, 2017.
J. Togelius, S.M. Lucas, “Evolving Controllers for Simulated Car Racing,” Proc. of 2005 IEEE Congress on Evolutionary Computation, Vol.2, pp.1906-1913, 2005.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors contributing to this journal agree to publish their articles under the Creative Commons Attribution 4.0 International License, allowing third parties to share their work (copy, distribute, transmit) and to adapt it, under the condition that the authors are given credit and that in the event of reuse or distribution, the terms of this license are made clear.
