Evolutionary Training of Binary Neural Networks by Particle Swarm Optimization - Okada, Hidehiko (International Journal of Scientific Research in Computer Science and Engineering, 2022)
A problem with deep neural networks is that the memory size for recording a trained model becomes large. A solution to this problem is to make the parameter values binary. A challenge for the binary neural networks is that they cannot be trained by the ordinary gradient-based optimization methods. The author previously applied Evolution Strategy (ES), Genetic Algorithm (GA) and Differential Evolution (DE) to the training of binary neural networks and evaluates its performance. In this paper, the author applies Particle Swarm Optimization, an instance of swarm intelligence algorithms, and compares PSO with ES, GA and DE. The experimental results with a classification task revealed that PSO could optimize binary weights so that the trained model classified both trained and untrained data with 90%+ accuracies, but the accuracies were significantly worse than those by the three algorithms. To apply PSO to a binary optimization problem, real-valued particle position vectors need to be binarized. PSO suffers from the design of this binarization. Improving the binarization is a future research challenge.