Research in neural networks is challenging and continuing. Here, we are going to review some of the research in a series of posts. In the starting post we are discussing a recurrent neural network proposed by J. Namikawa and J. Tani which implements the dynamics of a multiple attractor.
"Recurrent neural networks (RNNs) have been successfully applied to the modeling of various types of dynamical systems. Since the universal approximation ability of multilayer neural networks has been proved, RNNs can model arbitrary dynamical systems and turing machines [1–3]. However, applying RNNs to a desired model may be very difficult even if such RNNs exist [4]. For example, building RNNs to implement required multiple attractor dynamics is a difficult problem for standard training, such as the gradient descent method. Doya and Yoshizawa [5] demonstrated that RNNs can acquire two limit cycles in the gradient descent method using initialization with small connection weights, whereas learning for more than three limit cycles is difficult [6]. This is due to the fact that the learning of several time series causes a conflict with respect to the changing of the connection weights. How to form RNN models that can learn several temporal sequence patterns has proved to be a challenging problem.
There have been some approaches to this problem. In order to avoid conflicts in the change of parameters, the mixture-of-experts-type architecture has been inves- tigated [7, 8]. The mixture-of-experts model consists of RNNs as experts and a hierarchical gating mechanism. At the end of successful learning, each expert implements attractor dynamics as locally represented knowledge, and a gating mechanism chooses only one expert at any time. The system can acquire many attractor patterns although there is a disadvantage in that the system does not have the generalization ability on the attractor patterns. As the other approach to implement multiple patterns, the parametric bias (PB) method has been developed to improve the learning capability of RNNs [9, 10]. In an RNN that employs the PB method (RNNPB), PB values provide the information needed in order to individualize each sequence. It has been reported that the number of time series that RNNPBs can learn is greater than that which RNNs without PB can learn. However, the PB method cannot avoid the conflict caused by each attractor learning. Therefore, learning multiple time series by an RNNPB tends to fail when the number of time series increases."
[1] K. Funahashi and Y. Nakamura, “Approximation of dynamical systems by continuous time recurrent neural networks,” Neural Networks, vol. 6, no. 6, pp. 801–806, 1993.
[2] H. T. Siegelmann and E. D. Sontag, “Analog computation via neural networks,” Theoretical Computer Science, vol. 131, no. 2, pp. 331–360, 1994.
[3] H. T. Siegelmann and E. D. Sontag, “On the computational power of neural nets,” Journal of Computer and System Sciences, vol. 50, no. 1, pp. 132–150, 1995.
[4] Y. Bengio, P. Simard, and P. Frasconi, “Learning long- term dependencies with gradient descent is difficult,” IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 157–166, 1994.
[5] K. Doya and S. Yoshizawa, “Memorizing oscillatory patterns in the analog neuron network,” in Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN ’89), vol. 1, pp. 27–32, Washington, DC, USA, June 1989.
[6] F.-S. Tsung, Modeling dynamical systems with recurrent neural networks, Ph.D. thesis, Department of Computer Science, University of California, San Diego, Calif, USA, 1994.
[7] D. M. Wolpert and M. Kawato, “Multiple paired forward and inverse models for motor control,” Neural Networks, vol. 11, no. 7-8, pp. 1317–1329, 1998.
[8] J. Tani and S. Nolfi, “Learning to perceive the world as articulated: an approach for hierarchical learning in sensory- motor systems,” Neural Networks, vol. 12, no. 7-8, pp. 1131– 1141, 1999.
[9] J. Tani, “Learning to generate articulated behavior through the bottom-up and the top-down interaction processes,” Neural Networks, vol. 16, no. 1, pp. 11–23, 2003.
[10] J. Tani and M. Ito, “Self-organization of behavioral primitives as multiple attractor dynamics: a robot experiment,” IEEE Transactions on Systems, Man and Cybernetics Part A, vol. 33, no. 4, pp. 481–488, 2003.
* Content is published under Creative Commons License by J. Namikawa and J. Tani in the Hindawi Publishing Corporation journal: Advances in Artificial Neural Systems.
