Dynamics of Learning in Neural Networks: Supervised and Unsupervised

Shun-ichi Amari

RIKEN Brain Science Institute JAPAN

Neural networks have ability to modify its behavior by learning. This makes neural networks to have powerful information-processing capability. There are two types of learning, supervised (with teacher) and unsupervised (without teacher).

The present talk focuses on fundamental mathematical aspects of learning of neural networks. We begin with a general theory of learning of a neuron, followed by the mechanism of unsupervised learning of feature extractors and self-organization maps.

The MLP (multilayer perceptron) is a model of supervised learning, which has a universal power of approximating any functions. Backpropagation is a well-known method of learning. However, its learning behavior is known to be very slow, being trapped in plateaus and taking long time before getting rid of them.
The present talk uses information geometry to understand the dynamical behavior of learning in MLP. The set of MLPs forms a Riemannian manifold, in which the trajectories of learning are described. The manifold includes continua of singular points, which cause the plateau phenomena. Such strange geometrical structure is ubiquitous in hierarchical systems. We analyze the dynamics of learning near singularities explicitly, and propose a modified method of learning called the natural gradient method, which is free of the plateau phenomenon.
 

Shun-ichi Amari, Director, RIKEN Brain Science Institute (RIKEN BSI), Wako City, JAPAN Additional posts: Group Director, Brain-Style Intelligence Research Group, RIKEN BSI, Group Director, Brain-Style Information Systems Research Group, RIKEN BSI, Laboratory Head, Laboratory for Mathematical Neuroscience, RIKEN BSI, Professor Emeritus, University of Tokyo Educational Background: Doctor of Engineering (1963), Graduate School of Mathematical Engineering, University of Tokyo, Tokyo, Japan. Recent Professional Experience: Professor, Department of Mathematical Engineering and Information Physics, University of Tokyo Recent Awards and Honours: APNNA Special Award, Japanese Statistical Society Special Award, Best Paper Award, IEEE Signal Processing, Caianiello Memorial Award, IEEE Emanuel R. Piore Award, IEEE Neural Networks Pioneer Award, Japan Academy Award, RECENT Affiliations: President-Elect, IEICE; Fellow, IEICE; IEEE; President of INNS (1996); President of APNNA (1993)