Hartung-Gorre Verlag

Inh.: Dr. Renate Gorre

D-78465 Konstanz

Fon: +49 (0)7533 97227

Fax: +49 (0)7533 97228

www.hartung-gorre.de

S

Series in Signal and Information Processing, Vol. 35
edited by Hans-Andrea Loeliger

 

 

 

 

Patrick Murer

 

A New Perspective on Memorization in

Recurrent Networks of Spiking Neurons

 

1st Edition 2022. XVIII, 206 pages, € 64,00.

ISBN 978-3-86628-758-7

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Abstract

 

This thesis studies the capability of spiking recurrent neural network models to memorize dynamical pulse patterns (or firing signals).

 

In the first part, discrete-time firing signals (or firing sequences) are considered. A recurrent network model, consisting of neurons with bounded disturbance, is introduced to analyze (simple) local learning. Two modes of learning/memorization are considered: The first mode is strictly online, with a single pass through the data, while the second mode uses multiple passes through the data. In both modes, the learning is strictly local (quasi-Hebbian): At any given time step, only the weights between the neurons firing (or supposed to be firing) at the previous time step and those firing (or supposed to be firing) at the present time step are modified. The main result is an upper bound on the probability that the single-pass memorization is not perfect. It follows  that the memorization capacity in this mode asymptotically scales like that of the classical Hopfield model (which, in contrast, memorizes static patterns). However, multiple-rounds memorization is shown to achieve a higher capacity with an asymptotically nonvanishing number of bits per connection/synapse. These mathematical findings may be helpful for understanding the functionality of short-term memory and long-term memory in neuroscience.

 

In the second part, firing signals in continuous-time are studied. It is shown how firing signals, containing firings only on a regular time grid, can be (robustly) memorized with a recurrent network model. In principle, the corresponding weights are obtained by supervised (quasi-Hebbian) multi-pass learning. The asymptotic memorization capacity is a nonvanishing number measured in bits per connection/synapse as its discrete-time analogon. Furthermore, the timing robustness of the memorized firing signals is investigated for different disturbance models.

 

The regime of disturbances, where the relative occurrence-time of the firings is preserved over a long time span, is elaborated for the various disturbance models. The proposed models have the potential for energy efficient self-timed neuromorphic hardware implementations.

 

Keywords: Spiking recurrent neural network; dynamical pulse pattern; associative memory; Hopfield model; quasi-Hebbian learning; online learning; multi-pass memorization; asymptotic memorization capacity; timing robustness.

 

Series / Reihe "Series in Signal and Information Processing" im Hartung-Gorre Verlag

Direkt bestellen bei / to order directly from

Hartung-Gorre Verlag / D-78465 Konstanz / Germany

Telefon: +49 (0) 7533 97227  Telefax: +49 (0) 7533 97228
http://www.hartung-gorre.de   eMail: verlag@hartung-gorre.de