Foreword

The course “Neural networks: foundations and mathematics” showed you a powerful brick: the neuron that computes a weighted sum of its inputs, runs it through a non-linear function, and returns an output. That brick has a quiet but deep flaw. It has no memory. It receives, it answers, it forgets. Time does not exist for it.

This course fixes that flaw. We are going to give neurons back the dimension they were missing: time. You will meet another family of networks, spiking neural networks Spiking neural network A family of neural networks that communicate through discrete spikes in time, closer to biological operation than classical continuous networks. An active research field, rarely used in industrial practice so far. , where each neuron carries an internal state that evolves, accumulates, and fires discrete spikes. Closer to biology, far more energy-frugal, and carrying a way of computing that classical networks cannot easily imitate.

Three generations of networks

To place what we study, a classic grid due to Wolfgang Maass (Maass, 1997) sorts neural networks into three generations:

1. First generation: the binary threshold neuron of McCulloch and Pitts (McCulloch & Pitts, 1943). All-or-nothing output, like the switch you saw in chapter 1 of the foundations course.
2. Second generation: the continuous-activation neuron (sigmoid, ReLU). It underpins all of modern deep learning, and the whole foundations course.
3. Third generation: the spiking neuron. Its output is no longer a number, it is a train of spikes placed in time. That is the subject of this course.

Maass proved that a third-generation network can, with fewer neurons, compute functions that earlier generations need more resources to approximate. The promise is there as early as 1997. The difficulty of training it, however, took decades to tame.

The path

The course follows a simple thread: give time back to the neuron, make it compute, then make it learn, and finally make it run on dedicated hardware.

Block 1: the neuron regains time

Why the classical neuron fails to hear time, what a membrane potential is, the integrate-and-fire model and its zoo of variants.

Block 2: coding and connecting in time

How information is coded in spikes, and how a synapse transmits a signal that has a duration.

Block 3: learning without differentiating a spike

Timing-dependent plasticity, the wall of non-differentiability, the surrogate gradient, and converting a classical network into a spiking one.

Block 4: silicon that imitates the brain

Neuromorphic computing Neuromorphic computing A branch of computer science that designs hardware imitating biological brain operation (spiking neurons, local memory, asynchronous computation). An active research field at Intel (Loihi), IBM (TrueNorth) and several academic laboratories. , its chips, its promises of energy efficiency, then an honest assessment of the limits and real uses.

Who this course is for

• You finished the foundations course and want to see what lies beyond the continuous neuron. This is the ideal audience: you have exactly the right background.
• You are interested in computational neuroscience and want the bridge to machine learning. This course builds that bridge without drowning the biology in formalism.
• You work on the energy efficiency of AI and have heard of neuromorphic computing. You will find here what the promise rests on, and what it is worth.

This course is not a tutorial for one particular simulation library, nor a survey of the very latest research, which moves too fast for a course.

In one sentence

Spiking neural networks give the neuron back the dimension of time: their output is no longer a frozen number but a train of spikes, and it is that difference this course will unfold, from the basic model to silicon.

On to chapter 1

It all starts from a concrete frustration. A barn owl locates a mouse in pitch darkness thanks to a gap of a few microseconds between its two ears. The foundations neuron cannot do this, and for a precise reason. That is where chapter 1 begins.

Sources

• McCulloch, W. S. & Pitts, W. (1943). “A Logical Calculus of Ideas Immanent in Nervous Activity.” Bulletin of Mathematical Biophysics 5(4), 115-133. DOI 10.1007/BF02478259
• Maass, W. (1997). “Networks of spiking neurons: the third generation of neural network models.” Neural Networks 10(9), 1659-1671. DOI 10.1016/S0893-6080(97)00011-7

To go further before tackling chapter 1

• Gerstner, W., Kistler, W. M., Naud, R. & Paninski, L. (2014). Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition. Cambridge University Press. A reference of the field, free online. neuronaldynamics.epfl.ch
• Eshraghian, J. K. et al. (2023). “Training Spiking Neural Networks Using Lessons From Deep Learning.” Proceedings of the IEEE 111(9). arXiv:2109.12894