Spiking Neural Networks, the Next Generation of Machine Learning

Everyone who has been remotely tuned in to recent progress in machine learning has heard of the current 2nd generation artificial neural networks used for machine learning. These are generally fully connected, take in continuous values, and output continuous values. Although they have allowed us to make breakthrough progress in many fields, they are biologically inn-accurate and do not actually mimic the actual mechanisms of our brain’s neurons.

The 3rd generation of neural networks, spiking neural networks, aims to bridge the gap between neuroscience and machine learning, using biologically-realistic models of neurons to carry out computation. A spiking neural network (SNN) is fundamentally different from the neural networks that the machine learning community knows. SNNs operate using spikes, which are discrete events that take place at points in time, rather than continuous values. The occurrence of a spike is determined by differential equations that represent various biological processes, the most important of which is the membrane potential of the neuron. Essentially, once a neuron reaches a certain potential, it spikes, and the potential of that neuron is reset. The most common model for this is the Leaky integrate-and-fire (LIF) model. Additionally, SNNs are often sparsely connected and take advantage of specialized network topologies.

Differential equation for membrane potential in the LIF modelMembrane potential behavior during a spikeSpike trains for a network of 3 neuronsA full spiking neural network

At first glance, this may seem like a step backwards. We have moved from continuous outputs to binary, and these spike trains are not very interpretable. However, spike trains offer us enhanced ability to process spatio-temporal data, or in other words, real-world sensory data. The spatial aspect refers to the fact that neurons are only connected to neurons local to them, so these inherently process chunks of the input separately (similar to how a CNN would using a filter). The temporal aspect refers to the fact that spike trains occur over time, so what we lose in binary encoding, we gain in the temporal information of the spikes. This allows us to naturally process temporal data without the extra complexity that RNNs add. It has been proven, in fact, that spiking neurons are fundamentally more powerful computational units than traditional artificial neurons.

Given that these SNNs are more powerful, in theory, than 2nd generation networks, it is natural to wonder why we do not see widespread use of them. The main issue that currently lies in practical use of SNNs is that of training. Although we have unsupervised biological learning methods such as Hebbian learning and STDP, there are no known effective supervised training methods for SNNs that offer higher performance than 2nd generation networks. Since spike trains are not differentiable, we cannot train SNNs using gradient descent without losing the precise temporal information in spike trains. Therefore, in order to properly use SNNs for real-world tasks, we would need to develop an effective supervised learning method. This is a very difficult task, as doing so would involve determining how the human brain actually learns, given the biological realism in these networks.

Another issue, that we are much closer to solving, is that simulating SNNs on normal hardware is very computationally-intensive since it requires simulating differential equations. However, neuromorphic hardware such as IBM’s TrueNorth aims to solve this by simulating neurons using specialized hardware that can take advantage of the discrete and sparse nature of neuronal spiking behavior.

The future of SNNs therefore remains unclear. On one hand, they are the natural successor of our current neural networks, but on the other, they are quite far from being practical tools for most tasks. There are some current real-world applications of SNNs in real-time image and audio processing, but the literature on practical applications remains sparse. Most papers on SNNs are either theoretical, or show performance under that of a simple fully-connected 2nd generation network. However, there are many teams working on developing SNN supervised learning rules, and I remain optimistic for the future of SNNs.