A neuron embedded in a complex network can influence its own input via feedback loops within the network and possibly via longer feedback loops that extend through the animal’s external environment. Computational study of hypothetical synaptic learning rules, which I call closed-loop learning rules, allow a learning element to adjust its behavior so as to influence its input in a “desirable” manner, that is, to control aspects of its input. Signal timing plays a key role in these learning rules due to the range of time delays associated with the feedback loops in which the element is embedded. Learning rules of this kind belong to the area of machine learning known as Reinforcement Learning. Although some of these ideas have become well known, such as the Actor-Critic architecture and the TD-algorithm’s success in modeling phasic activity of dopamine neurons, this talk will focus on other implications of closed-loop learning rules that are less well-known and that may have implications for our understanding of time dependencies in synaptic plasticity.
Orchestrating learning: signaling across scales in synaptic plasticity
Many processes must mesh precisely to achieve sustained state changes in synaptic plasticity. There is an intricate interplay between input activity patterns from the network, through cellular biophysics and calcium dynamics, and down to molecular and gene-regulatory events.
I will discuss two converging efforts of our work to understand these events. First, we have been performing a breadth-first analysis of a wide range of biochemical signaling pathways that are implicated in plasticity. These include G-protein and kinase networks, receptor trafficking, control of local dendritic protein synthesis, and transcriptional control. Second, we have been developing capabilities for a vertical, mutiscale analysis of interactions between network, electrical, and chemical signaling. These models put plasticity into the context of the larger networks, both electrical and molecular. The goal is to work out how the cell orchestrates not just the processing and storage of input activity, but also the maintenance and reconfiguration of its own structure.
Modulation of Synaptic Plasticity
Robust rules of activity-induced synaptic plasticity such as spike-timing-dependent plasticity (STDP) are important for learning and memory. However, these rules can take quantitatively and qualitatively different forms under different context of extrinsic and intrinsic modulation. For example, neuromodulator dopamine can dramatically enhance the sensitivity of STDP while reducing its temporal contrast in cultured hippocampal neurons. In addition, the evolution of network activity also suggests that plasticity as well as its molecular mediator can be altered by activity history of different timescales. Furthermore, synaptic plasticity rules characterized in vivo can be shifted by severe stress and rescued by antidepressant. Thus, modulation of synaptic plasticity is likely to play important roles in both physiological and pathological learning.
A calcium-based plasticity model explains sensitivity of synaptic changes to spike pattern, rate and dendritic location
Presynaptic NMDA receptors as burst detectors. How the biophysical properties of a receptor define a plasticity rule
Synaptic plasticity stands for changes in the functional properties of synapses as a result of their history. Different activity patterns may result in different forms of plasticity. We study how activity patterns are selected. More precisely which actors at the molecular level determine the activity patterns resulting in synaptic plasticity.
In young rodents, the glutamatergic cerebellar synapses between parallel fibers and Purkinje cells lack of postsynaptic NMDA receptors. However, synaptic plasticity is NMDA receptor dependent. We show how presynaptic NMDA receptors are responsible for this effect. NMDA receptors need simultaneous bound glutamate and membrane depolarization. This makes of them widely recognized coincidence detectors. When present on a postsynaptic membrane this is translated into simultaneous presynaptic (synaptic glutamate) and postsynaptic (depolarization) activities. When present on a presynaptic membrane, the very same biophysical properties translate into a different learning rule. These receptors may function as autoreceptors in glutamatergic synapses but the necessary depolarization is here associated with the passage of action potentials. Here we show how only repetitive activity at precise frequencies can activate presynaptic NMDA receptors and thus result in plasticity.
Storage of correlated patterns in binary Purkinje cell models
Purkinje cells (PCs) of the cerebellar cortex have long been considered to perform similarly as perceptrons: Given an input pattern in the granular layer, they should learn to provide an adequate motor output, thanks to plasticity of the parallel fiber (PF) to PC synapses, under the supervision of the climbing fiber input which is assumed to carry an error signal (Marr 1969, Albus 1971). Supervised learning in the perceptron model has been studied extensively in the case of random uncorrelated input/output associations. In particular, it is known that when synapses are constrained to be positive (to account for the fact that PF-PC synapses are excitatory), the synaptic weight distribution at maximal storage capacity is composed of a large fraction of zero-weight synapses (‘silent’ synapses, Brunel et al. 2004). However, in the case of the cerebellum, the assumption of uncorrelated inputs and outputs is clearly unrealistic, as any naturalistic inputs/motor sequences will carry some substantial degree of temporal correlations.
We therefore investigated both the capacity and the optimal connectivity in feed-forward networks learning associations between temporally correlated input/output sequences. We first consider a perceptron with binary inputs and outputs, in which sequences are defined as simple Markov chains with a given correlation. We show analytically that the capacity is independent of the correlation in the output if inputs are not correlated. However, if the inputs are correlated, we show numerically that the capacity grows with output correlation. The weight distribution at maximal capacity is shown to be independent on the level of the correlations and has a finite and large fraction of silent synapses.
We then consider a bistable output to mimic the postulated bistability of the PC (Yartsev et al. 2009, Loewenstein et al. 2005, Williams et al. 2002, Oldfield et al. 2010). We show analytically that, for a given output correlation, there is an optimal bistable range maximizing the capacity. In addition, we show numerically that the capacity indeed grows with the optimal bistability, however only when the output correlation is bigger than the input correlation. The weight distribution of the PF-PC synapses also consists of a large number of silent synapses.
Activity-dependent regulation of intrinsic excitability in hippocampal neurons: Hebbian plasticity and homeostatic regulation
Synaptic plasticity is not the exclusive mode of memory storage, and persistent regulation of voltage-gated ionic channels also participates to information storage. Long-term changes in neuronal excitability have been reported in several brain areas following learning. Synaptic activation of glutamate receptors initiates long-lasting modification in neuronal excitability (reviewed in Debanne & Poo, Front Syn Neurosci 2010). I will first present experimental evidence showing that the spike-timing-dependent plasticity (STDP) rule defined for synaptic transmission is also valid for plasticity of dendritic integration in CA1 pyramidal neurons (Campanac & Debanne, J Physiol 2008). Regulation of dendritic integration requires NMDA receptor activation, results from modification in EPSP amplification and is input specific. The role of hyperpolarization-activated cationic (H) channels in the expression of facilitation in dendritic integration will be discussed (Campanac et al., J Neurosci 2008). Synergistic potentation in synaptic and intrinsic excitation may eventually lead to hyper-excitable circuits if they are not compensated. I will discuss two possible mechanisms limiting over-excitation in hippocampal CA1 circuits: i) homeostatic regulation of intrinsic excitability in pyramidal neurons and ii) long-lasting potentiation in intrinsic plasticity in a subset of GABAergic interneurons.
Functional Requirements for Reward-modulated STDP
As a biologically plausible paradigm for learning in spiking neural networks, spike-timing dependent plasticity (STDP) has been shown to perform well in unsupervised learning tasks such as receptive field development. However, STDP fails to take behavioral relevance into account, and as such is inadequate to explain a vast range of learning tasks in which the final outcome is conditioned
on the prior execution of a series of actions.
In this talk, I will show that the addition of a third, global, reward-based factor to the pre- and post-synaptic factors of STDP is a promising solution to this problem, with strong experimental and theoretical motivations. I will derive simple functional requirements for these rules, and illustrate them in a motor sequence learning task. These requirements suggest that the brain needs a “critic” structure, constantly evaluating the potential for rewarding events. I will propose a biologically plausible implementation of such a structure, that performs motor or navigational tasks.
Long-term cortical synaptic plasticity improves sensory perception
Synapses and receptive fields of the cerebral cortex are plastic. However, changes to specific inputs must be coordinated within neural networks to ensure that excitability and feature selectivity are appropriately configured for perception of the sensory environment. Here I will discuss previous and unpublished results showing how long-lasting positive and negative changes to auditory cortical synapses were induced by pairing sounds with activation of the cholinergic neuromodulatory system. Synaptic modifications were precisely orchestrated across entire receptive fields, conserving mean excitation while reducing overall variance, and parameters of cortical synaptic receptive fields (here, sound frequency and intensity) could be modified independently of the other. Computational analysis indicated that decreased variability should increase detection and discrimination of near-threshold or previously imperceptible stimuli, and we confirmed this psychophysically in behaving animals. I will also describe newer results comparing the effects of acetylcholine to the other main attention-related modulator, noradrenalin. Our work indicates that direct modification of specific cortical inputs leads to wide-scale synaptic changes, which collectively support improved sensory perception and enhanced behavioral performance.
A multi-stage memory model: the computational advantage of memory consolidation processes
Long-term memories are likely stored in the synaptic weights of neuronal networks in the brain. The storage capacity of these networks depends on the plasticity of the synapses, as very plastic synapses provide for strong memories which are quickly overwritten, while unlabile synapses result in long-lasting yet weak memories.
Here we show that the trade-off between memory strength and lifetime can be overcome by initially storing memories in a highly plastic network, which then transfers patterns of synaptic weights to less plastic downstream networks during an off-line mode. This model is reminiscent of the process of memory consolidation, whereby memories are transferred from the hippocampus to cortical sites for long-term storage. This work has been in done in collaboration with Alex Roxin.
Modeling Synaptic Plasticity across Multiple Timescales
Induction of synaptic plasticity depends on voltage, spike timing, and frequency- as known from in vitro experiments. But synaptic plasticity can play its role for memory ‘in vivo’ only if the induced changes are stable for a long time after the induction. In this talk, present our model of plasticity induction and stabilization.
The induction step depends on postsynaptic voltage and, as a consequence, also in spike timing and frequency. As in the tagging hypothesis of Frey and Morris, the induction of plasticity sets tags at the synapses, but a further triggering step is needed before the synapse is consolidated.
A few applications of the induction and stabilization process in the Tag-Trigger-Consolidation model are presented, with a potential link to behavioral results.
Claudia Clopath, Lars Busing, Eleni Vasilaki and Wulfram Gerstner (2010)
Connectivity reflects coding: A model of voltage-based spike-timing-dependent-plasticity with homeostasis.
Nature Neuroscience, 13:344 – 352
Claudia Clopath, Lorric Ziegler, Eleni Vasilaki, Lars Busing, Wulfram Gerstner (2008)
Tag-Trigger-Consolidation: A Model of Early and Late Long-Term-Potentiation and Depression
PLoS Computational Biology 4(12): e1000248
Spike-timing-dependent plasticity in recurrently connected networks
This talk will outline the relationship between the weight dynamics generated by spike-timing-dependent plasticity (STDP) and the emergence of network structure in a recurrently connected network stimulated by external pools of input spike trains, where both input and recurrent synapses are plastic. The weight dynamics is determined by interplay between the neuronal activation mechanisms, the input spike-time correlations, dendritic delays, and the learning parameters. A theoretical framework is used that is based upon Poisson neurons with a temporally inhomogeneous firing rate and the asymptotic distribution of weights generated by the learning dynamics. Different network configurations will be discussed and an overview of the current understanding of STDP in recurrently connected neuronal networks presented, in which the resulting neuronal self-organization can be seen as a first step towards the emergence of neuronal maps induced by activity-dependent plasticity.
Calcium-dependent plasticity in cerebellar Purkinje cells
Cerebellar parallel fiber (PF) – Purkinje cell synapses can undergo postsynaptically expressed long-term depression (LTD) or long-term potentiation (LTP) depending on whether or not the climbing fiber (CF) is coactivated during tetanization. Larger calcium transients are required for LTD than for LTP induction, thus cerebellar bidirectional plasticity is governed by induction rules that provide a ‘mirror image’ to their neocortical and hippocampal counterparts. The LTD / LTP balance can be shifted by factors that influence synaptic strength and /or calcium signaling, such as ampakine drugs. Similarly, plasticity of the instructive CF signal can shift the probabilities for LTD and LTP induction, respectively. Finally, a novel type of intrinsic plasticity in Purkinje cells, which is mediated by a modulation of small-conductance calcium-dependent SK2-type K channels, affects the LTD / LTP balance, and furthermore affects the Purkinje cell spike output.
Changing a visual cortex neuron’s outlook using in vivo STDP: rules of engagement
Stimulus features that cause a neuron to spike emerge during sensory map formation and can change with experience, and after removal of visual input. Although visual cortex neuronal responses and receptive fields in the mammalian brain are known to be highly plastic, how these large-scale changes in spiking across neuronal populations arise from single spikes in individual neurons remains unclear. While in vivo STDP-like plasticity protocols have produced small and short term changes in receptive fields and orientation tuning and weak and variable subthreshold receptive field shifts, these were not as striking as model predictions and cannot account for suprathreshold receptive field changes across neuronal populations in vivo. What is missing is the link between the observed changes in spiking across the receptive field and changes in subthreshold responses induced by STDP at the single neuron level. Therefore it remains unclear whether an STDP protocol with a few tens of spikes can induce plasticity reliably in vivo to change a neuron’s stimulus response from non-spiking to spiking, and whether these changes can simultaneously alter both the subthreshold and suprathreshold receptive fields as strongly as observed with traditional protocols based on firing rates. In this presentation I will outline some results from our lab addressing these issues.
Synaptic Tagging, Evaluation of Memories, and the Distal Reward Problem
A network model of online learning will be motivated by in-vivo and in-vitro physiology of the hippocampus, specifically sequence replay during sharp-wave rippels in the slow-wave-sleep state. Subsequently this model will be used to study the effect of synaptic tagging on memory retention times. Synaptic tagging is implemented by multistate synapses with unstable early and stable late states. It is shown that an
equilibrium distribution exists, which is very susceptible to the storage of new items: Most synapses are in a state in which they are plastic and can be changed easily, whereas only those synapses that are essential for the retrieval of the important memory items are in the stable late phase. The model can solve the distal reward problem, where the initial exposure of a memory item and its evaluation are temporally separated. Synaptic tagging hence provides a viable mechanism to consolidate and evaluate memories on a synaptic basis.
The morning after: the consequences of synaptic plasticity for memory retrieval
Storing memories of the past is only useful if we can retrieve them in the future. While substantial work has been done to analyse the amount of information stored by various synaptic plasticity rules, it is unclear how efficiently the information once stored can be retrieved by neural circuit dynamics. We have shown that retrieving memories from synaptic efficacies presents unique computational challenges and predicted how the retrieval dynamics of a circuit need to be matched to the properties of the synaptic plasticity tule that was used to store those memories. In particular, we have shown that spike timing-based interactions between CA3 pyramidal cells in the hippocampus are optimally matched to retrieve memories stored by spike timing-dependent plasticity. More recently, we have analysed the effects of the limited dynamical range of synapses on memory retrieval and found that several motifs of hippocampal circuit dynamics, including feedback inhibition, homeostatic plasticity, and network oscillations, can be seen as adaptations to the task of retrieving palimpsest memories from such synapses.
The control of cerebellar long-term depression by NMDA receptors of mature Purkinje cells
LTP—-Solved and Unsolved Problems about this Memory Mechanism
I will try to give an overview of the LTP process, with special emphasis on the cells of the CA1 hippocampal region. The ability to excite single spines using 2-photon uncaging has established that LTP is specific to the stimulated spine. Under most conditions, LTP induction is dependent on Ca entry through NMDARs and the subsequent activation of the enzyme CaMKII. In order for NMDARs to open, the postsynaptic cell must be depolarized. Synatically-induced LTP does not backpropagating Na spikes; thus the depolarization necessary to open NMDARs probably results from NMDA spikes and Ca spikes. It now seems clear that the spine head is a quasi-independent electrical compartment and that understanding spine-dendrite voltage gradients will be important for understanding the events in spines during plasticity. Both Ca elevation and CaMKII activation are localized to the active spine and thus explain the synapse-specificity of LTP. The diffusion restriction provided by the thin spine neck is important for this localization. The Ca signals in spines are integrated by CaMKII over about one minute; integration involves the cumulate autophosphorylation of T-286. Activated CaMKII binds to the NMDAR, where it enhances AMPAR-mediated transmission by 1) phosphorylating GluR1, thereby enhancing AMPA channel conductance and 2) phosphorylating the AMPAR auxiliary subunit, stargazin, which allows more AMPA channels to be captured at the synapse by binding to PSD-95. This explains early LTP, but later, protein-synthesis dependent late-LTP requires additional processes. Late LTP is “neoHebbian”; a third factor, dopamine activation of D1 receptors, is required to allow the required protein synthesis. Dopamine is released by reward, aversive stimuli and novel stimuli, making late LTP dependent on systems-level processes. Late LTP involves a trans-synaptic process that enlarges the synapse; there is thus more presynaptic release of vesicles and more postsynaptic AMPAR. The molecular mechanisms that maintain late LTP involve a persistent CaMKII/NR2B complex. Adhesion proteins such as cadherins coordinate the presynaptic and postsynaptic growth. The question of how these changes persist despite protein turnover remains to be worked out.
Regulation of postsynaptic excitability by muscarinic receptors and the facilitation of synaptic plasticity
Activation of muscarinic acetylcholine receptors (mAChR) facilitates the induction of synaptic plasticity and enhances cognitive function. In the hippocampus, M1 mAChR on CA1 pyramidal cells inhibit both small conductance
Ca2+-activated KCa2 potassium channels and voltage-activated Kv7
channels. Inhibition of KCa2 channels facilitates long-term potentiation
(LTP) by enhancing Ca2+calcium influx through postsynaptic NMDA receptors (NMDAR). Inhibition of Kv7 channels also facilitates LTP by enhancing depolarisation during and after a burst of action potentials. Thus, during the induction of LTP by coincident presynaptic and postsynaptic activity,
Memory, associations and solving cognitive tasks by plasticity in randomly connected neural circuits
The innate properties of neurons, when coupled with experience of their local biophysical environment, enable them to connect together into neural circuitry that can produce an amazing wealth of animal behaviors.
We focus on how local, biophysically realistic synaptic plasticity rules can sculpt initially randomly connected recurrent circuits of spiking neurons and enable animals to solve behavioral tasks. We focus on tasks fundamental to cognition that require responses to specific pairs of stimuli and/or require short-term memory. We find that plasticity of inhibitory-to-excitatory synapses enhances the selectivity of responses, improving the network’s ability to produce solutions to tasks that require the formation of stimulus-pair selective cells. However, functional and structural plasticity of connections between excitatory cells are required to produce the selective persistent activity necessary to solve tasks with a delay between stimuli. Using a winner-takes-all decision-making network as a behavioral readout, we show how synaptic facilitation boosts performance in our tasks as its intrinsic nonlinearity can produce the solution to linearly non-separable tasks. In many cases network behavior is very sensitive to initial conditions and parameters, suggesting the need for homeostatic mechanisms stronger than those characterized to date.
Role of NMDA receptors in Hebbian and non-Hebbian cortical synaptic plasticity
Timing-dependent long-term potentiation (t-LTP) and timing-dependent long-term depression (t-LTD) both depend on NMDA receptors. At cortical layer 4-to-layer 2/3 synapses, we recently demonstrated that, whilst NMDA receptors responsible for induction of t-LTP are located postsynaptically, NMDA receptors required for induction of t-LTD are located presynaptically. In this presentation, I will discuss experimental data demonstrating more precisely the location of these receptors, and the locus of expression of t-LTD. I will show that postsynaptic metabotropic and Ca2+-dependent mechanisms are, in fact, not required for induction of LTD at this synapse, and discuss the computational versatility of employing presynaptic, rather than postsynaptic, NMDA receptors in the induction of LTD.
Synaptic Learning as Dynamic Filter
A minimal set of differential equations is proposed for variables driven by pre- and postsynaptic activities that in combination contribute to synaptic changes. While this phenomenological model can quantitatively reproduce a range of experiments on spike-timing dependent plasticity the postulated variables are related to known biophysical processes at the synapse. In particular, we find that mechanisms related to NMDA-receptors and Calcium are critical for explaining the dynamics of factors contributing to the LTP-part of synaptic plasticity. The proposed contribution dynamics makes a range of testable predictions. In particular, it reveals a maximal sensitivity of synaptic modifyability to oscillatory firing rate modulations to a frequency in the theta-range. When modulated by a factor signaling reward the model can be used as a learning rule for excitatory connections in recurrent networks. However, in this case weight changes depend on an intricate mixture of reward-related and activity dependent signals which I will discuss.
A triplet STDP model generalizes the BCM rule to higher-order spatio-temporal correlations
Spike-Timing Dependent Plasticity (STDP) is often expressed as a function of pairs of spikes (1 pre and 1 post). However, it has been shown recently that triplets of spikes (1 pre and 2 post) are more appropriate to describe a wide range of experimental data on synaptic plasticity. In this talk I will describe the functional consequences of the triplet model of STDP. Firstly, I will demonstrate that in the case of rate-based patterns this triplet STDP model can be mapped to the well-known Bienenstock-Cooper-Munro (BCM) synaptic learning rule, which has been shown to maximize the selectivity of the postsynaptic neuron. Secondly, I will show that for input patterns consisting of higher-order spatio-temporal correlations, the triplet STDP rule also elicits selectivity thereby generalizing the BCM learning rule. Moreoever this sensitivity to higher-order correlations can be used to develop direction and speed selectivity.
Neuronal correlates for basic terms of reward prediction, reinforcement learning and value updating.
The functions of rewards are based primarily on their effects on behavior and are less directly governed by the physics and chemistry of input events as in sensory systems. Therefore the investigation of neural mechanisms underlying reward functions requires behavioral theories that conceptualize the different effects of rewards on behavior. The investigation of behavioral processes by animal learning theory has produced a theoretical framework that can help to elucidate the neural correlates for reward functions in reinforcement learning. Individual neurons are studied in the reward systems of the brain, including dopamine neurons, orbitofrontal cortex and amygdala. Neural activity in these brain structures can be related to the basic theoretical terms that govern reward prediction, learning and value updating, namely contingency and prediction error.
STDP modulated by global factors: the more, the better
Decision making and learning in general is likely involving neuronal populations and global modulatory factors across several stages. For a single synapse to make a meaningful update it is important to have access to downstream information relevant in the decision process. In a reinforcement learning scenario where binary decisions are based on the activity of a neuronal population this information includes both the global population signal and the reward signal. We show how a spike-timing dependent plasticity (STDP) rule can be derived from maximizing the expected reward. The rule depends on 4 factors: the pre- and postsynaptic activities, the population- and the reward signal. These signals are integrated within a single synapse by eligibility traces of increasing memory lengths reflecting the processing time of the synapse, the neuron, the decision making, and the reward feedback, respectively. We show that the rule succeeds in quickly learning stimulus-response associations, that it can deal with long reward delays, and that it learns non-Markovian tasks where e.g. TD-learning fails. We also apply the rule to the classical worker-employer game and show that it finds the Nash-equilibrium and reproduces human learning curves.
Learning Reward Timing Using Reinforced Expression of Synaptic Plasticity
Neuronal plasticity affects not only the static properties of cortical neurons, but the temporal aspects of their responses as well. One relatively simple example of temporal dynamics with behavioral implications is the ability to represent interval timing. It has been commonly believed that the underlying neural mechanisms for representing time reside in high order cortical regions but recent studies show sustained neural activity in primary sensory cortices that can represent the timing of expected reward. Here I show that local cortical networks can learn temporal representations through a simple framework predicated on reward dependent expression of synaptic plasticity. I assert that temporal representations are stored in the lateral synaptic connections between neurons and demonstrate that reward-modulated plasticity is sufficient to learn these representations. I will also describe mean field theories that capture the major qualitative properties of the networks that can represent temporal intervals, and how this leads to a contrast invariant representation. The plasticity model depends on assumptions about the cellular mechanisms of synaptic plasticity. However, the methods currently used in experiments prevent us from being able to either support or reject these assumptions, and I will point out how such methods could be improved.
Balancing excitation and inhibition with inhibitory plasticity
A number of recent studies suggest that the excitatory synaptic input received by individual cortical neurons is balanced by concurrent inhibition, possibly down to the level of individual synaptic events.
This balance is thought essential for maintaining stability of cortical networks, it may explain the highly irregular activity patterns that are observed in cortical neurons and probably has a strong influence on how information propagates in the brain. Despite the recent interest in this phenomenon, no mechanism has been suggested that allows the establishment of such balanced networks and its maintenance in the presence of plasticity.
We show analytically and in simulations that such a balance can be dynamically established and maintained by a simple plasticity rule in inhibitory synapses (ISP). For networks with feedforward inhibition, ISP leads to a detailed balance: inhibitory synapses adapt such that excitatory and inhibitory currents develop the same stimulus tuning.
In recurrent networks, ISP establishes a balance that stabilizes the asynchonous irregular state that most resembles cortical activity patterns in vivo. When synaptic memories are introduced into the network in the form of Hebbian assemblies, ISP causes a rebalancing of the network until a global asynchronous irregular state is reached and network activity no longer reveals the memories. The memory can be recalled, however, by reactivating a subset of the neurons in a given assembly.
Title available soon