Show simple item record

dc.contributor.authorSoo, Wayne W.M.
dc.contributor.authorLengyel, Máté
dc.date.accessioned2023-08-11T09:28:35Z
dc.date.available2023-08-11T09:28:35Z
dc.date.issued2022
dc.identifier.isbn9781713871088
dc.identifier.urihttp://hdl.handle.net/20.500.14018/14090
dc.description.abstractThere continues to be a trade-off between the biological realism and performance of neural networks. Contemporary deep learning techniques allow neural networks to be trained to perform challenging computations at (near) human-level, but these networks typically violate key biological constraints. More detailed models of biological neural networks can incorporate many of these constraints but typically suffer from subpar performance and trainability. Here, we narrow this gap by developing an effective method for training a canonical model of cortical neural circuits, the stabilized supralinear network (SSN), that in previous work had to be constructed manually or trained with undue constraints. SSNs are particularly challenging to train for the same reasons that make them biologically realistic: they are characterized by strongly-connected excitatory cells and expansive firing rate non-linearities that together make them prone to dynamical instabilities unless stabilized by appropriately tuned recurrent inhibition. Our method avoids such instabilities by initializing a small network and gradually increasing network size via the dynamics-neutral addition of neurons during training. We first show how SSNs can be trained to perform typical machine learning tasks by training an SSN on MNIST classification. We then demonstrate the effectiveness of our method by training an SSN on the challenging task of performing amortized Markov chain Monte Carlo-based inference under a Gaussian scale mixture generative model of natural image patches with a rich and diverse set of basis functions -- something that was not possible with previous methods. These results open the way to training realistic cortical-like neural networks on challenging tasks at scale.
dc.language.isoeng
dc.publisherNeural Information Processing Systems Foundation, Inc. (NeurIPS)en_US
dc.relation.ispartofAdvances in Neural Information Processing Systems 35 (NeurIPS 2022)
dc.titleTraining stochastic stabilized supralinear networks by dynamics-neutral growthen_US
dc.typeConference paper
dc.source.spage1
dc.source.epage13
dc.description.versionPublished version
refterms.dateFOA2023-08-12T01:45:21Z
dc.identifier.urlhttps://proceedings.neurips.cc/paper_files/paper/2022/hash/bc827452450356f9f558f4e4568d553b-Abstract-Conference.html


Files in this item

Thumbnail
Name:
Soo-Wayne_2022.pdf
Size:
996.9Kb
Format:
PDF

This item appears in the following Collection(s)

Show simple item record