Collection of papers in cognitive neuroscience with a focus on using artificial neural networks to model brain data. There is also a section on brain-computer interfaces with indication which papers use artificial neural networks for brain decoding.
Decoding of attempted speech
- Online speech synthesis using a chronically implanted brain-computer interface in an individual with ALS, medRxiv, 2023 - ANNs
- A high-performance speech neuroprosthesis, Nature, 2023 - ANNs
- A high-performance neuroprosthesis for speech decoding and avatar control, Nature, 2023 - ANNs
- Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis, Nature Communications, 2022 - ANNs
- Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria, NEJM, 2021 - ANNs
Decoding of produced speech
- Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models, Journal of Neural Engineering, 2023
- Imagined speech can be decoded from low- and cross-frequency features in perceptual space, Nature Communications, 2022
- Synthesizing speech from intracranial depth electrodes using an encoder-decoder framework, arXiv, 2021 - ANNs
- Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity, Communications Biology, 2020
- Brain2Char: a deep architecture for decoding text from brain recordings, Journal of Neural Engineering, 2020 - ANNs
- Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus, Journal of Neural Engineering, 2020
- Machine translation of cortical activity to text with an encoder–decoder framework, Nature Neuroscience, 2020 - ANNs
- Speech synthesis from neural decoding of spoken sentences, Nature, 2019 - ANNs
- Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis, Elife, 2019.
- Speech synthesis from ECoG using densely connected 3D convolutional neural networks, Journal of Neural Engineering, 2019 - ANNs
- Generating natural, intelligible speech from brain activity in motor, premotor, and inferior frontal cortices, Frontiers in Neuroscience, 2019
- Decoding inner speech using electrocorticography: progress and challenges toward a speech prosthesis, Frontiers in Neuroscience, 2018
- Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids, NeuroImage, 2018
- Brain-to-text: decoding spoken phrases from phone representations in the brain, Frontiers in Neuroscience, 2015
- Direct classification of all American English phonemes using signals from functional speech motor cortex, Journal of Neural Engineering, 2014
- Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2014
- Neural decoding of single vowels during covert articulation using electrocorticography, Frontiers in Human Neuroscience, 2014
- Decoding spectrotemporal features of overt and covert speech from the human cortex, Frontiers in Neuroengineering, 2014
- Structured neuronal encoding and decoding of human speech features, Nature Communications, 2012
- Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech-motor cortex, Frontiers in Neuroscience, 2011
- Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans, Journal of Neural Engineering, 2011
- Using the electrocorticographic speech network to control a brain–computer interface in humans, Journal of Neural Engineering, 2011
- Decoding spoken words using local field potentials recorded from the cortical surface, Journal of Neural Engineering, 2010
- Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2008
- A survey of signal processing algorithms in brain–computer interfaces based on electrical brain signals, Journal of Neural Engineering, 2007
Decoding of perceived speech
- Stimulus Speech Decoding from Human Cortex with Generative Adversarial Network Transfer Learning, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), 2020 - ANNs
- Towards reconstructing intelligible speech from the human auditory cortex, Scientific Reports, 2019 - ANNs
- Reconstructing Speech Stimuli From Human Auditory Cortex Activity Using a WaveNet Approach, 2018 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), 2018 - ANNs
- Real-time classification of auditory sentences using evoked cortical activity in humans, Journal of Neural Engineering, 2018
- Word pair classification during imagined speech using direct brain recordings, Scientific Reports, 2016
- Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity, Journal of Neural Engineering, 2016
- Reconstructing speech from human auditory cortex, PLOS Biology, 2012
Other
- High-performance brain-to-text communication via imagined handwriting, bioRxive, 2020 - ANNs
- Sequence transfer learning for neural decoding, bioRxiv, 2017 - ANNs
Language and semantics
- The neural architecture of language: Integrative modeling converges on predictive processing, PNAS, 2021
- Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines, bioRxiv, 2021
- GPT-2’s activations predict the degree of semantic comprehension in the human brain, bioRxiv, 2021
- Cortical processing of reference in language revealed by computational models, bioRxiv, 2021
- Semantic knowledge of famous people and places is represented in hippocampus and distinct cortical networks, Journal of Neuroscience, 2021
- Voxelwise encoding models show that cerebellar language representations are highly conceptual, bioRxiv, 2021
- A hierarchy of linguistic predictions during natural language comprehension, bioRxiv, 2021
- Cortical network responses map onto data-driven features that capture visual semantics of movie fragments, Scientific reports, 2020
- Language processing in brains and deep neural networks: computational convergence and its limits, bioRxiv, 2020
- Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech, bioRxiv, 2020
- Multi-timescale representation learning in LSTM Language Models, bioRxiv, 2020
- Connecting concepts in the brain by mapping cortical representations of semantic relations, Nature Communications, 2020
- Incorporating context into language encoding models for fMRI, bioRxiv, 2018
- The hierarchical cortical organization of human speech processing, Journal of Neuroscience, 2017
- Natural speech reveals the semantic maps that tile human cerebral cortex, Nature, 2016
- A continuous semantic space describes the representation of thousands of object and action categories across the human brain, Neuron, 2012
Auditory processing & speech perception
- Training neural networks to recognize speech increased their correspondence to the human auditory pathway but did not yield a shared hierarchy of acoustic features, bioRxiv, 2021
- Brain-optimized extraction of complex sound features that drive continuous auditory perception, PLOS Computational Biology, 2020
- Estimating and interpreting nonlinear receptive field of sensory neural responses with deep neural network models, Elife, 2020
- Towards reconstructing intelligible speech from the human auditory cortex, Scientific reports, 2019
- Interpretation of convolutional neural networks for speech spectrogram regression from intracranial recordings, Neurocomputing, 2019
- A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy, Neuron, 2018
- Modeling brain responses to perceived speech with LSTM networks, 2017 Benelearn conference proceedings, 2017
- Brains on beats, NIPS, 2016
Vision
- Brain2Pix: Fully convolutional naturalistic video reconstruction from brain activity, bioRxiv, 2021
- End-to-end neural system identification with neural information flow, PLOS Computational Biology, 2021
- Hyperrealistic neural decoding: Linear reconstruction of face stimuli from fMRI measurements via the GAN latent space, bioRxiv, 2020
- Individual differences among deep neural network models, Nature Communications, 2020
- Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision, PLOS Computational Biology, 2020
- Recurrence is required to capture the representational dynamics of the human visual system, PNAS, 2019
- Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior, Nature Neuroscience, 2019
- Activations of deep convolutional neural networks are aligned with gamma band activity of human visual cortex, Nature Communications Biology, 2018
- Generative adversarial networks for reconstructing natural images from brain activity, Neuroimage, 2018
- Convolutional neural network-based encoding and decoding of visual object recognition in space and time, Neuroimage, 2018
- Modeling the dynamics of human brain activity with recurrent neural networks, Frontiers in Computational Neuroscience, 2017
- Using goal-driven deep learning models to understand sensory cortex, Nature Neuroscience, 2016
- Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream, Journal of Neuroscience, 2015
- Deep learning for neuroimaging: a validation study, Frontiers in Neuroscience, 2014
- Deep supervised, but not unsupervised, models may explain IT cortical representation, PLOS Computational Biology, 2014
Reviews
- Direct fit to nature: an evolutionary perspective on biological and artificial neural networks, Neuron, 2020
- A critique of pure learning and what artificial neural networks can learn from animal brains, Nature Communications, 2019
- Neural network models and deep learning, Current Biology, 2019
- A deep learning framework for neuroscience, Nature Neuroscience, 2019
- Deep neural networks in computational neuroscience, bioRxiv, 2017
- Toward an integration of deep learning and neuroscience, Frontiers in Computational Neuroscience, 2016
- Deep neural networks: a new framework for modelling biological vision and brain information processing, bioRxiv, 2015
This section is difficult to keep up to date with, only the most influential or personally relevant papers are mentioned.
Convolutional neural networks
- Recent Advances in Convolutional Neural Networks
- The Reversible Residual Network: Backpropagation without Storing Activations
- Deep Networks with Stochastic Depth
Recurrent neural networks
- Supervised Sequence Labelling with Recurrent Neural Networks
- Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
Recurrent convolutional neural networks
- An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition
- Recurrent Convolutional Neural Network for Object Recognition
- Inception Recurrent Convolutional Neural Network for Object Recognition
Feature visualization
- Visualizing and Understanding Convolutional Networks
- Visualizing Higher-layer Features of a Deep Network
- Understanding Neural Networks Through Deep Visualization
- Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
- Learning Deep Features for Discriminative Localization
- Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline
- Visualizing and Understanding Recurrent Networks
Neural network pruning
- Fairseq - many implemented papers on language and sequence modeling, pretrained models shared
- WaveGan and SpecGan - official implementation of Adversarial Audio Synthesis, TensorFlow
- Parallel WaveGan implementation - unofficial implementation of Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram
- ESPnet - toolking for training ASR and TTS models, PyTorch and Chainer, Kaldi recepies
- Google Brain Magenta - several models for music generation and compressed representations including implementation of Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
- Detectron Facebook - models for visual object detection, Caffe2
- Fasttext Facebook - subword alternative to word2vec
- Netscope - tool for visualizing network architectures
- Optuna - hyperparameter optimization for neural nets, supports TensorFlow, Keras, PyTorch, Chainer
- Image Caption Generation with Chainer