Guclu_and_van_Gerven_2015_ventral_representation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Guclu and van Gerven 2015-ventral representation.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/adowaconan/Deep_learning_fMRI/blob/master/Guclu_and_van_Gerven_2015_ventral_representation.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "ZyHBywM9cK6r",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from IPython.display import Image"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TgahA6IYddVR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Introduction"
      ]
    },
    {
      "metadata": {
        "id": "ogVbbvjedmp2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "1. object recognition appears to be solved in the primate brain via a cascade of neural computational along the visual ventral stream that **represent increasingly complex stimulus features**, which derive from retinal input (Tanaka, 1996)\n",
        "2. neurons in early visual areas have smaller receptive fields and respond to simple features such as edge orientations, whereas neurons further along the ventral pathway have larger receptive fields and are more invariant to transformations and can be selective for complex shapes (Hubel and Wiesel, 1962; Gross et al., 1972; Hung et al., 2005)\n",
        "3. while the receptive fields in early visual have been characterized in terms of preferred orientation, location, and spatial frequency (Jones and Palmer, 1987)\n",
        "4. exactly what stimulus features are represented in downstream areas is less clear (Cox, 2014)\n",
        "5. $mapping ({DNN}\\rightarrow{image})$, deeper layers can be shown to respond to increasingly complex stimulus features (Zeiler and Fergus, 2012)\n",
        "6. past works (Key et al., 2008; **van Gerven, et al., 2010 [unsupervised]()**; Guclu and van Gerven, 2014 [linear response model]())\n",
        "7.  \\*individual neural network layers were used to predict single-voxel responses to natural images, and this allowed us to isolate different voxel groups, whose population receptive fields are best predicted by a particular neural network layer (Dumoulin and Wandell, 2008)\n",
        "\n",
        "## goal\n",
        "$$mapping (\\text{receptive fields},{DNN_{layers}})$$\n",
        "$$complextity$$\n",
        "$$invariant$$\n",
        "$$size$$\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "xN8TKzINEJFt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 838
        },
        "outputId": "f46354f9-0310-489e-8bae-a8fd28f51fa7"
      },
      "cell_type": "code",
      "source": [
        "print('Unsupervised feature learning improves prediction of human brain activity in response to natural images')\n",
        "Image(url='http://journals.plos.org/ploscompbiol/article/figure/image?size=large&id=10.1371/journal.pcbi.1003724.g001',height=800)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Unsupervised feature learning improves prediction of human brain activity in response to natural images\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"http://journals.plos.org/ploscompbiol/article/figure/image?size=large&id=10.1371/journal.pcbi.1003724.g001\" height=\"800\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "metadata": {
        "id": "o9Qz9k9WdKwL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 556
        },
        "outputId": "d577c3c6-df97-4c30-d3c2-6c6e31000f65"
      },
      "cell_type": "code",
      "source": [
        "print(\"Tanaka, 1996\\nKravitz et al., 2014\")\n",
        "Image(url='http://www.jneurosci.org/content/jneuro/35/27/10005/F1.large.jpg?width=800&height=600&carousel=1',height=500)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tanaka, 1996\n",
            "Kravitz et al., 2014\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"http://www.jneurosci.org/content/jneuro/35/27/10005/F1.large.jpg?width=800&height=600&carousel=1\" height=\"500\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "metadata": {
        "id": "LKikKN8XhE8N",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Method\n",
        "\n",
        "## 2 male subjects\n",
        "## 5 sessions of data\n",
        "## training images - 1750, each was repeated twice\n",
        "## testing images - 120, each was repeated 13 times\n",
        "\n",
        "## MRI\n",
        "### slice thickness: 2.25 mm\n",
        "### slice gap 0.15 mm\n",
        "### field of view 128 x 128 $mm^2$\n",
        "### TR: 1s\n",
        "### TE: 28 ms\n",
        "### spatial resolution: 2 x 2 x 2.5 $mm^3$\n",
        "\n",
        "## preprocessing\n",
        "### fMRI scans were coregistered and used to estimate voxel-specific response time course\n",
        "### deconvolution (Zeiler and Fergus, 2012) $\\rightarrow$ estimate of response amplitude  for each each unique image in each voxel $\\rightarrow$ voxels were assigned to visual areas using retinotopic mapping data acquired in separate session $\\rightarrow$ anatomical and functional volumes were coregistered manually (free surfer)"
      ]
    },
    {
      "metadata": {
        "id": "zUASYXWrijzJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Encoding and Decoding model\n",
        "\n",
        "## transforming image features to abstract representations ---- encoding\n",
        "feature model that transform a visual stimulus to a **nonlinear** feature representation (Chatfield et al., 2014; Krizhevsky et al., 2012)\n",
        "## transforming abstract representations to BOLD responses ---- decoding\n",
        "1. a **linear** response model that transforms nonlinear feature representations toa voxel response (Guclu and van Gerven, 2014)\n",
        "2. a separate response model was trained **for each feature map/voxel combination** using regularized linear regression [what are the regressors? what is the prediction?]()\n",
        "3. to examine which DNN layer was most predictive of individual respnoses, we used each of the 8 layers of feature representations as input\n",
        "4. estimation of the regression coefficients $\\beta_i$, \n",
        "$$\\mu_i(\\mathbf{x})=\\beta_i^T\\Phi(\\mathbf{x})$$\n",
        "$$\\text{as the predicted response of voxel }i\\text{ to input stimulus }\\mathbf{x}\\text{ given a chosen feature representation }\\Phi(\\mathbf{x})$$ "
      ]
    },
    {
      "metadata": {
        "id": "XRZWXTtjmuJI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Similarities and differences compared to Autoencoder ([optional](http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/))\n",
        "1. neural network that is an **[unsupervised learning algorithm](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.408.1839&rep=rep1&type=pdf)(Baldi and Hornik, 1988)** that applies backprogation, setting the targets values to be equal to the inputs\n",
        "2. learn an **approximation to the identify function**, $h_{W,b}(x)\\approx x$, so as to output $\\hat{x}$ that is similar to $x$\n",
        "3. if there is structure in the data, for example, if some of the inputs $x$ features are correlated, then this algorithm will be able to discover some of those correlations. In fact, this simple autoencoder often ends up learning a low-dimensional representation very similar to **PCAs** [tSNE](http://scikit-learn.org/stable/auto_examples/manifold/plot_t_sne_perplexity.html#sphx-glr-auto-examples-manifold-plot-t-sne-perplexity-py)\n",
        "4. if the hidden layer is large, we can still discover interesting structure, by imposing other constraints on the network: [sparsity-denoising autoencoder](https://www.doc.ic.ac.uk/~js4416/163/website/autoencoders/denoising.html)\n",
        "5. [variational autoencoder](https://arxiv.org/pdf/1312.6114.pdf): for i.i.d datasets with continous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (aka recognition model) to the intractable posterior using the lower bound estimator\n",
        "6. latent space: highly abstract representation, not necessary informative\n",
        "7. can be used to map anything to anything **with constraints**\n"
      ]
    },
    {
      "metadata": {
        "id": "h6w-A587nHgN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "outputId": "12d9866b-c035-4e30-bc23-d7195e93cf69"
      },
      "cell_type": "code",
      "source": [
        "Image(url='https://www.doc.ic.ac.uk/~js4416/163/website/img/autoencoders/autoencoder.png')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://www.doc.ic.ac.uk/~js4416/163/website/img/autoencoders/autoencoder.png\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "metadata": {
        "id": "gcZBCiz_oT7q",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 421
        },
        "outputId": "b77e7911-6842-49d8-9191-ee75904afb25"
      },
      "cell_type": "code",
      "source": [
        "Image(url='https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/7a9b6d996fed89bc327a19c72adfe8f80e9b522e/4-Figure3-1.png',height=400)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/7a9b6d996fed89bc327a19c72adfe8f80e9b522e/4-Figure3-1.png\" height=\"400\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "metadata": {
        "id": "W_jrbBsKoZMT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 421
        },
        "outputId": "64b1b124-6570-42ad-a75a-f329ec8d0fff"
      },
      "cell_type": "code",
      "source": [
        "Image(url='https://www.researchgate.net/profile/Javier_Turek/publication/306258062/figure/fig1/AS:396269664129025@1471489458335/Proposed-4D-Convolutional-Autoencoder.png',height=400)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"https://www.researchgate.net/profile/Javier_Turek/publication/306258062/figure/fig1/AS:396269664129025@1471489458335/Proposed-4D-Convolutional-Autoencoder.png\" height=\"400\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "metadata": {
        "id": "w07hqhJWlqQU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Quantification of model performance\n",
        "1. a voxel's prediction accuracy as the pearson's $r$ between its observed and predicted responses on the *test set*\n",
        "2. to account for performance variability across voxels, we compared prediction accuracies of voxels with their SNRs and the mean activities of the DNN layers across the *training set*\n",
        "3. SNR was estimated as:\n",
        "$$\\frac{\\text{mean time series}}{\\text{median }\\{abs[diff(\\text{time point}_i,\\text{time point}_{i+1})]\\}}$$\n",
        "4. to computing the prediction accuracy for individual voxels, we can use the accuracy of reconstructing a presented image from observed brain activity as a measure of model performance\n",
        "$$\\text{compute the most probable stimulus by maximizing the likelihood}$$\n",
        "$$\\mathbf{x}^* = arg\\,max_{\\mathbf{x}\\in\\mathbf{X}}\\{-(\\mathbf{y} - \\mu(\\mathbf{x}))^T\\,\\Sigma^{-1}\\, (\\mathbf{y} - \\mu(\\mathbf{x}))\\}$$\n",
        "$$\\mu({\\mathbf{x}})\\text{ is the predicted response by the encoding model using the optimal layer asssignment for each voxel}$$\n",
        "$$\\Sigma \\text{ is an estimate of the noise covariance}$$(**what noise covariance it belongs to? How to estimate such noise covariance?**)\n",
        "5. those voxels that have the highest prediction accuracy onthe *test set* are chosen **without using the target stimulus** *(sounds like double dipping to me)*\n",
        "6. the identification accuracy is:\n",
        "$$\\frac{\\text{correctly identify test images from the {train and test combined}}}{120}$$\n",
        "\n",
        "## to improve decoding performance\n",
        "1. preditions were made by **refitting** an encoding model each voxel *(by what)*\n",
        "2. the receptive field of each voxel was estimated by **refitting** another set of encoding models that take as input all features in the preferred layer of the voxel at individual spatial locations. The receptive field was then taken as the spatial locations whose corresponding models accurately predicted the response of the voxel. *(why can you do that)*\n",
        "\n",
        "## control model\n",
        "1.  [gabor wavelet pyramid](https://www.researchgate.net/publication/220869739_A_Gabor_Wavelet_Pyramid-Based_Object_Detection_Algorithm)\n",
        "2. vgg-verydeep-16 and vgg-verydeep-19 (Simonyan and Zisserman, 2014)\n",
        "3. vgg-f (fast, strides = 4), vgg-m (medium), vgg-m-2048, vgg-m-1024, and vgg-m-128 (Charfield et al., 2014)\n",
        "4. caffe-ref [(Jia et al., 2014)](https://arxiv.org/pdf/1408.5093.pdf): fast feature embedding\n",
        "5. caffe-alex [(Krizhevsky et al., 2012)](http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf): the original Alex net"
      ]
    },
    {
      "metadata": {
        "id": "ILRBz4clzy75",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Analysis of internal representations\n",
        "1. deconvolutional network (Zeiler and Fergus, 2012) was used to reconstruct the internal representations of artifical neurons\n",
        "2. the image that maximally activates each artificial neuron was selected from the ImageNet *validation set*\n",
        "3. the image was first feed forward until it reached the layer of the neuron of interest\n",
        "4. all the activation except the maximum activation of the neuron were set to zero (sparsity)\n",
        "5. the activation of the neuron was deconvolved to produce a representation in image space\n",
        "$$\\text{inverting the order of the layers }\\rightarrow \\text{transposing the filters }\\rightarrow\\text{replacing max pooling with max uppooling}$$\n",
        "6. after an initial evaluation of the internal representations, **9** feature classes were defined such that they were representative of the most common *low-level (blob, contrast, and edge)*, *mid-level (contour, shape, and texture)*, and *high level (irregular pattern and object part and entire object)* internal representations of the 1888 artificial neurons in the convolutional layers (sounds like a small model)\n",
        "7. each of these neurons was assigned a **predefined label** by a naive subject across five-hour long sessions\n",
        "8. the subject was presented with 4 instantiations of the internal representations of the neurons *(tegether with the images that were used to reconstruct them)* in a random order and was asked to assign one of the following feature classes: see above\n",
        "9. each instatiation corresponded to the reconstruction of the internal representation of a neuron using one of th 4 images that activated the neuron the most (very confusing)"
      ]
    },
    {
      "metadata": {
        "id": "-VkdaDPV5mhL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Analysis of voxel groups\n",
        "1. individual voxels were assigned to their optimal according to their optimal layer according to maximal prediction accuracy computed using 5-fold cross validation on the *training data*\n",
        "2. voxels were grouped together according to their assigned neural network layer\n",
        "3. layer complexity is defined as the mean [Kolmogorove complexity](https://github.com/MLWave/koolmogorov) of the internal representations of the artificial neurons in that layer, approximated by their normalized compressed file size\n",
        "4. layer invariance is defined as the median full-width at half-maximum of 2D Gaussian surfaces that have been fitted to the 2D response surfaces of the artificial neurons in that layer (reflecting tolerance to small translations of a stimulus feature)\n",
        "## the reconstruction of the internal representation of the artificial neuron is shifted to different spatial locations (the brain)\n",
        "## the activity of the neurons is computed for each translation and a 2D response surface is constructed (I still don't know where is this \"2D response surface\" coming from)\n"
      ]
    },
    {
      "metadata": {
        "id": "lpf96u737lao",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Cluster analysis\n",
        "## don't understand at all\n",
        "1. goal: identify fine-grained structure within individual visual areas, use: **peralignment**Haxby et al., 2011) $\\rightarrow$ **nparametric Bayesian biclustering**Meeds and Roweis, 2007)\n",
        "\n",
        "## hyperalignment\n",
        "1. select {individual representational space of the subject that has the **most** number of voxels} as the initial common representational space\n",
        "2. the common representational space was then iteratively updated for 100 iterations\n",
        "3. for each iteration:\n",
        "        Procrustes transformation {project the individual functional data of the 2 subjects to the common representational space} after {which the common representational space was set to the **mean** of the individual functional data of the two subjects}\n",
        "4. each visual area was hyperaligned *separately*\n",
        "\n",
        "## nonparametric Bayesian biclustering\n",
        "1. simultaneously cluster rows:[individual feature maps] and columns:[region-specific voxels of the common representational space] of a z-scored prediction accuracy matrix\n",
        "2. Assumption:\n",
        "        the observed prediction accuracies for each feature map/voxel pair are drawn from a Gaussian with zero mean and unit standard deviation\n",
        "3. [Gibbs sampler - matlab](https://github.com/ppletscher/npbb)"
      ]
    },
    {
      "metadata": {
        "id": "pLc8_Aew908F",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Results\n",
        "\n",
        "1. assign voxels to one of the 8 layers of the DNN, each voxel was assigned to the layer of the DNN that resulted in the lowest CV error on the *training set*\n",
        "2. S1: $$\\text{3381 of 25915} = 0.1304$$only the main afferent pathway of the ventral stream $$\\text{1785 of 6017}= 0.2968$$\n",
        "3. S2: $$\\text{1185 of 26329} = 0.0450$$only the main afferent pathway of the ventral stream $$\\text{768 of 4875}=0.1575$$\n",
        "4. a combination of the striate and extrastriate decoding models would have a higher accuracy since the striate voxels can be used to resolve the ambiguities in the feature representations of the extrastriate voxels and vice versa"
      ]
    },
    {
      "metadata": {
        "id": "MQ2-QJ7edMDU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 438
        },
        "outputId": "a2a6154d-062a-4ea5-975c-3540441f2f7f"
      },
      "cell_type": "code",
      "source": [
        "print('to what extent the deep model allows decoding of a perceived stimulus from observed multiple voxel responses alone')\n",
        "Image(url='http://www.jneurosci.org/content/jneuro/35/27/10005/F2.large.jpg?width=800&height=600&carousel=1',height=400)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "to what extent the deep model allows decoding of a perceived stimulus from observed multiple voxel responses alone\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"http://www.jneurosci.org/content/jneuro/35/27/10005/F2.large.jpg?width=800&height=600&carousel=1\" height=\"400\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "metadata": {
        "id": "KwIWGAMq_LY3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The DNN model accurately predicts voxel responses across the occipital cortex. **A**, Prediction accuracies of the significant voxels across the occipital cortex (p < 2e-6 for both subjects, Bonferroni corrected for number of voxels, Student's t test across cross-validated training images within subjects). **B**, Prediction accuracies of the significant voxels across V1[striate](), V2, V4, and LO [extrastriate]() (p < 5e-8 for both subjects, Bonferroni corrected for number of layers and voxels, Student's t test across cross-validated training images within subjects). **C**, SNRs of the voxels across the occipital cortex.\n",
        "\n",
        "### identification - machine learning: identify the correct stimulus from a set of novel stimuli - decoding model\n",
        "#### [pattern recognition](https://en.wikipedia.org/wiki/Pattern_recognition)\n",
        "#### [Kay et al., 2008 Nature](https://www.nature.com/articles/nature06713)\n",
        "#### [Michel et al., 2008 meanings of nouns](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.649.5692&rep=rep1&type=pdf)\n",
        "1. all decoding models performed significantly better than the chance level of 5e-4% (p < 2d-308, binomial test across test images within subjects)\n",
        "2. the striate decoding model correctly identified a stimulus from a set of 1870 potential stimuli at 96%-500 voxels and 79%-250 voxels accuracy, whereas the extrastriate decoding model correctly identified a stimulus from the same set at 95% and 63% accuracy\n",
        "3. the ventral stream decoding model showed higher identification accuracy than either of the previous 2 decoding model (striate and extrastriate): 98 % and 93%\n"
      ]
    },
    {
      "metadata": {
        "id": "qisD9cvREjfo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Image decoding is driven by discriminative and categorical information\n",
        "1. Question: to what extent decoding performance is driven by {disrimination}[identifying an image based on its unique characteristics]() or {categorization}[identifying an image based on categorical information]()\n",
        "2. assigned each image in the *test set* to one of two categories (**animate vs inanimate**) as this apears to be the strongest categorical division in [*inferior temporal cortex*](https://upload.wikimedia.org/wikipedia/commons/1/18/Gray726_inferior_temporal_gyrus.png) (Khaligh-Razavi and Kriegeskorte, 2014 RSA paper)\n",
        "3. 99 of 120 *test images* could be assigned to either of these categories and were used for further analysis\n",
        "4. compute the pariwise linear correlations between {the observed} and {predicted responses} to each pair of images (how do you pair images?)\n",
        "5. it was found that  $$\\mathbf{r}_{Image\\,i}(\\text{observed responses},\\text{predicted responses})\\, >>>\\, mean(\\mathbf{r}(\\text{observed responses}_{Image\\,i},\\text{predicted responses}_{Image\\,not\\,i}))$$\n",
        "6. **this points towards idenfication based on each image's unique characteristic** (in other words, the abstract representations of the images were not overlaped too much)\n",
        "7. For high-level voxels only: $$mean(\\mathbf{r}(\\text{observed responses}_{(Image\\,i,\\,Category\\,j)},\\text{predicted responses}_{(Image\\,k,\\,Category\\,j)})) >>> mean (\\mathbf{r}(\\text{observed responses}_{(Image\\,i,\\,Category\\,j)},\\text{predicted responses}_{(Image\\,k,\\,Category\\,p)}))$$\n",
        "8. for downstream areas, **not only** characteristics of an image, **but also** its semantic content is involved in response prediction"
      ]
    },
    {
      "metadata": {
        "id": "2S7KWidgF_zh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 821
        },
        "outputId": "a774c37f-52d5-4db6-b0ca-95afb1f60274"
      },
      "cell_type": "code",
      "source": [
        "Image(url='http://www.jneurosci.org/content/jneuro/35/27/10005/F3.large.jpg?width=800&height=600&carousel=1',height=800)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"http://www.jneurosci.org/content/jneuro/35/27/10005/F3.large.jpg?width=800&height=600&carousel=1\" height=\"800\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "metadata": {
        "id": "zs5tgPTyGEPn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Properties of the voxel groups systematically change as a function of layer assignment. **A**, Significant linear partial correlations between the predicted responses of each pair of voxel groups. Line widths are proportional to mean partial correlation coefficients across subjects. **B**, Distribution of the receptive field centers for both subjects. [more voxels dedicated to foveal than peripheral vision]() **C**, Example reconstructions of the internal representations of the convolutional layers. Reconstructions are enlarged, and automatic tone, contrast, and color enhancement are applied for visualization purposes. **D**, Proportions of the internal representations of the convolutional layers that are assigned to low-level (blob, contrast, and edge), mid-level (contour, shape, and texture), and high-level (irregular pattern, object part, and entire object) feature classes. **E**, Receptive field complexity (K), invariance, and size of the voxel groups."
      ]
    },
    {
      "metadata": {
        "id": "rzk1yEG7JI-a",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Voxel groups exhibit coherent representational characteristics\n",
        "## don't understand at all\n",
        "1. pooled voxels that were assigned to the same DNN layer together and analyzed their properties (what do you mean by \"pool\")\n",
        "2. the responses of successive voxel groups were more partially correlated than those of nonsuccessive groups\n",
        "3. information flow mainly takes place between neighboring visual areas, providing quantitative evidence for the thesis that the visual ventral stream is *hierarchically* organized (Markov et al., 2014), with downstream areas processing increasing complex features of the retinal input"
      ]
    },
    {
      "metadata": {
        "id": "AXsVAp_JKPL9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 521
        },
        "outputId": "4ba534cc-d60b-43cb-9247-d336416c7b15"
      },
      "cell_type": "code",
      "source": [
        "Image(url='http://www.jneurosci.org/content/jneuro/35/27/10005/F4.large.jpg?width=800&height=600&carousel=1',height=500)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"http://www.jneurosci.org/content/jneuro/35/27/10005/F4.large.jpg?width=800&height=600&carousel=1\" height=\"500\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "metadata": {
        "id": "-uix-lupKZsC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Layer assignments of the voxels systematically increase as a function of position on the occipital cortex. **A**, Layer assignments of the significant voxels across occipital cortex (p < 2e-6 for both subjects, Bonferroni corrected for number of voxels, Student's t test across cross-validated training images within subjects). **B**, Layer assignments of the significant voxels across V1, V2, V4, and LO (p < 5e-8 for both subjects, Bonferroni corrected for number of layers and voxels, Student's t test across cross-validated training images within subjects). **C**, Proportions of voxels in areas V1, V2, V4, and LO that are assigned to low-level (blob, contrast, and edge), mid-level (contour, shape, and texture), and high-level (irregular pattern, object part, and entire object) feature classes."
      ]
    },
    {
      "metadata": {
        "id": "YXYmseUyKNxV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Voxel groups reveal a gradient in the complexity of neural representations\n",
        "1. an increase in layer assignment was observed when moving from posterior to anterior points on the cortical surface"
      ]
    },
    {
      "metadata": {
        "id": "Bgb_LGzH_KfZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 821
        },
        "outputId": "f50b8d3f-18d9-40ab-8db9-ee6c5eb90543"
      },
      "cell_type": "code",
      "source": [
        "Image(url='http://www.jneurosci.org/content/jneuro/35/27/10005/F5.large.jpg?width=800&height=600&carousel=1',height=800)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"http://www.jneurosci.org/content/jneuro/35/27/10005/F5.large.jpg?width=800&height=600&carousel=1\" height=\"800\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "metadata": {
        "id": "XZaeNr5xK9Cm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Voxels in different visual areas are differentially selective to feature maps in different layers. **A**, Selectivity of the significant voxels in the occipital cortex to three distinct feature maps of varying complexity (p < 2e-6 for both subjects, Bonferroni corrected for number of voxels, Student's t test across cross-validated training images within subjects). **B**, Biclusters of hyperaligned voxels and feature maps. Horizontal and vertical red lines delineate the boundaries of clusters of feature maps and voxels, respectively. The rows and columns are thresholded such that each row and column contain at least one element that survives the threshold of $r^2$ = 0.15. The numbers in parentheses denote the number of remaining feature maps and voxels after thresholding."
      ]
    },
    {
      "metadata": {
        "id": "B3mKp14MLFlW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Selectivity of voxels to individual feature maps reveals distributed representations\n",
        "1. individual features accurately predicted multiple voxels and invidual voxels were accurately predicted by multiple features (many-to-many mapping)\n",
        "2. *for features of either low or high complexity this relationship tended to be spatially confined to either upstream or downstream visual areas, respectively*\n",
        "3. biclustering of the prediction accuracy matrix revealed horizontal bands with fluctuating magnitude that point to features with similar information content, and vertical bands that point to clusters of voxels with congruent responses"
      ]
    },
    {
      "metadata": {
        "id": "TWBeLQORK8ef",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 821
        },
        "outputId": "7e95d2e5-cc23-4adc-f84b-99971ede1203"
      },
      "cell_type": "code",
      "source": [
        "Image(url='http://www.jneurosci.org/content/jneuro/35/27/10005/F6.large.jpg?width=800&height=600&carousel=1',height=800)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<img src=\"http://www.jneurosci.org/content/jneuro/35/27/10005/F6.large.jpg?width=800&height=600&carousel=1\" height=\"800\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "metadata": {
        "id": "ImBuVnurPJET",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Our model performs similarly to the control models that are task optimized but outperforms those that are not task optimized across V1, V2, V4, and LO voxels of both subjects. **A**, Comparison between the prediction accuracies for our model ($r_0$) with those for the pretrained DNN ($r_P$), random DNN ($r_R$), and GWP ($r_{GWP}$) models. Red dots denote the individual voxels. Asterisks indicate the visual areas where the prediction accuracies are significantly different. **B**, Comparison between the layer assignments for our model ($DNN_0$) with those of the pretrained DNN ($DNN_P$) and random DNN ($DNN_R$) models. Red dots denote the individual voxels. Crosses indicate the mean layer assignments of the $DNN_0$ model."
      ]
    },
    {
      "metadata": {
        "id": "ylmsm4F0PhEi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Discussion\n",
        "\n",
        "1. semantic selectivity is organized as smooth gradients across cortext (Huth et al., 2012)\n",
        "2. Ventral stream responses to scrambleed versus nonscrambled images (Grill-Spector et al., 1998)\n",
        "3. Downstream receptive fields become larger and more invariant (Smith et al., 2001; DiCarlo and Cox, 2007)\n",
        "4. Some downstream neurons are tuned to relatively simple features and some uptream neurons are tuned to relatively complex features in primates (Desimone et al., 1984; Hedge and Van Essen, 2007)\n",
        "5. Individual features are represented in a distributed manner across a patch of cortex and multiple features are superimposed on the same cortical expanse (Fig 5,6) (Grill-Spector and Weiner, 2014)\n",
        "6. complex, ecologically valid naturalistic stimuli (Felsen and Dan, 2005)\n",
        "7. Highly constraint artificial stimuli (Rust and Movshon, 2005)\n",
        "8. mapping of individual stimulus features confirm that \n",
        "        Low-level stimulus properties were mainly confined to early visual areas\n",
        "        High-level stimulus properties were mostly represented in posterior inferior temporal areas\n",
        "9. biclustering of feature-specific prediction accurarices revealed a more fine-grained functional specialization in downstream visual areas (Larsson and Heeger, 2006; Tanigawas et al., 2010)\n",
        "10. the general applicability of DNN-based encoding models permits the investigation of neural representations in the other visual areas (what does this sentence mean?) (Agrawal et al., 2014)\n",
        "11. in other brain regions involved in the representation of sensory information: dorsal stream (Goodale and Milner, 1992)\n",
        "12.in multimodal association areas (Mesulam, 1998)\n",
        "13. probbing other brain regions:\n",
        "        Top-down: changes in attention (Cukur et al., 2013)\n",
        "        Task demand (Emadi and Esteky, 2014; McKee et al., 2014)\n",
        "        Function of experience (Rainer et al., 2004; Cukur et al., 2013)\n",
        "        Neurodegenerative disorders: semantic dementia (Patterson et al., 2007)\n",
        "14. probbing internally generated percepts that occur during:\n",
        "        Imagery (Thirion et al., 2006)\n",
        "        Memory retrieval (Harrison and Tong, 2009)\n",
        "        Visual illussions (Kok and de Lange, 2014)\n",
        "15. one way to improve **encoding performance** is to develop feature that outperform DNNs when it comes to capturing neural representations of *low-, mid-, and high-* level stimulus features\n",
        "16. **arguably**, *unsupervised learning of statistical struture* in the environment or the maximization of expected reward during *reinforcement learning* offer more biologically plausible explanantions for the formation of receptive field properties:\n",
        "        object categorization (Olshausen and Field, 1996; Schultz et al., 1997)\n",
        "17. **another avenue for further research is the development of more sophisticated responses models**\n",
        "18. the current response models make use of a **linear mapping** from a **nonlinear feature representation** onto **peak BOLD amplitude**, HOWEVER, the mapping from stimulus features to responses that result from changes in neuronal processing (Logothetis and Wandell, 2004; Norris, 2006)\n",
        "19. It is likely that **encoding performance** will further improve by using more sophisticate and/or biophysically realistic response models (Pedregosa et al., 2014; Aquino et al., 2014)\n",
        "\n",
        "# Encoding models as hypothese about brain function\n",
        "1. DNN-based encoding models does **NOT** follow that they provide a mechanistic account for perceptual processing in their biological counterpartes\n",
        "2. the use of a strictly feedforward architecture **cannot** easily be reconciled with the feedback processing inherent to neural information processing (Hochstein and Ahissar, 2002)\n",
        "3. the utility of the encoding approach lies in testing *whether a particular computational model outperforms alternative computational models* **when it comes to explaining observed data** (Bayesian approach)(Naselaris et al., 2011)\n",
        "4. DNN-based encoding model can be considered as implementing a hypothesis about the emergence of receptive field properties across the ventral stream (Fukushima, 1980)\n",
        "5. DNNs rely on the notion of object categorization to explain the emergence of a hierachy of increasingly complex represenations (Serre et al., 2007)\n",
        "6. the proposition that object categorization drives the formation of receptive field properties in the ventral stream is supported by the observation that performance-optimized hierachical models can reliably predict single-neuron responses in area IT of the macaque monkey (Yamin et al., 2014)\n",
        "7. it is also substantiated by recent findings that DNNs better predict voxel responses in the human system and the representational geometry of IT responses in both macaques and humans, compared with other computational models (Cadieu et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014) -- voxel sin downstream areas of the ventral stream code for increasingly complex stimulus features that drive object categorization\n",
        "\n",
        "# The goal of future computational models\n",
        "1. incorporating different assumptions or invoking other objective functions\n",
        "2. reflecting alternative theories of brain functions\n",
        "3. already at the earliest levels of visual processing, there remains ample room for debate as to what form an optimal computational model should take\n",
        "4. Notwithstanding the debate that remains, this study substribe to a model-based approach to cognitive neuroscience in which theories about brain function are tested against each other by validating *generative models* on neural and/or behavioral data"
      ]
    },
    {
      "metadata": {
        "id": "jdAw0Yy9PIdA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}