notebooks/Linear Algebra.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Algebra"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2018-2024 by [Damir Cavar](http://damir.cavar.me/)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Version:** 1.4, January 2024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Download:** This and various other Jupyter notebooks are available from my [GitHub repo](https://github.com/dcavar/python-tutorial-notebooks)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**License:** [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/) ([CA BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Prerequisites:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a tutorial related to the L665 course on Machine Learning for NLP focusing on Deep Learning, Spring 2018, and Advanced Natural Language Processing L645 in Fall 2023 at Indiana University."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following material is based on *Linear Algebra Review and Reference* by Zico Kolter (updated by Chuong Do) from September 30, 2015. This means, many passages are literally copied, many are rewritten. I do not mark sections that are added or different. Consider this notebook a extended annotation of Kolter's (and Do's) notes. See also James E. Gentle (2017) [Matrix Algebra: Theory, Computations and Applications in Statistics](http://www.springer.com/us/book/9780387708720). Second edition. Springer. Another good resource is Philip N. Klein (2013) [Coding the Matrix: Linear Algebra through Applications to Computer Science](http://codingthematrix.com/), Newtonian Press."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Concepts and Notation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following system of equations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{equation}\n",
    "\\begin{split}\n",
    "4 x_1 - 5 x_2 & = -13 \\\\\n",
    " -2x_1 + 3 x_2 & = 9\n",
    "\\end{split}\n",
    "\\end{equation}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are looking for a unique solution for the two variables $x_1$ and $x_2$.  The system can be described as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$Ax = b$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as matrices:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\begin{bmatrix}\n",
    "       4  & -5 \\\\[0.3em]\n",
    "       -2 &  3 \n",
    "     \\end{bmatrix},\\ \n",
    " b = \\begin{bmatrix}\n",
    "       -13 \\\\[0.3em]\n",
    "       9 \n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **scalar** is an element in a vector, containing a real number **value**. In a vector space model or a vector mapping of (symbolic, qualitative, or quantitative) properties the scalar holds the concrete value or property of a variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **vector** is an array, tuple, or ordered list of scalars (or elements) of size $n$, with $n$ a positive integer. The **length** of the vector, that is the number of scalars in the vector, is also called the **order** of the vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Vectorization** is the process of creating a vector from some data using some process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vectors of the length $n$ could be treated like points in $n$-dimensional space. One can calculate the distance between such points using measures like [Euclidean Distance](https://en.wikipedia.org/wiki/Euclidean_distance). The similarity of vectors could also be calculated using [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **matrix** is a list of vectors that all are of the same length. $A$ is a matrix with $m$ rows and $n$ columns, entries of $A$ are real numbers:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A \\in \\mathbb{R}^{m \\times n}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A vector $x$ with $n$ entries of real numbers, could also be thought of as a matrix with $n$ rows and $1$ column, or as known as a **column vector**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$x = \\begin{bmatrix}\n",
    "       x_1 \\\\[0.3em]\n",
    "       x_2 \\\\[0.3em]\n",
    "       \\vdots \\\\[0.3em]\n",
    "       x_n\n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Representing a **row vector**, that is a matrix with $1$ row and $n$ columns, we write $x^T$ (this denotes the transpose of $x$, see above)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$x^T = \\begin{bmatrix}\n",
    "       x_1 & x_2 & \\cdots & x_n\n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the notation $a_{ij}$ (or $A_{ij}$, $A_{i,j}$, etc.) to denote the entry of $A$ in the $i$th row and\n",
    "$j$th column:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\begin{bmatrix}\n",
    "       a_{11} & a_{12} & \\cdots & a_{1n} \\\\[0.3em]\n",
    "       a_{21} & a_{22} & \\cdots & a_{2n} \\\\[0.3em]\n",
    "       \\vdots & \\vdots & \\ddots & \\vdots \\\\[0.3em]\n",
    "       a_{m1} & a_{m2} & \\cdots & a_{mn} \n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We denote the $j$th column of $A$ by $a_j$ or $A_{:,j}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\begin{bmatrix}\n",
    "       \\big| & \\big| &  & \\big| \\\\[0.3em]\n",
    "       a_{1} & a_{2} & \\cdots & a_{n} \\\\[0.3em]\n",
    "       \\big| & \\big| &  & \\big|  \n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We denote the $i$th row of $A$ by $a_i^T$ or $A_{i,:}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\begin{bmatrix}\n",
    "      -- & a_1^T  & -- \\\\[0.3em]\n",
    "       -- & a_2^T  & -- \\\\[0.3em]\n",
    "          & \\vdots &  \\\\[0.3em]\n",
    "       -- & a_m^T  & -- \n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A $n \\times m$ matrix is a two-dimensional array with $n$ rows and $m$ columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix Multiplication"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of the multiplication of two matrixes $A \\in \\mathbb{R}^{m \\times n}$ and $B \\in \\mathbb{R}^{n \\times p}$ is the matrix:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$C = AB \\in \\mathbb{R}^{m \\times n}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is, we are multiplying the columns of $A$ with the rows of $B$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$C_{ij}=\\sum_{k=1}^n{A_{ij}B_{kj}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of columns in $A$ must be equal to the number of rows in $B$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector-Vector Products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inner or Dot Product of Two Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For two vectors $x, y \\in \\mathbb{R}^n$, the **inner product** or **dot product** $x^T y$ is a real number:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$x^T y \\in \\mathbb{R} = \\begin{bmatrix}\n",
    "       x_1 & x_2 & \\cdots & x_n\n",
    "     \\end{bmatrix} \\begin{bmatrix}\n",
    "       y_1 \\\\[0.3em]\n",
    "       y_2 \\\\[0.3em]\n",
    "       \\vdots \\\\[0.3em]\n",
    "       y_n\n",
    "     \\end{bmatrix} = \\sum_{i=1}^{n}{x_i y_i}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **inner products** are a special case of matrix multiplication."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is always the case that $x^T y = y^T x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate the inner product of two vectors $x = [1 2 3 4]$ and $y = [5 6 7 8]$, we can loop through the vector and multiply and sum the scalars (this is simplified code):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "70\n"
     ]
    }
   ],
   "source": [
    "x = (1, 2, 3, 4)\n",
    "y = (5, 6, 7, 8)\n",
    "n = len(x)\n",
    "if n == len(y):\n",
    "    result = 0\n",
    "    for i in range(n):\n",
    "        result += x[i] * y[i]\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is clear that in the code above we could change line 7 to `result += y[i] * x[i]` without affecting the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum([ a*b for a, b in zip(x, y) ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the *numpy* module to apply the same operation, to calculate the **inner product**. We import the *numpy* module and assign it a name *np* for the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the vectors $x$ and $y$ using *numpy*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [1 2 3 4]\n",
      "y: [5 6 7 8]\n"
     ]
    }
   ],
   "source": [
    "x = np.array([1, 2, 3, 4])\n",
    "y = np.array([5, 6, 7, 8])\n",
    "print(\"x:\", x)\n",
    "print(\"y:\", y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now calculate the $dot$ or $inner product$ using the *dot* function of *numpy*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The order of the arguments is irrelevant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(y, x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that both vectors are actually **row vectors** in the above code. We can transpose them to column vectors by using the *shape* property:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [1 2 3 4]\n",
      "xT: [[1]\n",
      " [2]\n",
      " [3]\n",
      " [4]]\n",
      "y: [5 6 7 8]\n",
      "yT: [[5]\n",
      " [6]\n",
      " [7]\n",
      " [8]]\n"
     ]
    }
   ],
   "source": [
    "print(\"x:\", x)\n",
    "x.shape = (4, 1)\n",
    "print(\"xT:\", x)\n",
    "print(\"y:\", y)\n",
    "y.shape = (4, 1)\n",
    "print(\"yT:\", y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, in our understanding of Linear Algebra, we take the arrays above to represent **row vectors**. *Numpy* treates them differently."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see the issues when we try to transform the array objects. Usually, we can transform a row vector into a column vector in *numpy* by using the *T* method on vector or matrix objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [1 2 3 4]\n",
      "y: [5 6 7 8]\n",
      "xT: [1 2 3 4]\n",
      "yT: [5 6 7 8]\n"
     ]
    }
   ],
   "source": [
    "x = np.array([1, 2, 3, 4])\n",
    "y = np.array([5, 6, 7, 8])\n",
    "print(\"x:\", x)\n",
    "print(\"y:\", y)\n",
    "print(\"xT:\", x.T)\n",
    "print(\"yT:\", y.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem here is that this does not do, what we expect it to do. It only works, if we declare the variables not to be arrays of numbers, but in fact a matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [[1 2 3 4]]\n",
      "y: [[5 6 7 8]]\n",
      "xT: [[1]\n",
      " [2]\n",
      " [3]\n",
      " [4]]\n",
      "yT: [[5]\n",
      " [6]\n",
      " [7]\n",
      " [8]]\n"
     ]
    }
   ],
   "source": [
    "x = np.array([[1, 2, 3, 4]])\n",
    "y = np.array([[5, 6, 7, 8]])\n",
    "print(\"x:\", x)\n",
    "print(\"y:\", y)\n",
    "print(\"xT:\", x.T)\n",
    "print(\"yT:\", y.T)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the *numpy* functions *dot* and *outer* are not affected by this distinction. We can compute the dot product using the mathematical equation above in *numpy* using the new $x$ and $y$ row vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [[1 2 3 4]]\n",
      "y: [[5]\n",
      " [6]\n",
      " [7]\n",
      " [8]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[70]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"x:\", x)\n",
    "print(\"y:\", y.T)\n",
    "np.dot(x, y.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or by reverting to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [[1]\n",
      " [2]\n",
      " [3]\n",
      " [4]]\n",
      "y: [[5 6 7 8]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[70]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"x:\", x.T)\n",
    "print(\"y:\", y)\n",
    "np.dot(y, x.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read the result from this array of arrays, we would need to access the value this way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(y, x.T)[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Outer Product of Two Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For two vectors $x \\in \\mathbb{R}^m$ and $y \\in \\mathbb{R}^n$, where $n$ and $m$ do not have to be equal, the **outer product** of $x$ and $y$ is:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$xy^T \\in \\mathbb{R}^{m\\times n}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **outer product** results in a matrix with $m$ rows and $n$ columns by $(xy^T)_{ij} = x_i y_j$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$xy^T \\in \\mathbb{R}^{m\\times n} = \\begin{bmatrix}\n",
    "       x_1 \\\\[0.3em]\n",
    "       x_2 \\\\[0.3em]\n",
    "       \\vdots \\\\[0.3em]\n",
    "       x_n\n",
    "     \\end{bmatrix} \\begin{bmatrix}\n",
    "       y_1 & y_2 & \\cdots & y_n\n",
    "     \\end{bmatrix} = \\begin{bmatrix}\n",
    "       x_1 y_1 & x_1 y_2 & \\cdots & x_1 y_n \\\\[0.3em]\n",
    "       x_2 y_1 & x_2 y_2 & \\cdots & x_2 y_n \\\\[0.3em]\n",
    "       \\vdots  & \\vdots  & \\ddots & \\vdots \\\\[0.3em]\n",
    "       x_m y_1 & x_m y_2 & \\cdots & x_m y_n \\\\[0.3em]\n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some useful property of the outer product: assume $\\mathbf{1} \\in \\mathbb{R}^n$ is an $n$-dimensional vector of scalars with the value $1$. Given a matrix $A \\in \\mathbb{R}^{m\\times n}$ with all columns equal to some vector $x \\in \\mathbb{R}^m$, using the outer product $A$ can be represented as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\begin{bmatrix}\n",
    "       \\big| & \\big| &  & \\big| \\\\[0.3em]\n",
    "       x & x & \\cdots & x \\\\[0.3em]\n",
    "       \\big| & \\big| &  & \\big|  \n",
    "     \\end{bmatrix} = \\begin{bmatrix}\n",
    "       x_1    & x_1    & \\cdots & x_1    \\\\[0.3em]\n",
    "       x_2    & x_2    & \\cdots & x_2    \\\\[0.3em]\n",
    "       \\vdots & \\vdots & \\ddots & \\vdots \\\\[0.3em]\n",
    "       x_m    &x_m     & \\cdots & x_m\n",
    "     \\end{bmatrix} = \\begin{bmatrix}\n",
    "       x_1 \\\\[0.3em]\n",
    "       x_2 \\\\[0.3em]\n",
    "       \\vdots \\\\[0.3em]\n",
    "       x_m\n",
    "     \\end{bmatrix} \\begin{bmatrix}\n",
    "       1 & 1 & \\cdots & 1\n",
    "     \\end{bmatrix} = x \\mathbf{1}^T$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to compute the outer product of two vectors $x$ and $y$, we need to transpose the row vector $x$ to a column vector $x^T$. This can be achieved by the *reshape* function in *numpy*, the *T* method, or the *transpose()* function. The *reshape* function takes a parameter that describes the number of colums and rows for the resulting transposing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [[1 2 3 4]]\n",
      "xT: [[1]\n",
      " [2]\n",
      " [3]\n",
      " [4]]\n",
      "xT: [[1]\n",
      " [2]\n",
      " [3]\n",
      " [4]]\n",
      "xT: [[1]\n",
      " [2]\n",
      " [3]\n",
      " [4]]\n"
     ]
    }
   ],
   "source": [
    "x = np.array([[1, 2, 3, 4]])\n",
    "print(\"x:\", x)\n",
    "print(\"xT:\", np.reshape(x, (4, 1)))\n",
    "print(\"xT:\", x.T)\n",
    "print(\"xT:\", x.transpose())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now compute the **outer product** by multiplying the column vector $x$ with the row vector $y$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5,  6,  7],\n",
       "       [10, 12, 14],\n",
       "       [15, 18, 21],\n",
       "       [20, 24, 28]])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([[1, 2, 3, 4]])\n",
    "# y = np.array([[5, 6, 7, 8]])\n",
    "y = np.array([[5, 6, 7]])\n",
    "x.T * y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Numpy* provides an *outer* function that does all that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5,  6,  7],\n",
       "       [10, 12, 14],\n",
       "       [15, 18, 21],\n",
       "       [20, 24, 28]])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.outer(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, in this simple case using the simple arrays for the data structures of the vectors does not affect the result of the *outer* function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([1, 2, 3, 4])\n",
    "y = np.array([5, 6, 7, 8])\n",
    "np.outer(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matrix-Vector Products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assume a matrix $A \\in \\mathbb{R}^{m\\times n}$ and a vector $x \\in \\mathbb{R}^n$ the product results in a vector $y = Ax \\in \\mathbb{R}^m$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$Ax$ could be expressed as the dot product of row $i$ of matrix $A$ with the column value $j$ of vector $x$. Let us first consider matrix multiplication with a scalar:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\begin{bmatrix}\n",
    "       1 & 2 \\\\[0.3em]\n",
    "       3 & 4\n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute the product of $A$ with a scalar $n = 2$ as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\begin{bmatrix}\n",
    "       1 * n & 2 * n \\\\[0.3em]\n",
    "       3 * n & 4 * n\n",
    "     \\end{bmatrix} = \\begin{bmatrix}\n",
    "       1 * 2 & 2 * 2 \\\\[0.3em]\n",
    "       3 * 2 & 4 * 2\n",
    "     \\end{bmatrix} = \\begin{bmatrix}\n",
    "       2 & 4 \\\\[0.3em]\n",
    "       6 & 8\n",
    "     \\end{bmatrix} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using *numpy* this can be achieved by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 8, 10, 12],\n",
       "       [14, 16, 18]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "A = np.array([[4, 5, 6],\n",
    "             [7, 8, 9]])\n",
    "A * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assume that we have a column vector $x$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$x = \\begin{bmatrix}\n",
    "       1 \\\\[0.3em]\n",
    "       2 \\\\[0.3em]\n",
    "       3 \n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be able to multiply this vector with a matrix, the number of columns in the matrix must correspond to the number of rows in the column vector. The matrix $A$ must have $3$ columns, as for example: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\begin{bmatrix}\n",
    "       4 & 5 & 6\\\\[0.3em]\n",
    "       7 & 8 & 9\n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute $Ax$, we multiply row $1$ of the matrix with column $1$ of $x$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{bmatrix}\n",
    "  4 & 5 & 6\n",
    " \\end{bmatrix}\n",
    " \\begin{bmatrix}\n",
    " 1 \\\\[0.3em]\n",
    " 2 \\\\[0.3em]\n",
    " 3 \n",
    "\\end{bmatrix} = 4 * 1 + 5 * 2 + 6 * 3 = 32 $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do the compute the dot product of row $2$ of $A$ and column $1$ of $x$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{bmatrix}\n",
    "  7 & 8 & 9\n",
    " \\end{bmatrix}\n",
    " \\begin{bmatrix}\n",
    " 1 \\\\[0.3em]\n",
    " 2 \\\\[0.3em]\n",
    " 3 \n",
    "\\end{bmatrix} = 7 * 1 + 8 * 2 + 9 * 3 = 50 $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting column vector $Ax$ is:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$Ax = \\begin{bmatrix}\n",
    "       32 \\\\[0.3em]\n",
    "       50 \n",
    "     \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using *numpy* we can compute $Ax$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[4, 5, 6],\n",
    "             [7, 8, 9]])\n",
    "x = np.array([1, 2, 3])\n",
    "A.dot(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can thus describe the product writing $A$ by rows as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$y = Ax = \\begin{bmatrix}\n",
    " -- & a_1^T  & -- \\\\[0.3em]\n",
    " -- & a_2^T  & -- \\\\[0.3em]\n",
    "    & \\vdots &  \\\\[0.3em]\n",
    " -- & a_m^T  & -- \n",
    "\\end{bmatrix} x = \\begin{bmatrix}\n",
    " a_1^T x \\\\[0.3em]\n",
    " a_2^T x \\\\[0.3em]\n",
    " \\vdots \\\\[0.3em]\n",
    " a_m^T x \n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that the $i$th scalar of $y$ is the inner product of the $i$th row of $A$ and $x$, that is $y_i = a_i^T x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we write $A$ in column form, then:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$y = Ax =\n",
    "\\begin{bmatrix}\n",
    " \\big| & \\big| &  & \\big| \\\\[0.3em]\n",
    " a_1 & a_2 & \\cdots & a_n \\\\[0.3em]\n",
    " \\big| & \\big| &  & \\big|  \n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    " x_1 \\\\[0.3em]\n",
    " x_2 \\\\[0.3em]\n",
    " \\vdots \\\\[0.3em]\n",
    " x_n\n",
    "\\end{bmatrix} =\n",
    "\\begin{bmatrix}\n",
    " a_1\n",
    "\\end{bmatrix} x_1 + \n",
    "\\begin{bmatrix}\n",
    " a_2\n",
    "\\end{bmatrix} x_2 + \\dots +\n",
    "\\begin{bmatrix}\n",
    " a_n\n",
    "\\end{bmatrix} x_n\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case $y$ is a **[linear combination](https://en.wikipedia.org/wiki/Linear_combination)** of the *columns* of $A$, the coefficients taken from $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above examples multiply be the right with a column vector. One can multiply on the left by a row vector as well, $y^T = x^T A$ for $A \\in \\mathbb{R}^{m\\times n}$, $x\\in \\mathbb{R}^m$, $y \\in \\mathbb{R}^n$. There are two ways to express $y^T$, with $A$ expressed by its columns, with $i$th scalar of $y^T$ corresponds to the inner product of $x$ and the $i$th column of $A$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$y^T = x^T A = x^t \\begin{bmatrix}\n",
    " \\big| & \\big| &  & \\big| \\\\[0.3em]\n",
    " a_1 & a_2 & \\cdots & a_n \\\\[0.3em]\n",
    " \\big| & \\big| &  & \\big|  \n",
    "\\end{bmatrix} = \n",
    "\\begin{bmatrix}\n",
    " x^T a_1 & x^T a_2 & \\dots & x^T a_n   \n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can express $A$ by rows, where $y^T$ is a linear combination of the rows of $A$ with the scalars from $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{equation}\n",
    "\\begin{split}\n",
    "y^T & = x^T A \\\\\n",
    "    & = \\begin{bmatrix}\n",
    " x_1 & x_2 & \\dots & x_n   \n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    " -- & a_1^T  & -- \\\\[0.3em]\n",
    " -- & a_2^T  & -- \\\\[0.3em]\n",
    "    & \\vdots &  \\\\[0.3em]\n",
    " -- & a_m^T  & -- \n",
    "\\end{bmatrix} \\\\\n",
    "   & = x_1 \\begin{bmatrix}-- & a_1^T  & --\\end{bmatrix} + x_2 \\begin{bmatrix}-- & a_2^T  & --\\end{bmatrix} + \\dots + x_n \\begin{bmatrix}-- & a_n^T  & --\\end{bmatrix}\n",
    "\\end{split}\n",
    "\\end{equation}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matrix-Matrix Products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can view matrix-matrix multiplication $C = AB$ as a set of vector-vector products. The $(i,j)$th entry of $C$ is the inner product of the $i$th row of $A$ and the $j$th column of $B$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$C = AB =\n",
    "\\begin{bmatrix}\n",
    " -- & a_1^T  & -- \\\\[0.3em]\n",
    " -- & a_2^T  & -- \\\\[0.3em]\n",
    "    & \\vdots &  \\\\[0.3em]\n",
    " -- & a_m^T  & -- \n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    " \\big| & \\big| &  & \\big| \\\\[0.3em]\n",
    " b_1 & b_2 & \\cdots & b_p \\\\[0.3em]\n",
    " \\big| & \\big| &  & \\big|  \n",
    "\\end{bmatrix} = \n",
    "\\begin{bmatrix}\n",
    " a_1^T b_1 & a_1^T b_2 & \\cdots & a_1^T b_p \\\\[0.3em]\n",
    " a_2^T b_1 & a_2^T b_2 & \\cdots & a_2^T b_p \\\\[0.3em]\n",
    " \\vdots    & \\vdots    & \\ddots & \\vdots    \\\\[0.3em]\n",
    " a_m^T b_1 & a_m^T b_2 & \\cdots & a_m^T b_p \n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here $A \\in \\mathbb{R}^{m\\times n}$ and $B \\in \\mathbb{R}^{n\\times p}$, $a_i \\in \\mathbb{R}^n$ and $b_j \\in \\mathbb{R}^n$, and $A$ is represented by rows, $B$ by columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we represent $A$ by columns and $B$ by rows, then $AB$ is the sum of the outer products:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$C = AB =\n",
    "\\begin{bmatrix}\n",
    " \\big| & \\big| &  & \\big| \\\\[0.3em]\n",
    " a_1 & a_2 & \\cdots & a_n \\\\[0.3em]\n",
    " \\big| & \\big| &  & \\big|  \n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    " -- & b_1^T  & -- \\\\[0.3em]\n",
    " -- & b_2^T  & -- \\\\[0.3em]\n",
    "    & \\vdots &  \\\\[0.3em]\n",
    " -- & b_n^T  & -- \n",
    "\\end{bmatrix}\n",
    "= \\sum_{i=1}^n a_i b_i^T\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that $AB$ is the sum over all $i$ of the outer product of the $i$th column of $A$ and the $i$th row of $B$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can interpret matrix-matrix operations also as a set of matrix-vector products. Representing $B$ by columns, the columns of $C$ are matrix-vector products between $A$ and the columns of $B$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$C = AB = A\n",
    "\\begin{bmatrix}\n",
    " \\big| & \\big| &  & \\big| \\\\[0.3em]\n",
    " b_1 & b_2 & \\cdots & b_p \\\\[0.3em]\n",
    " \\big| & \\big| &  & \\big|  \n",
    "\\end{bmatrix} = \n",
    "\\begin{bmatrix}\n",
    " \\big| & \\big| &  & \\big| \\\\[0.3em]\n",
    " A b_1 & A b_2 & \\cdots & A b_p \\\\[0.3em]\n",
    " \\big| & \\big| &  & \\big|  \n",
    "\\end{bmatrix}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this interpretation the $i$th column of $C$ is the matrix-vector product with the vector on the right, i.e. $c_i = A b_i$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Representing $A$ by rows, the rows of $C$ are the matrix-vector products between the rows of $A$ and $B$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$C = AB = \\begin{bmatrix}\n",
    " -- & a_1^T  & -- \\\\[0.3em]\n",
    " -- & a_2^T  & -- \\\\[0.3em]\n",
    "    & \\vdots &  \\\\[0.3em]\n",
    " -- & a_m^T  & -- \n",
    "\\end{bmatrix}\n",
    "B = \n",
    "\\begin{bmatrix}\n",
    " -- & a_1^T B & -- \\\\[0.3em]\n",
    " -- & a_2^T B & -- \\\\[0.3em]\n",
    "    & \\vdots &  \\\\[0.3em]\n",
    " -- & a_n^T B & -- \n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The $i$th row of $C$ is the matrix-vector product with the vector on the left, i.e. $c_i^T = a_i^T B$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Notes on Matrix-Matrix Products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Matrix multiplication is associative:** $(AB)C = A(BC)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Matrix multiplication is distributive:** $A(B + C) = AB + AC$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Matrix multiplication is, in general, not commutative;** It can be the case that $AB \\neq BA$. (For example, if $A \\in \\mathbb{R}^{m\\times n}$ and $B \\in \\mathbb{R}^{n\\times q}$, the matrix product $BA$ does not even exist if $m$ and $q$ are not equal!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identity Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **identity matrix** $I \\in \\mathbb{R}^{n\\times n}$ is a square matrix with the value $1$ on the diagonal and $0$ everywhere else:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$I_{ij} = \\left\\{\n",
    "\\begin{array}{lr}\n",
    " 1 & i = j\\\\\n",
    " 0 & i \\neq j\n",
    "\\end{array}\n",
    "\\right.\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For all $A \\in \\mathbb{R}^{m\\times n}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$AI = A = IA$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the equation above multiplication has to be made possible, which means that in the portion $AI = A$ the dimensions of $I$ have to be $n\\times n$, while in $A = IA$ they have to be $m\\times m$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can generate an *identity matrix* in *numpy* using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A: [[ 0  1  2]\n",
      " [ 3  4  5]\n",
      " [ 6  7  8]\n",
      " [ 9 10 11]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "A = np.array([[0, 1, 2],\n",
    "              [3, 4, 5],\n",
    "              [6, 7, 8],\n",
    "              [9, 10, 11]])\n",
    "print(\"A:\", A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can ask for the shape of $A$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, 3)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *shape* property of a matrix contains the $m$ (number of rows) and $n$ (number of columns) properties in a tuple, in that particular order. We can create an identity matrix for the use in $AI$ by using the $n$ value: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0],\n",
       "       [0, 1, 0],\n",
       "       [0, 0, 1]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.identity(A.shape[1], dtype=\"int\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we specify the *dtype* parameter to *identity* as *int*, since the default would return a matrix of *float* values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To generate an identity matrix for the use in $IA$ we would use the $m$ value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.identity(A.shape[0], dtype=\"int\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute the dot product of $A$ and its identity matrix $I$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = A.shape[1]\n",
    "I = np.array(np.identity(n, dtype=\"int\"))\n",
    "np.dot(A, I)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same is true for the other direction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = A.shape[0]\n",
    "I = np.array(np.identity(m, dtype=\"int\"))\n",
    "np.dot(I, A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Diagonal Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the **diagonal matrix** non-diagonal elements are $0$, that is $D = diag(d_1, d_2, \\dots{}, d_n)$, with:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$D_{ij} = \\left\\{\n",
    "\\begin{array}{lr}\n",
    " d_i & i = j\\\\\n",
    " 0 & i \\neq j\n",
    "\\end{array}\n",
    "\\right.\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The identity matrix is a special case of a diagonal matrix: $I = diag(1, 1, \\dots{}, 1)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In *numpy* we can create a *diagonal matrix* from any given matrix using the *diag* function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "A = np.array([[0,   1,  2,  3],\n",
    "              [4,   5,  6,  7],\n",
    "              [8,   9, 10, 11],\n",
    "              [12, 13, 14, 15]])\n",
    "np.diag(A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An optional parameter *k* to the *diag* function allows us to extract the diagonal above the main diagonal with a positive *k*, and below the main diagonal with a negative *k*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.diag(A, k=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.diag(A, k=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transpose of a Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Transposing** a matrix is achieved by *flipping* the rows and columns. For a matrix $A \\in \\mathbb{R}^{m\\times n}$ the transpose $A^T \\in \\mathbb{R}^{n\\times m}$ is the $n\\times m$ matrix given by:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$(A^T)_{ij} = A_{ji}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Properties of transposes:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- $(A^T)^T = A$\n",
    "- $(AB)^T = B^T A^T$\n",
    "- $(A+B)^T = A^T + B^T$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Symmetric Metrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Square metrices $A \\in \\mathbb{R}^{n\\times n}$ are **symmetric**, if $A = A^T$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A$ is **anti-symmetric**, if $A = -A^T$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For any matrix $A \\in \\mathbb{R}^{n\\times n}$, the matrix $A + A^T$ is **symmetric**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For any matrix $A \\in \\mathbb{R}^{n\\times n}$, the matrix $A - A^T$ is **anti-symmetric**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, any square matrix $A \\in \\mathbb{R}^{n\\times n}$ can be represented as a sum of a symmetric matrix and an anti-symmetric matrix:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\frac{1}{2} (A + A^T) + \\frac{1}{2} (A - A^T)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first matrix on the right, i.e. $\\frac{1}{2} (A + A^T)$ is symmetric. The second matrix $\\frac{1}{2} (A - A^T)$ is anti-symmetric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathbb{S}^n$ is the set of all symmetric matrices of size $n$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A \\in \\mathbb{S}^n$ means that $A$ is symmetric and of the size $n\\times n$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Trace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **trace** of a square matrix $A \\in \\mathbb{R}^{n\\times n}$ is $tr(A)$ (or $trA$) is the sum of the diagonal elements in the matrix:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$trA = \\sum_{i=1}^n A_{ii}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Properties of the **trace**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For $A \\in \\mathbb{R}^{n\\times n}$, $\\mathrm{tr}A = \\mathrm{tr}A^T$\n",
    "- For $A,B \\in \\mathbb{R}^{n\\times n}$, $\\mathrm{tr}(A + B) = \\mathrm{tr}A + \\mathrm{tr}B$\n",
    "- For $A \\in \\mathbb{R}^{n\\times n}$, $t \\in \\mathbb{R}$, $\\mathrm{tr}(tA) = t \\mathrm{tr}A$\n",
    "- For $A,B$ such that $AB$ is square, $\\mathrm{tr}AB = \\mathrm{tr}BA$\n",
    "- For $A,B,C$ such that $ABC$ is square, $\\mathrm{tr}ABC = \\mathrm{tr}BCA = \\mathrm{tr}CAB$, and so on for the product of more matrices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See for proofs the paper!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Norms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **norm** of a vector $x$ is $\\| x\\|$, informally the length of a vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: the Euclidean or $\\mathscr{l}_2$ norm:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\|x\\|_2 = \\sqrt{\\sum_{i=1}^n{x_i^2}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: $\\|x\\|_2^2 = x^T x$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **norm** is any function $f : \\mathbb{R}^n \\rightarrow \\mathbb{R}$ that satisfies the following properties:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For all $x \\in \\mathbb{R}^n$, $f(x) \\geq 0$ (non-negativity)\n",
    "- $f(x) = 0$ if and only if $x = 0$ (definiteness)\n",
    "- For all $x \\in \\mathbb{R}^n$, $t \\in \\mathbb{R}$, $f(tx) = |t|\\ f(x)$ (homogeneity)\n",
    "- For all $x, y \\in \\mathbb{R}^n$, $f(x + y) \\leq f(x) + f(y)$ (triangle inequality)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Norm $\\mathscr{l}_1$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\|x\\|_1 = \\sum_{i=1}^n{|x_i|}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Norm $\\mathscr{l}_\\infty$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\|x\\|_\\infty = \\max_i|x_i|$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All these three norms are examples of the $\\mathscr{l}_p$ norms, with $p$ a real number parameter $p \\geq 1$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\|x\\|_p = \\left(\\sum_{i=1}^n{|x_i|^p}\\right)^{\\frac{1}{p}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Frobenius norm* for matrices:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\|A\\|_F = \\sqrt{\\sum_{i=1}^m\\sum_{i=1}^n A_{ij}^2} = \\sqrt{\\mathrm{tr}(A^T A)}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And many more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Independence and Rank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A set of vectors $\\{x_1, x_2, \\dots{}, x_n\\} \\subset \\mathbb{R}^m$ is said to be **(linearly) independent** if no vector can be represented as a linear combination of the remaining vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A set of vectors $\\{x_1, x_2, \\dots{}, x_n\\} \\subset \\mathbb{R}^m$ is said to be **(lineraly) dependent** if one vector from this set can be represented as a linear combination of the remaining vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some scalar values $\\alpha_1, \\dots{}, \\alpha_{n-1} \\in \\mathbb{R}$ the vectors $x_1, \\dots{}, x_n$ are linerly dependent, if:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{equation}\n",
    "x_n = \\sum_{i=1}^{n-1}{\\alpha_i x_i}\n",
    "\\end{equation}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: The following vectors are lineraly dependent, because $x_3 = -2 x_1 + x_2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$x_1 = \\begin{bmatrix}\n",
    " 1 \\\\[0.3em]\n",
    " 2 \\\\[0.3em]\n",
    " 3 \n",
    "\\end{bmatrix}\n",
    "\\quad\n",
    "x_2 = \\begin{bmatrix}\n",
    " 4 \\\\[0.3em]\n",
    " 1 \\\\[0.3em]\n",
    " 5 \n",
    "\\end{bmatrix}\n",
    "\\quad\n",
    "x_3 = \\begin{bmatrix}\n",
    " 2 \\\\[0.3em]\n",
    " -1 \\\\[0.3em]\n",
    " -1 \n",
    "\\end{bmatrix}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Column Rank of a Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **column rank** of a matrix $A \\in \\mathbb{R}^{m\\times n}$ is the size of the largest subset of columns of $A$ that constitute a linear independent set. Informaly this is the number of linearly independent columns of $A$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Row Rank of a Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **row rank** of a matrix $A \\in \\mathbb{R}^{m\\times n}$ is the largest number of rows of $A$ that constitute a lineraly independent set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rank of a Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For any matrix $A \\in \\mathbb{R}^{m\\times n}$, the column rank of $A$ is equal to the row rank of $A$. Both quantities are referred to collectively as the rank of $A$, denoted as $rank(A)$. Here are some basic properties of the rank:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For $A \\in \\mathbb{R}^{m\\times n}$, $rank(A) \\leq \\min(m, n)$. If $rank(A) = \\min(m, n)$, then $A$ is said to be\n",
    "**full rank**.\n",
    "- For $A \\in \\mathbb{R}^{m\\times n}$, $rank(A) = rank(A^T)$\n",
    "- For $A \\in \\mathbb{R}^{m\\times n}$, $B \\in \\mathbb{R}^{n\\times p}$, $rank(AB) \\leq \\min(rank(A), rank(B))$\n",
    "- For $A,B \\in \\mathbb{R}^{m\\times n}$, $rank(A + B) \\leq rank(A) + rank(B)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subtraction and Addition of Metrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assume $A \\in \\mathbb{R}^{m\\times n}$ and $B \\in \\mathbb{R}^{m\\times n}$, that is $A$ and $B$ are of the same size, to add $A$ to $B$, or to subtract $B$ from $A$, we add or subtract corresponding entries:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A + B =\n",
    "\\begin{bmatrix}\n",
    " a_{11} & a_{12} & \\cdots & a_{1n} \\\\[0.3em]\n",
    " a_{21} & a_{22} & \\cdots & a_{2n} \\\\[0.3em]\n",
    " \\vdots & \\vdots & \\ddots & \\vdots \\\\[0.3em]\n",
    " a_{m1} & a_{m2} & \\cdots & a_{mn}\n",
    "\\end{bmatrix} +\n",
    "\\begin{bmatrix}\n",
    " b_{11} & b_{12} & \\cdots & b_{1n} \\\\[0.3em]\n",
    " b_{21} & b_{22} & \\cdots & b_{2n} \\\\[0.3em]\n",
    " \\vdots & \\vdots & \\ddots & \\vdots \\\\[0.3em]\n",
    " b_{m1} & b_{m2} & \\cdots & b_{mn}\n",
    "\\end{bmatrix} =\n",
    "\\begin{bmatrix}\n",
    " a_{11} + b_{11} & a_{12} + b_{12} & \\cdots & a_{1n} + b_{1n} \\\\[0.3em]\n",
    " a_{21} + b_{21} & a_{22} + b_{22} & \\cdots & a_{2n} + b_{2n} \\\\[0.3em]\n",
    " \\vdots & \\vdots & \\ddots & \\vdots \\\\[0.3em]\n",
    " a_{m1} + b_{m1} & a_{m2} + b_{m2} & \\cdots & a_{mn} + b_{mn}\n",
    "\\end{bmatrix}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same is applies to subtraction:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A - B =\n",
    "\\begin{bmatrix}\n",
    " a_{11} & a_{12} & \\cdots & a_{1n} \\\\[0.3em]\n",
    " a_{21} & a_{22} & \\cdots & a_{2n} \\\\[0.3em]\n",
    " \\vdots & \\vdots & \\ddots & \\vdots \\\\[0.3em]\n",
    " a_{m1} & a_{m2} & \\cdots & a_{mn}\n",
    "\\end{bmatrix} -\n",
    "\\begin{bmatrix}\n",
    " b_{11} & b_{12} & \\cdots & b_{1n} \\\\[0.3em]\n",
    " b_{21} & b_{22} & \\cdots & b_{2n} \\\\[0.3em]\n",
    " \\vdots & \\vdots & \\ddots & \\vdots \\\\[0.3em]\n",
    " b_{m1} & b_{m2} & \\cdots & b_{mn}\n",
    "\\end{bmatrix} =\n",
    "\\begin{bmatrix}\n",
    " a_{11} - b_{11} & a_{12} - b_{12} & \\cdots & a_{1n} - b_{1n} \\\\[0.3em]\n",
    " a_{21} - b_{21} & a_{22} - b_{22} & \\cdots & a_{2n} - b_{2n} \\\\[0.3em]\n",
    " \\vdots & \\vdots & \\ddots & \\vdots \\\\[0.3em]\n",
    " a_{m1} - b_{m1} & a_{m2} - b_{m2} & \\cdots & a_{mn} - b_{mn}\n",
    "\\end{bmatrix}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Python using *numpy* this can be achieved using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "print(\"np.arange(9):\", np.arange(9))\n",
    "print(\"np.arange(9, 18):\", np.arange(9, 18))\n",
    "A = np.arange(9, 18).reshape((3, 3))\n",
    "B = np.arange(9).reshape((3, 3))\n",
    "print(\"A:\", A)\n",
    "print(\"B:\", B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *numpy* function *arange* is similar to the standard Python function *range*. It returns an array with $n$ elements, specified in the one parameter version only. If we provide to parameters to *arange*, it generates an array starting from the value of the first parameter and ending with a value one less than the second parameter. The function *reshape* returns us a matrix with the corresponding number of rows and columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now add and subtract the two matrices $A$ and $B$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A + B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A - B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inverse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **inverse** of a square matrix $A \\in \\mathbb{R}^{n\\times n}$ is $A^{-1}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A^{-1} A = I = A A^{-1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all matrices have inverses. Non-square matrices do not have inverses by definition. For some square matrices $A$ the inverse might not exist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A$ is **invertible** or **non-singular** if $A^{-1}$ exists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A$ is **non-invertible** or **singular** if $A^{-1}$ does not exist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>Note: **non-singular** means the opposite of **non-invertible**!</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For $A$ to have an inverse $A^{-1}$, $A$ must be **full rank**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming that $A,B \\in \\mathbb{R}^{n\\times n}$ are non-singular, then:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- $(A^{-1})^{-1} = A$\n",
    "- $(AB)^{-1} = B^{-1} A^{-1}$\n",
    "- $(A^{-1})^T = (A^T)^{-1}$ (often simply $A^{-T}$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Orthogonal Matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two vectors $x, y \\in \\mathbb{R}^n$ are **orthogonal** if $x^T y = 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A vector $x \\in \\mathbb{R}^n$ is **normalized** if $\\|x\\|^2 = 1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A square matrix $U \\in \\mathbb{R}^{n\\times n}$ is **orthogonal** if all its columns are orthogonal to each other and are **normalized**. The columns are then referred to as being **orthonormal**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It follows immediately from the definition of orthogonality and normality that:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$U^T U = I = U U^T$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that the inverse of an orthogonal matrix is its transpose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If U is not square - i.e., $U \\in \\mathbb{R}^{m\\times n}$, $n < m$ - but its columns are still orthonormal, then $U^T U = I$, but $U U^T \\neq I$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We generally only use the term orthogonal to describe the case, where $U$ is square."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another nice property of orthogonal matrices is that operating on a vector with an orthogonal matrix will not change its Euclidean norm. For any $x \\in \\mathbb{R}^n$, $U \\in \\mathbb{R}^{n\\times n}$ orthogonal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\|U_x\\|^2 = \\|x\\|^2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Range and Nullspace of a Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **span** of a set of vectors $\\{ x_1, x_2, \\dots{}, x_n\\}$ is the set of all vectors that can be expressed as\n",
    "a linear combination of $\\{ x_1, \\dots{}, x_n \\}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathrm{span}(\\{ x_1, \\dots{}, x_n \\}) = \\{ v : v = \\sum_{i=1}^n \\alpha_i x_i, \\alpha_i \\in \\mathbb{R} \\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be shown that if $\\{ x_1, \\dots{}, x_n \\}$ is a set of n linearly independent vectors, where each $x_i \\in \\mathbb{R}^n$, then $\\mathrm{span}(\\{ x_1, \\dots{}, x_n\\}) = \\mathbb{R}^n$. That is, any vector $v \\in \\mathbb{R}^n$ can be written as a linear combination of $x_1$ through $x_n$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The projection of a vector $y \\in \\mathbb{R}^m$ onto the span of $\\{ x_1, \\dots{}, x_n\\}$ (here we assume $x_i \\in \\mathbb{R}^m$) is the vector $v \\in \\mathrm{span}(\\{ x_1, \\dots{}, x_n \\})$, such that $v$ is as close as possible to $y$, as measured by the Euclidean norm $\\|v − y\\|^2$. We denote the projection as $\\mathrm{Proj}(y; \\{ x_1, \\dots{}, x_n \\})$ and can define it formally as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathrm{Proj}( y; \\{ x_1, \\dots{}, x_n \\}) = \\mathrm{argmin}_{v\\in \\mathrm{span}(\\{x_1,\\dots{},x_n\\})}\\|y − v\\|^2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **range** (sometimes also called the columnspace) of a matrix $A \\in \\mathbb{R}^{m\\times n}$, denoted $\\mathcal{R}(A)$, is the the span of the columns of $A$. In other words,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathcal{R}(A) = \\{ v \\in \\mathbb{R}^m : v = A x, x \\in \\mathbb{R}^n\\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making a few technical assumptions (namely that $A$ is full rank and that $n < m$), the projection of a vector $y \\in \\mathbb{R}^m$ onto the range of $A$ is given by:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathrm{Proj}(y; A) = \\mathrm{argmin}_{v\\in \\mathcal{R}(A)}\\|v − y\\|^2 = A(A^T A)^{−1} A^T y$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"red\">See for more details in the notes page 13.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **nullspace** of a matrix $A \\in \\mathbb{R}^{m\\times n}$, denoted $\\mathcal{N}(A)$ is the set of all vectors that equal $0$ when multiplied by $A$, i.e.,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathcal{N}(A) = \\{ x \\in \\mathbb{R}^n : A x = 0 \\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that vectors in $\\mathcal{R}(A)$ are of size $m$, while vectors in the $\\mathcal{N}(A)$ are of size $n$, so vectors in $\\mathcal{R}(A^T)$ and $\\mathcal{N}(A)$ are both in $\\mathbb{R}^n$. In fact, we can say much more. It turns out that:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\{ w : w = u + v, u \\in \\mathcal{R}(A^T), v \\in \\mathcal{N}(A) \\} = \\mathbb{R}^n$ and $\\mathcal{R}(A^T) \\cap \\mathcal{N}(A) = \\{0\\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In other words, $\\mathcal{R}(A^T)$ and $\\mathcal{N}(A)$ are disjoint subsets that together span the entire space of\n",
    "$\\mathbb{R}^n$. Sets of this type are called **orthogonal complements**, and we denote this $\\mathcal{R}(A^T) = \\mathcal{N}(A)^\\perp$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determinant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The determinant of a square matrix $A \\in \\mathbb{R}^{n\\times n}$, is a function $\\mathrm{det} : \\mathbb{R}^{n\\times n} \\rightarrow \\mathbb{R}$, and is denoted $|A|$ or $\\mathrm{det}A$ (like the trace operator, we usually omit parentheses)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A geometric interpretation of the determinant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{bmatrix}\n",
    " -- & a_1^T  & -- \\\\[0.3em]\n",
    " -- & a_2^T  & -- \\\\[0.3em]\n",
    "    & \\vdots &  \\\\[0.3em]\n",
    " -- & a_n^T  & -- \n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "consider the set of points $S \\subset \\mathbb{R}^n$ formed by taking all possible linear combinations of the row vectors $a_1, \\dots{}, a_n \\in \\mathbb{R}^n$ of $A$, where the coefficients of the linear combination are all\n",
    "between $0$ and $1$; that is, the set $S$ is the restriction of $\\mathrm{span}( \\{ a_1, \\dots{}, a_n \\})$ to only those linear combinations whose coefficients $\\alpha_1, \\dots{}, \\alpha_n$ satisfy $0 \\leq \\alpha_i \\leq 1$, $i = 1, \\dots{}, n$. Formally:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$S = \\{v \\in \\mathbb{R}^n : v = \\sum_{i=1}^n \\alpha_i a_i \\mbox{ where } 0 \\leq \\alpha_i \\leq 1, i = 1, \\dots{}, n \\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The absolute value of the determinant of $A$, it turns out, is a measure of the *volume* of the set $S$. The volume here is intuitively for example for $n = 2$ the area of $S$ in the Cartesian plane, or with $n = 3$ it is the common understanding of *volume* for 3-dimensional objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A = \\begin{bmatrix}\n",
    " 1 & 3\\\\[0.3em]\n",
    " 3 & 2 \n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rows of the matrix are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$a_1 = \\begin{bmatrix}\n",
    " 1 \\\\[0.3em]\n",
    " 3 \n",
    "\\end{bmatrix}\n",
    "\\quad\n",
    "a_2 = \\begin{bmatrix}\n",
    " 3 \\\\[0.3em]\n",
    " 2 \n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The set S corresponding to these rows is shown in:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"LinearAlgebra_Determinant_2x2.png\" style=\"max-width:100%; width: 30%; max-width: none\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure above is an illustration of the determinant for the $2\\times 2$ matrix $A$ above. Here, $a_1$ and $a_2$\n",
    "are vectors corresponding to the rows of $A$, and the set $S$ corresponds to the shaded region (i.e., the parallelogram). The absolute value of the determinant, $|\\mathrm{det}A| = 7$, is the area of the parallelogram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For two-dimensional matrices, $S$ generally has the shape of a parallelogram. In our example, the value of the determinant is $|A| = −7$ (as can be computed using the formulas shown later), so the area of the parallelogram is $7$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In three dimensions, the set $S$ corresponds to an object known as a parallelepiped (a three-dimensional box with skewed sides, such that every face has the shape of a parallelogram). The absolute value of the determinant of the $3 \\times 3$ matrix whose rows define $S$ give the three-dimensional volume of the parallelepiped. In even higher dimensions, the set $S$ is an object known as an $n$-dimensional parallelotope."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Algebraically, the determinant satisfies the following three properties (from which all other properties follow, including the general formula):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The determinant of the identity is $1$, $|I| = 1$. (Geometrically, the volume of a unit hypercube is $1$).\n",
    "- Given a matrix $A \\in \\mathbb{R}^{n\\times n}$, if we multiply a single row in $A$ by a scalar $t \\in \\mathbb{R}$, then the determinant of the new matrix is $t|A|$,<br/>\n",
    "$\\left| \\begin{bmatrix}\n",
    " -- & t a_1^T  & -- \\\\[0.3em]\n",
    " -- & a_2^T  & -- \\\\[0.3em]\n",
    "    & \\vdots &  \\\\[0.3em]\n",
    " -- & a_m^T  & -- \n",
    "\\end{bmatrix}\\right|  = t|A|$<br/>\n",
    "(Geometrically, multiplying one of the sides of the set $S$ by a factor $t$ causes the volume\n",
    "to increase by a factor $t$.)\n",
    "- If we exchange any two rows $a^T_i$ and $a^T_j$ of $A$, then the determinant of the new matrix is $−|A|$, for example<br/>\n",
    "$\\left| \\begin{bmatrix}\n",
    " -- & a_2^T  & -- \\\\[0.3em]\n",
    " -- & a_1^T  & -- \\\\[0.3em]\n",
    "    & \\vdots &  \\\\[0.3em]\n",
    " -- & a_m^T  & -- \n",
    "\\end{bmatrix}\\right|  = -|A|$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Several properties that follow from the three properties above include:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For $A \\in \\mathbb{R}^{n\\times n}$, $|A| = |A^T|$\n",
    "- For $A,B \\in \\mathbb{R}^{n\\times n}$, $|AB| = |A||B|$\n",
    "- For $A \\in \\mathbb{R}^{n\\times n}$, $|A| = 0$ if and only if $A$ is singular (i.e., non-invertible). (If $A$ is singular then it does not have full rank, and hence its columns are linearly dependent. In this case, the set $S$ corresponds to a \"flat sheet\" within the $n$-dimensional space and hence has zero volume.)\n",
    "- For $A \\in \\mathbb{R}^{n\\times n}$ and $A$ non-singular, $|A−1| = 1/|A|$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"red\">Continue on page 16.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A [**tensor**](https://en.wikipedia.org/wiki/Tensor) could be thought of as an organized multidimensional array of numerical values. A vector could be assumed to be a sub-class of a tensor. Rows of tensors extend alone the y-axis, columns along the x-axis. The **rank** of a scalar is 0, the rank of a **vector** is 1, the rank of a **matrix** is 2, the rank of a **tensor** is 3 or higher."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperplane"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **hyperplane** is a sub-space in the ambient space with one dimension less. In a two-dimensional space the hyperplane is a line, in a three-dimensional space it is a two-dimensional plane, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyperplanes divide an $n$-dimensional space into sub-spaces that might represent clases in a machine learning algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dot Product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is also the *inner product*. It is a function that returns a number computed from two vectors of the same length by summing up the product of the corresponding dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For two vectors $a = [a_1, a_2, \\dots{}, a_n]$ and $b = [b_1, b_2, \\dots{}, b_n]$ the dot product is:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathbf{a} \\cdot \\mathbf{b} = \\sum_{i=1}^{n} a_{i} b_{i} = a_{1} b_{1} + a_{2} b_{2} + \\cdots + a_{n} b_{n}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we normalize two vectors and compute the dot product, we get the *cosine similarity*, which can be used as a metric for cimilarity of vectors. Independent of the absolute length we look at the angle between the vectors, i.e. the lenght is neutralized via normalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cosine of two non-zero vectors can be derived by using the Euclidean dot product formula (see [Wikipedia: Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathbf{a} \\cdot \\mathbf{b} = \\left\\|\\mathbf{a}\\right\\| \\left\\|\\mathbf{b}\\right\\| \\cos\\theta$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given two vectors of attributes, $A$ and $B$, the cosine similarity, $cos(\\theta)$, is represented using a dot product and magnitude as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\text{similarity} = \\cos(\\theta) = \\frac{\\mathbf{A} \\cdot \\mathbf{B}}{ \\|\\mathbf{A} \\|\\|\\mathbf{B} \\| } = \\frac{\\sum \\limits_{i=1}^{n}{A_{i}B_{i}}}{{\\sqrt {\\sum \\limits _{i=1}^{n}{A_{i}^{2}}}}{\\sqrt {\\sum \\limits _{i=1}^{n}{B_{i}^{2}}}}}$, with $A_i$ and $B_i$ components of vector $A$ and $B$ respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hadamard Product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is also known as the **entrywise product**. For two matrices $A \\in \\mathbb{R}^{m\\times n}$ and $B \\in \\mathbb{R}^{m\\times n}$ the Hadamard product $A\\circ B$ is:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$(A\\circ B)_{i,j} = (A)_{i,j} (B)_{i,j}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\begin{bmatrix}\n",
    " a_{11} & a_{12} & a_{13} \\\\[0.3em]\n",
    " a_{21} & a_{22} & a_{23} \\\\[0.3em]\n",
    " a_{31} & a_{32} & a_{33}\n",
    "\\end{bmatrix} \\circ\n",
    "\\begin{bmatrix}\n",
    " b_{11} & b_{12} & b_{13} \\\\[0.3em]\n",
    " b_{21} & b_{22} & b_{23} \\\\[0.3em]\n",
    " b_{31} & b_{32} & b_{33}\n",
    "\\end{bmatrix} = \n",
    "\\begin{bmatrix}\n",
    " a_{11}b_{11} & a_{12}b_{12} & a_{13}b_{13} \\\\[0.3em]\n",
    " a_{21}b_{21} & a_{22}b_{22} & a_{23}b_{23} \\\\[0.3em]\n",
    " a_{31}b_{31} & a_{32}b_{32} & a_{33}b_{33}\n",
    "\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outer Product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is also called the **tensor product** of two vectors. Compute the resulting matrix by multiplying each element from a column vector with all alements in a row vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2018-2024 by [Damir Cavar](http://damir.cavar.me/)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}