vlm/vlm_input_projector.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "AI多模态大模型发展至今，每年都有非常优秀的工作产出，按照当前模型设计思路，多模态大模型的架构主要包括以下几个部分：\n",
        "\n",
        "- 模态编码器(`Modality Encoder, ME`)：负责将不同模态的输入编码成特征。常见的编码器包括图像的NFNet-F6、ViT、CLIP ViT等，音频的Whisper、CLAP等，视频编码器等。\n",
        "\n",
        "- 输入投影器(`Input Projector`)：负责将其他模态的特征投影到文本特征空间，并与文本特征一起输入给语言模型。常用的投影器包括线性投影器、MLP、交叉注意力，Q-Former，P-Former等。\n",
        "\n",
        "- 语言模型骨架(`LLM Backbone`)：利用预训练的语言模型，负责处理各种模态的特征，进行语义理解、推理和决策。常用的语言模型包括Flan-T5、ChatGLM、UL2等。\n",
        "\n",
        "- 输出投影器(`Output Projector`)：负责将语言模型输出的信号转换成其他模态的特征，以供后续模态生成器使用。常用的投影器包括Tiny Transformer、MLP等。\n",
        "\n",
        "- 模态生成器(`Modality Generator, MG`)：负责生成其他模态的输出。常用的生成器包括图像的Stable Diffusion、视频的Zeroscope、音频的AudioLDM等。\n",
        "\n",
        "本文一手会详细解读AI多模态架构中的输入投影器(Input Projector)，并从线性投影器（`Linear Projector`）、多层感知器（`Multi-Layer Perception, MLP`）和交叉注意力（`Cross-Attention`）三个角度，总结当前主流的工作方案！\n",
        "\n",
        "多模态大模型需要处理不同类型的输入数据，如图像、文本、音频等。为了将这些不同的数据转换到一个共同的表示空间，引入了输入投影器。"
      ],
      "metadata": {
        "id": "QhwyH_Q4pCI9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Linear Projector（线性投影器, LP）\n",
        "线性投影器是一种简单的投影方法，通过线性变换将输入数据映射到目标表示空间。\n",
        "\n",
        "特点:\n",
        "- 简单高效：计算速度快，易于实现。\n",
        "- 参数少：所需参数较少，适合参数敏感的场景。\n",
        "\n",
        "优缺点:\n",
        "- 优点：高效，适合大规模数据处理。\n",
        "- 缺点：表达能力有限，无法捕捉复杂的非线性关系。"
      ],
      "metadata": {
        "id": "YY1uA4yGpx_R"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "class LinearProjector(nn.Module):\n",
        "    def __init__(self, input_dim, output_dim):\n",
        "        super(LinearProjector, self).__init__()\n",
        "        self.linear = nn.Linear(input_dim, output_dim)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.linear(x)\n",
        "\n",
        "# 示例输入\n",
        "image_features = torch.randn(32, 2048)  # 32个样本，每个样本2048维\n",
        "text_features = torch.randn(32, 300)    # 32个样本，每个样本300维\n",
        "\n",
        "# 投影到相同的表示空间\n",
        "projector = LinearProjector(2048, 512)\n",
        "projected_image_features = projector(image_features)\n",
        "\n",
        "projector = LinearProjector(300, 512)\n",
        "projected_text_features = projector(text_features)\n",
        "print(projected_text_features)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tlellHRnpugy",
        "outputId": "0cb9dd55-8744-4c01-9cbc-12b59211a02c"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[-0.3049,  0.0563, -0.2754,  ..., -0.6008,  0.4132, -0.5658],\n",
            "        [ 0.4569,  0.8294,  0.0804,  ..., -1.2296,  0.2409,  0.1005],\n",
            "        [-0.8232, -0.4658,  0.0242,  ..., -0.2850, -0.8126,  0.7600],\n",
            "        ...,\n",
            "        [-0.6319, -0.6294,  0.7677,  ..., -0.5352,  0.0355,  0.7557],\n",
            "        [ 0.7881,  0.2220,  0.0029,  ...,  0.6534,  1.0186, -0.3021],\n",
            "        [-0.6378,  0.3613,  0.7785,  ..., -0.3830, -0.7432,  1.1319]],\n",
            "       grad_fn=<AddmmBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Multi-Layer Perception（多层感知器, MLP）\n",
        "\n",
        "多层感知器是一种神经网络，由多层线性变换和非线性激活函数组成，能够捕捉输入数据的复杂非线性关系。\n",
        "\n",
        "特点：\n",
        "- 非线性：能够表示和捕捉复杂的非线性关系。\n",
        "- 层次结构：通过多层结构逐步提取特征，表示数据更好。\n",
        "\n",
        "优缺点:\n",
        "- 优点：强大的表示能力，能够捕捉复杂的特征和模式。\n",
        "- 缺点：计算复杂度高，训练时间长，容易过拟合。"
      ],
      "metadata": {
        "id": "F2-KZNLsqLvV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "class MLPProjector(nn.Module):\n",
        "    def __init__(self, input_dim, hidden_dim, output_dim):\n",
        "        super(MLPProjector, self).__init__()\n",
        "        self.fc1 = nn.Linear(input_dim, hidden_dim)\n",
        "        self.relu = nn.ReLU()\n",
        "        self.fc2 = nn.Linear(hidden_dim, output_dim)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.fc1(x)\n",
        "        x = self.relu(x)\n",
        "        x = self.fc2(x)\n",
        "        return x\n",
        "\n",
        "# 示例输入\n",
        "image_features = torch.randn(32, 2048)\n",
        "text_features = torch.randn(32, 300)\n",
        "\n",
        "# 投影到相同的表示空间\n",
        "mlp_projector = MLPProjector(2048, 1024, 512)\n",
        "projected_image_features = mlp_projector(image_features)\n",
        "\n",
        "mlp_projector = MLPProjector(300, 512, 512)\n",
        "projected_text_features = mlp_projector(text_features)\n",
        "print(projected_text_features)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YxbcGTHbqAbW",
        "outputId": "21052249-9928-4a14-948a-a037d98882ce"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[ 0.3541,  0.0064,  0.0598,  ..., -0.1948, -0.1849,  0.2483],\n",
            "        [ 0.0916, -0.0562, -0.0709,  ..., -0.3561,  0.0189, -0.5898],\n",
            "        [ 0.3568,  0.1058, -0.3574,  ..., -0.3725, -0.1003, -0.2642],\n",
            "        ...,\n",
            "        [ 0.2145,  0.0248, -0.1390,  ...,  0.0804, -0.3632, -0.2873],\n",
            "        [ 0.3753, -0.0794, -0.1604,  ..., -0.0943,  0.0925, -0.2909],\n",
            "        [ 0.0123, -0.0384,  0.0872,  ..., -0.0048, -0.1921, -0.3131]],\n",
            "       grad_fn=<AddmmBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Cross-Attention（交叉注意力）\n",
        "交叉注意力机制在多模态模型中非常重要，通过计算不同模态间的注意力权重，实现信息的交互和融合。\n",
        "\n",
        "特点：\n",
        "- 信息融合：在不同模态间有效地交换和融合信息。\n",
        "- 权重自适应：根据输入动态计算注意力权重，更加灵活和智能。\n",
        "\n",
        "优缺点：\n",
        "- 优点：在不同模态之间高效地捕捉相关性，适应不同类型的输入。\n",
        "- 缺点：计算复杂度较高，尤其在处理长序列输入时。"
      ],
      "metadata": {
        "id": "Gv4ReCRjqfN1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "class CrossAttention(nn.Module):\n",
        "    def __init__(self, dim, num_heads):\n",
        "        super(CrossAttention, self).__init__()\n",
        "        self.multihead_attn = nn.MultiheadAttention(embed_dim=dim, num_heads=num_heads)\n",
        "\n",
        "    def forward(self, query, key, value):\n",
        "        attn_output, _ = self.multihead_attn(query, key, value)\n",
        "        return attn_output\n",
        "\n",
        "# 示例输入\n",
        "# 10个图像特征序列，每个特征序列32个时间步，每个时间步512维\n",
        "image_features = torch.randn(10, 32, 512)\n",
        "# 20个文本特征序列，每个特征序列32个时间步，每个时间步512维\n",
        "text_features = torch.randn(20, 32, 512)\n",
        "\n",
        "# 使用交叉注意力\n",
        "cross_attention = CrossAttention(dim=512, num_heads=8)\n",
        "# 让图像特征作为query，文本特征作为key和value\n",
        "projected_features = cross_attention(image_features, text_features, text_features)\n",
        "print(projected_features)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OWS0OKAaqbp_",
        "outputId": "6d163f8c-4f0f-4661-f7b3-7761b0161ace"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[[-9.5709e-02, -5.6605e-02, -2.4755e-02,  ..., -4.4275e-02,\n",
            "          -3.2871e-02, -1.2942e-01],\n",
            "         [ 2.8259e-01, -2.2002e-01, -5.4394e-02,  ...,  1.9745e-02,\n",
            "          -1.4952e-01,  5.5191e-02],\n",
            "         [ 1.1994e-01, -3.6203e-02,  1.8334e-04,  ...,  9.5060e-02,\n",
            "          -6.7025e-02, -9.0659e-02],\n",
            "         ...,\n",
            "         [-7.2133e-02,  1.8860e-02, -1.3087e-01,  ...,  5.9735e-02,\n",
            "          -1.6699e-01,  5.0280e-02],\n",
            "         [-4.0124e-02,  5.9835e-02,  1.3879e-01,  ..., -1.2142e-01,\n",
            "          -1.2244e-01,  1.8140e-01],\n",
            "         [ 9.6986e-03,  1.4695e-01,  4.4646e-03,  ...,  1.5083e-02,\n",
            "          -2.2865e-02, -2.5747e-02]],\n",
            "\n",
            "        [[-5.1258e-02, -1.2547e-01,  6.6216e-02,  ...,  5.8145e-02,\n",
            "          -3.2417e-02, -9.4656e-02],\n",
            "         [ 2.9587e-01, -2.0168e-01,  1.5938e-02,  ..., -1.6048e-03,\n",
            "          -4.9942e-02,  9.3270e-02],\n",
            "         [-3.9374e-02, -1.4750e-02,  4.6779e-03,  ...,  9.9899e-02,\n",
            "          -4.0618e-03, -1.7834e-01],\n",
            "         ...,\n",
            "         [-6.0275e-02, -5.6223e-02, -9.4498e-02,  ...,  5.6883e-02,\n",
            "          -4.4757e-02, -6.5040e-02],\n",
            "         [-1.9600e-02,  4.8057e-02,  2.4997e-01,  ..., -1.6837e-01,\n",
            "          -1.8190e-01,  1.4856e-01],\n",
            "         [ 1.4684e-01,  1.0163e-01,  1.2301e-01,  ...,  1.9447e-02,\n",
            "           6.8359e-02,  2.8953e-02]],\n",
            "\n",
            "        [[ 3.6315e-02, -8.0123e-02,  3.3973e-02,  ...,  3.6968e-02,\n",
            "          -8.2285e-02, -6.4388e-02],\n",
            "         [ 2.7148e-01, -1.6537e-01, -1.1889e-01,  ..., -1.9268e-02,\n",
            "          -1.8062e-01,  5.8716e-02],\n",
            "         [ 2.9720e-02,  4.7801e-02, -5.6149e-02,  ...,  8.0374e-02,\n",
            "           6.7107e-02, -8.4169e-02],\n",
            "         ...,\n",
            "         [-2.3593e-02, -6.1750e-02, -3.6898e-02,  ...,  8.3114e-02,\n",
            "          -1.0384e-01,  6.0688e-02],\n",
            "         [ 6.5762e-02, -5.6020e-02,  2.0709e-01,  ..., -2.3038e-01,\n",
            "          -8.3004e-02,  2.2272e-01],\n",
            "         [ 3.2528e-02,  1.3517e-01,  7.9426e-02,  ..., -1.4967e-02,\n",
            "          -4.0313e-02,  2.6481e-02]],\n",
            "\n",
            "        ...,\n",
            "\n",
            "        [[ 7.8409e-02, -1.3518e-01, -1.6283e-02,  ...,  1.0595e-01,\n",
            "          -6.3693e-02, -6.2932e-02],\n",
            "         [ 2.4317e-01, -1.2544e-01, -4.2275e-02,  ...,  1.4431e-02,\n",
            "          -1.5305e-01,  7.6742e-02],\n",
            "         [ 2.8118e-02, -1.0949e-01, -8.5687e-02,  ...,  1.8252e-02,\n",
            "          -1.4201e-01, -1.4092e-01],\n",
            "         ...,\n",
            "         [-4.8062e-02, -3.2223e-02, -8.2972e-02,  ...,  6.9577e-02,\n",
            "          -1.1279e-01, -5.2421e-02],\n",
            "         [ 3.0050e-02,  4.9408e-02,  1.8419e-01,  ..., -1.6648e-01,\n",
            "          -9.8221e-02,  1.2046e-01],\n",
            "         [ 5.2447e-02,  1.1680e-01,  1.1093e-01,  ..., -4.4511e-02,\n",
            "           4.1845e-02, -1.0400e-01]],\n",
            "\n",
            "        [[-2.4127e-02, -1.4828e-01, -1.6746e-02,  ...,  3.6231e-02,\n",
            "          -9.2795e-02, -1.3043e-01],\n",
            "         [ 2.8283e-01, -1.0870e-01, -8.1299e-02,  ..., -2.8193e-04,\n",
            "          -8.0723e-02,  1.1699e-01],\n",
            "         [-1.0361e-02, -4.4766e-02,  3.5174e-02,  ...,  3.4431e-02,\n",
            "          -8.0021e-02, -1.5928e-01],\n",
            "         ...,\n",
            "         [-4.0114e-02, -8.1449e-02, -3.2635e-02,  ...,  1.3043e-01,\n",
            "          -1.0008e-01,  5.8427e-03],\n",
            "         [-3.7111e-02,  2.4196e-02,  1.5197e-01,  ..., -1.2487e-01,\n",
            "          -5.6407e-02,  2.5258e-01],\n",
            "         [ 1.0840e-01,  1.0082e-01,  6.8079e-02,  ..., -7.8874e-03,\n",
            "          -4.6713e-02,  3.1470e-02]],\n",
            "\n",
            "        [[ 7.4959e-02, -9.0071e-02, -5.9312e-02,  ...,  8.4120e-02,\n",
            "          -2.5030e-02, -1.0585e-01],\n",
            "         [ 2.0712e-01, -1.2042e-01, -6.7863e-02,  ...,  4.1073e-02,\n",
            "          -3.5854e-02,  8.7078e-02],\n",
            "         [ 1.9023e-02, -6.7059e-02, -8.0402e-02,  ...,  9.8239e-02,\n",
            "          -7.6377e-02, -1.8217e-01],\n",
            "         ...,\n",
            "         [-4.0095e-02, -2.7215e-02, -5.0186e-02,  ...,  9.7832e-02,\n",
            "          -9.5413e-02,  2.1402e-02],\n",
            "         [-9.7766e-02,  3.5569e-02,  1.4342e-01,  ..., -1.7460e-01,\n",
            "          -1.7544e-01,  1.1999e-01],\n",
            "         [ 9.8000e-02,  1.1453e-01,  2.7894e-02,  ..., -1.8308e-03,\n",
            "          -2.7580e-02,  4.4171e-02]]], grad_fn=<ViewBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 总结\n",
        "\n",
        "1. 线性投影器简单高效，适用于计算资源有限的场景；\n",
        "2. 多层感知器具有强大的表示能力，适用于需要捕捉复杂关系的任务；\n",
        "3. 交叉注意力在多模态信息融合中表现出色，尤其适用于需要跨模态交互的任务。"
      ],
      "metadata": {
        "id": "dEJ9OzXRquyx"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "lqcJLzhyqmZe"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "name": "欢迎使用 Colaboratory",
      "toc_visible": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}