index.html

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="description"
        content="Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning">
  <meta name="keywords" content="Multimodal LLM, Face Analysis, Reasoning">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning</title>

  <!-- Global site tag (gtag.js) - Google Analytics -->
  <script async src="https://www.googletagmanager.com/gtag/js?id=G-PYVRSFMDRL"></script>
  <script>
    window.dataLayer = window.dataLayer || [];

    function gtag() {
      dataLayer.push(arguments);
    }

    gtag('js', new Date());

    gtag('config', 'G-PYVRSFMDRL');
  </script>

  <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
        rel="stylesheet">

  <link rel="stylesheet" href="./static/css/bulma.min.css">
  <link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
  <link rel="stylesheet" href="./static/css/bulma-slider.min.css">
  <link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
  <link rel="stylesheet"
        href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
  <link rel="stylesheet" href="./static/css/index.css">

  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
  <script defer src="./static/js/fontawesome.all.min.js"></script>
  <script src="./static/js/bulma-carousel.min.js"></script>
  <script src="./static/js/bulma-slider.min.js"></script>
  <script src="./static/js/index.js"></script>
</head>
<body>

<nav class="navbar" role="navigation" aria-label="main navigation">
  <div class="navbar-brand">
    <a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false">
      <span aria-hidden="true"></span>
      <span aria-hidden="true"></span>
      <span aria-hidden="true"></span>
    </a>
  </div>
</nav>


<section class="hero">
  <div class="hero-body">
    <div class="container is-max-desktop">
      <div class="columns is-centered">
        <div class="column has-text-centered">
          <h1 class="title is-1 publication-title">Face-LLaVA: Facial Expression and <br>Attribute Understanding through Instruction Tuning</h1>

          <div class="is-size-5 publication-authors">
            <span class="author-block">
              <a href="https://ashutoshchaubey.in/">Ashutosh Chaubey,</a></span> 
            <span class="author-block">
              <a href="">Xulang Gu,</a></span>
            <span class="author-block">
              <a href="https://www.ihp-lab.org/">Mohammad Soleymani</a>
            </span>
          </div>

          <div class="is-size-5 publication-authors">
            <span class="author-block">University of Southern California</span>
          </div>

          <div class="column has-text-centered">
            <div class="publication-links">
              <!-- PDF Link. -->
              <span class="link-block">
                <a href=""
                   class="external-link button is-normal is-rounded is-dark">
                  <span class="icon">
                      <i class="ai ai-arxiv"></i>
                  </span>
                  <span>arXiv</span>
                </a>
              </span>
              <!-- Code Link. -->
              <span class="link-block">
                <a href=""
                   class="external-link button is-normal is-rounded is-dark">
                  <span class="icon">
                      <i class="fab fa-github"></i>
                  </span>
                  <span>Code (Coming Soon!)</span>
                  </a>
              </span>
              <span class="link-block">
                <a href=""
                   class="external-link button is-normal is-rounded is-dark">
                  <span class="icon">
                      <i class="fab fa-github"></i>
                  </span>
                  <span>Dataset (Coming Soon!)</span>
                  </a>
              </span>
            </div>

          </div>
        </div>
      </div>
    </div>
  </div>
</section>

<section class="hero teaser">
  <div class="container is-max-desktop">
    <div class="hero-body">
        <img src="static/images/face_llava_intro_figure.png" alt="Face-LLaVA Chat Example">
        <p>
          <br />
          We propose Face-LLaVA, a multimodal large language model specialized in face-centered tasks like facial expression and attribute recognition, and capable of generating natural language descriptions for reasoning. We introduce FaceInstruct-1M, a dataset tailored for face-focused instruction tuning, and develop a novel face-specific visual encoder utilizing Face-Region Guided Cross-Attention.
        </p>
    </div>
  </div>
</section>

<section class="section">
  <div class="container is-max-desktop">
    <!-- Abstract. -->
    <div class="columns is-centered has-text-centered">
      <div class="column is-four-fifths">
        <h2 class="title is-3">Abstract</h2>
        <div class="content has-text-justified">
          <p>
            The human face plays a central role in social communication, necessitating the use of performant computer vision tools for human-centered applications. We propose Face-LLaVA, a multimodal large language model for face-centered, in-context learning, including facial expression and attribute recognition. Additionally, Face-LLaVA is able to generate natural language descriptions that can be used for reasoning. Leveraging existing visual databases, we first developed FaceInstruct-1M, a face-centered database for instruction tuning MLLMs for face processing. We then developed a novel face-specific visual encoder powered by Face-Region Guided Cross-Attention that integrates face geometry with local visual features. We evaluated the proposed method across nine different datasets and five different face processing tasks, including facial expression recognition, action unit detection, facial attribute detection, age estimation and deepfake detection. Face-LLaVA achieves superior results compared to existing open-source MLLMs and competitive performance compared to commercial solutions. Our model output also receives a higher reasoning rating by GPT under a zero-shot setting across all the tasks.
          </p>
        </div>
      </div>
    </div>
    <!--/ Abstract. -->
    <!-- Paper video. -->
    <!-- <div class="columns is-centered has-text-centered">
      <div class="column is-four-fifths">
        <h2 class="title is-3">Video</h2>
        <div class="publication-video">
          <iframe src="./static/images/demo/main_4.mp4"
                  frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
        </div>
      </div>
    </div> -->
    <!--/ Paper video. -->
  </div>
</section>

<section class="section">
    <div class="container is-max-desktop">
      <!-- Method. -->
      <div class="columns is-centered">
        <div class="column is-full-width">
          <h2 class="title is-3">FaceInstruct-1M</h2>
  
          <!-- Interpolating. -->
          <img src="./static/images/face_llava_data_annotation_pipeline.png" class="" alt="FaceInstruct-1M Data Annotation Pipeline"/>
          <p>
            <br />
            Overview of the data annotation pipeline used to construct FaceInstruct-1M dataset. We leverage existing face-analysis datasets and their annotations and prompt Gemini-1.5 flash to convert the annotations into natural language with reasoning. To ensure data quality, we rate the generated samples from GPT-4o-mini. 
          </p>
        <!--/ Interpolating. -->
        </div>
      </div>
      <!--/ Compare. -->
    </div>
  </section>

<section class="hero is-light is-small">
<div class="hero-body">
    <div class="container">
    <h2 class="title is-3 is-centered">Data Samples</h2>
    <div id="results-carousel" class="carousel results-carousel">
        <div class="item item-dfew">
            <img id="dfew" src="./static/images/data_samples/data_samples-dfew.png" alt="Samples Face Expression Recognition"/>
        </div>
        <div class="item item-au">
            <img id="au" src="./static/images/data_samples/data_samples-au.png" alt="Samples Action Unit Estimation"/>
        </div>
        <div class="item item-ff">
            <img  id="ff" src="./static/images/data_samples/data_samples-face_forensics.png" alt="Samples Deep Fake Detection"/>
        </div>
        <div class="item item-age">
            <img  id="age" src="./static/images/data_samples/data_samples-age.png" alt="Samples Age Estimation"/>
        </div>
    </div>
    </div>
</div>
</section>


<section class="section">
  <div class="container is-max-desktop">
    <!-- Method. -->
    <div class="columns is-centered">
      <div class="column is-full-width">
        <h2 class="title is-3">Face-LLaVA</h2>

        <!-- Interpolating. -->
        <img src="./static/images/face_llava_main_figure.png" class="interpolation-image" alt="Face-LLaVA Architecture"/>
        <p>
          <br />
          Overview of the proposed Face-LLaVA Architecture. We use a pretrained <i>face expert</i> model, i.e. a landmark detection model to extract landmarks from the visual input. We leverage our novel <i>face-region landmark projector (FRLP)</i> module to obtain landmark tokens which capture the face landmark information. To enrich the visual tokens, <i>face-region guided cross attention</i> is performed between visual tokens and landmark tokens guided by the <i>face-region patch proximity mask</i>. 
        </p>
      <!--/ Interpolating. -->
      </div>
    </div>
    <!--/ Compare. -->
  </div>
</section>


<section class="section">
    <div class="container is-max-desktop">
      <div class="columns is-centered">
        <div class="column is-full-width">
          <h2 class="title is-3">Results</h2>
  
          <div class="image-container">
            <img src="./static/images/results_fer.png" class="interpolation-image" alt="Face-LLaVA Comparison with baselines on Face Expression Recognition"/>
          </div>
  
          <div class="image-container">
            <img src="./static/images/results_age_attr.png" class="interpolation-image" alt="Face-LLaVA Comparison with baselines on Age, Attribute and Deepfake detection"/>
          </div>
  
          <div class="image-container">
            <img src="./static/images/results_reasoning.png" class="interpolation-image" alt="Face-LLaVA Comparison with baselines for reasoning abilities rated using GPT-4o"/>
          </div>
        </div>
      </div>
    </div>
  </section>
  
  <section class="hero is-light is-small">
    <div class="hero-body">
        <div class="container">
        <h2 class="title is-3 is-centered">Comparison with baselines</h2>
        <div id="results-carousel" class="carousel results-carousel">
            
            <div class="item item-dfew-6526">
                <img id="dfew-6526" src="./static/images/baseline_comparison/baseline_comparison_dfew-dfew-6526.png"/>
            </div>
            <div class="item item-dfew-6977">
                <img id="dfew-6977" src="./static/images/baseline_comparison/baseline_comparison_dfew-dfew-6977.png"/>
            </div>
            <div class="item item-dfew-7520">
                <img id="dfew-7520" src="./static/images/baseline_comparison/baseline_comparison_dfew-dfew-7520.png"/>
            </div>
            <div class="item item-dfew-9785">
                <img id="dfew-9785" src="./static/images/baseline_comparison/baseline_comparison_dfew-dfew-9785.png"/>
            </div>
        </div>
        </div>
    </div>
    </section>


<!-- <section class="section" id="BibTeX">
  <div class="container is-max-desktop content">
    <h2 class="title">BibTeX</h2>
    <pre><code>@misc{chang2024magicpose,
      title={MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion}, 
      author={Di Chang and Yichun Shi and Quankai Gao and Jessica Fu and Hongyi Xu and Guoxian Song and Qing Yan and Yizhe Zhu and Xiao Yang and Mohammad Soleymani},
      year={2024},
      eprint={2311.12052},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}</code></pre>
  </div>
</section> -->


<section class="section" id="Acknowledgement">
  <div class="container is-max-desktop content">
    <h2 class="title">Acknowledgement</h2>
    <p>
        Research was sponsored by the Army Research Office and was accomplished under Cooperative Agreement Number W911NF-25-2-0040. 
        The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. 
        The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein
    </p>
  </div>
</section>
<footer class="footer">
  <div class="container">
    <div class="columns is-centered">
      <div class="column is-8">
        <div class="content">
          <p>
            Website adapted from <a
              href="https://github.com/nerfies/nerfies.github.io">Nerfies</a>
          </p>
        </div>
      </div>
    </div>
  </div>
</footer>

</body>
</html>