Skip to content

Commit

Permalink
updata html
Browse files Browse the repository at this point in the history
  • Loading branch information
@ggg0919
ggg0919 committed Apr 24, 2024
1 parent 53e06a8 commit f1b21a8
Showing 1 changed file with 0 additions and 226 deletions.
226 changes: 0 additions & 226 deletions index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -164,31 +164,6 @@ <h2 class="title is-1 publication-title">Cantor<img id="painting_icon" width="6%
<span class="author-block"><sup>&dagger;</sup>Corresponding Author</span>
</div>


<!-- <div class="column has-text-centered">
<h3 class="title is-3 publication-title">Improved Baselines with Visual Instruction Fine-tuning</h3>
<div class="is-size-5 publication-authors">
<span class="author-block">
<a href="https://hliu.cc/" style="color:#f68946;font-weight:normal;">Haotian Liu<sup>*</sup></a>,
</span>
<span class="author-block">
<a href="https://chunyuan.li/" style="color:#008AD7;font-weight:normal;">Chunyuan Li<sup>*</sup></a>,
</span>
<span class="author-block">
<a href="https://yuheng-li.github.io" style="color:#008AD7;font-weight:normal;">Yuheng Li</a>,
</span>
<span class="author-block">
<a href="https://pages.cs.wisc.edu/~yongjaelee/" style="color:#f68946;font-weight:normal;">Yong Jae
Lee</a>
</span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block"><b style="color:#f68946; font-weight:normal">&#x25B6 </b> University of
Wisconsin-Madison</b></span>
<span class="author-block"><b style="color:#008AD7; font-weight:normal">&#x25B6 </b> Microsoft Research</span>
</div> -->

<div class="column has-text-centered">
<div class="publication-links">
<span class="link-block">
Expand All @@ -200,15 +175,6 @@ <h3 class="title is-3 publication-title">Improved Baselines with Visual Instruct
<span>arXiv</span>
</a>
</span>
<!-- <span class="link-block">
<a href="https://arxiv.org/abs/2310.03744" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv (LLaVA-1.5)</span>
</a>
</span> -->
<span class="link-block">
<a href="https://github.com/ggg0919/cantor" target="_blank"
class="external-link button is-normal is-rounded is-dark">
Expand All @@ -218,46 +184,7 @@ <h3 class="title is-3 publication-title">Improved Baselines with Visual Instruct
<span>Code</span>
</a>
</span>
<!-- <span class="link-block">
<a href="https://llava.hliu.cc" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="far fa-images"></i>
</span>
<span>Demo</span>
</a>
</span>
<span class="link-block">
<a href="https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-database"></i>
</span>
<span>Dataset</span>
</a>
</span>
<span class="link-block">
<a href="https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-share-square"></i>
</span>
<span>Model</span>
</a>
</span> -->




<!-- <span class="link-block">
<a href="#"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-youtube"></i>
</span>
<span>Video</span>
</a>
</span> -->
</div>
</div>
</div>
Expand All @@ -266,47 +193,6 @@ <h3 class="title is-3 publication-title">Improved Baselines with Visual Instruct
</div>
</section>

<!-- <section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<h4 class="subtitle has-text-centered">
🔥<span style="color: #ff3860">[NEW!]</span> LLaVA-1.5 achieves SoTA on 11 benchmarks, with just simple modifications to the original LLaVA, utilizes all public data, completes training in ~1 day on a single 8-A100 node, and surpasses methods that use billion-scale data.
<br><br>
LLaVA represents a novel end-to-end trained large multimodal model that combines a vision encoder and Vicuna
for general-purpose visual and language understanding,
achieving impressive chat capabilities mimicking spirits of the multimodal GPT-4 and setting a new state-of-the-art accuracy on Science QA.
</h4>
</div>
</div>
</section> -->



<!-- <section class="section" style="background-color:#efeff081">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-six-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks in the language domain, but the idea is less explored in the multimodal field.
<ol type="1">
<li><b>Multimodal Instruct Data</b>. <span style="font-size: 95%;">We present the first attempt to use <a href="https://openai.com/research/gpt-4">language-only GPT-4</a> to generate multimodal language-image instruction-following data. </span></li>
<li><b>LLaVA Model</b>. <span style="font-size: 95%;">We introduce <it><b>LLaVA</b> (<b>L</b>arge <b>L</b>anguage-<b>a</b>nd-<b>V</b>ision <b>A</b>ssistant)</it>, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.</li>
<li><b>Performance</b>. <span style="font-size: 95%;">Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset.
When fine-tuned on <a href="https://scienceqa.github.io/">Science QA</a>, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%.</li>
<li><b>Open-source</b>. <span style="font-size: 95%;">We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.</li>
</ol>
</p>
</div>
</div>
</div>
</div>
</section> -->


<section class="section" style="background-color:#efeff081">
<div class="container is-max-desktop">
<!-- Abstract. -->
Expand All @@ -332,40 +218,7 @@ <h2 class="title is-3">Abstract</h2>
</div>
</section>

<!--
<section class="section" style="background-color:#efeff081">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-8">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
With the advent of large language models(LLMs) enhanced by the chain-of-thought(CoT) methodology, visual reasoning problem is usually decomposed into manageable sub-tasks and tackled sequentially with various external tools.
However, such a paradigm faces the challenge of the potential "determining hallucinations" in decision-making due to insufficient visual information and the limitation of low-level perception tools that fail to provide abstract summaries necessary for comprehensive reasoning.
We argue that converging visual context acquisition and logical reasoning is pivotal for tackling visual reasoning tasks.
This paper delves into the realm of multimodal CoT to solve intricate visual reasoning tasks with multimodal large language models~(MLLMs) and their cognitive capability.
To this end, we propose an innovative multimodal CoT framework, termed Cantor, characterized by a perception-decision architecture.
Cantor first acts as a decision generator and integrates visual inputs to analyze the image and problem, ensuring a closer alignment with the actual context.
Furthermore, Cantor leverages the advanced cognitive functions of MLLMs to perform as multifaceted experts for deriving higher-level information, enhancing the CoT generation process.
Our extensive experiments demonstrate the efficacy of the proposed framework, showing significant improvements in multimodal CoT performance across two complex visual reasoning datasets, without necessitating fine-tuning or ground-truth rationales.
</p>
</div>
</div>

<div class="column is-4">
<div class="has-text-centered" style="display: flex; align-items: center; justify-content: center; height: 22%;"></div>
<figure >
<img src="images/fig1.png" alt="Descriptive Alt Text">
</figure>
</div>
</div>
</div>
</div>
</section>
-->



<section class="section">
Expand Down Expand Up @@ -575,85 +428,6 @@ <h2 class="title">BibTeX</h2>
chatHistory.innerHTML = "";
}

//
const conversations = [
{
"description": "monalisa",
"turns": [
// ["Description", "Please read the description and answer the question."],
["User", "", "/static/images/monalisa.jpg"],
["User", "Do you know who drew this painting?​"],
["LLaVA",
"The painting depicts a woman, commonly believed to be Mona Lisa, the famous artwork by Leonardo da Vinci. " +
"It is a portrait painting that showcases the woman's enigmatic smile and has become one of the most famous and iconic art pieces in the world. " +
"The original work is displayed in the Louvre Museum in Paris, and it is known for its intricate details, use of oil paint, and the artist's innovative techniques that contributed to its enduring appeal and mystery.​​"
]
]
},
{
"description": "monalisa with dog head",
"turns": [
["User", "", "/static/images/monalisa-fun.jpg"],
["User", "Do you know who drew this painting?​"],
["LLaVA", "The painting depicts a dog in a humorous situation, where it appears to be wearing a woman's dress and standing in front of a landscape. Although the artist of the painting is not explicitly identified, it could be a reference to the famous artist Da Vinci, known for his artistic works, including the Mona Lisa. The painting is likely a creative and comical take on the traditional portrait style, replacing a human figure with a dog dressed in a woman's clothing, making it an interesting and amusing visual piece.​"]
]
},
{
"description": "Titanic Rose and Jack",
"turns": [
["User", "", "/static/images/titanic.jpg"],
["User", "What's the ending of this movie?​​"],
["LLaVA", "​In the image, a young woman is leaning on a man in front of a beautiful sunset, with both of them standing on a boat. They appear to be enjoying a romantic moment together. The scene suggests that they could be depicted as the characters Jack and Rose from the popular movie Titanic. In the movie, their romance blossoms aboard the fictional ocean liner Titanic, with the pair ultimately sharing a passionate kiss as the ship sinks into the ocean. The ending of the movie is known for its emotional and heartfelt scenes, where the characters are forced to say goodbye to each other as they are separated by the sinking ship. The image evokes a sense of romance and nostalgia, paying tribute to the iconic love story portrayed in the movie Titanic.​"]
]
},
{
"description": "Fridge",
"turns": [
["User", "", "/static/images/fridge.jpg"],
["User", "What are the meals that I can cook with these?​​"],
["LLaVA", "With the variety of food items stored in the refrigerator, you can prepare several meals. For example, you can create a fruit salad using the strawberries, blueberries, and carrots. Additionally, you can make a delicious smoothie using the fruits and vegetables, which would be a healthy and nutritious snack or breakfast option. The milk and eggs can be used to create various dishes, such as a milk-based smoothie, a baked casserole, or scrambled eggs. The juice can also be used to make a fresh fruit and juice mix or as a refreshing homemade popsicle. The possibilities are vast, and the meals you can create will depend on your preferences and dietary needs.​​"],
["User", "I would like to do the fruit salad. Show me the recipe.​"],
["LLaVA",
`Certainly! Here's a simple fruit salad recipe using the items in your refrigerator:​
Ingredients:​
- 2 cups fresh strawberries, sliced​
- 1 cup fresh blueberries​
- 1 large carrot, grated​
- Juice of half a lemon​
- A handful of chopped fresh parsley or mint​
- A few ice cubes​
- Optional: honey or maple syrup (adjust to taste)​
Instructions:​
1. In a large bowl, combine the sliced strawberries, blueberries, and carrot. Gently toss them together to mix.​
2. Drizzle the juice of half a lemon over the fruit and vegetables to prevent them from oxidizing and add a refreshing citrus flavor.​
3. Sprinkle the chopped fresh parsley or mint over the fruit and gently toss once more.​
4. If desired, add a drizzle of honey or maple syrup for a touch of sweetness. Adjust the amount according to your taste preferences.​
5. Cover the bowl and refrigerate the fruit salad for at least 30 minutes or up to a few hours. This allows the flavors to meld and creates a more refreshing salad.​
6. Serve the fruit salad in cups or bowls, garnished with a sprinkle of fresh parsley or mint, if desired.​
This fruit salad recipe is perfect for a healthy snack or side dish, and with the variety of ingredients available in your refrigerator, it should be easy to put together quickly.`]]
},
{
"description": "Beautiful View",
"turns": [
["User", "", "/static/images/view.jpg"],
["User", "What are the things I should be cautious about when I visit here?​​"],
["LLaVA", "When visiting this peaceful location with a pier extending over a serene lake, surrounded by mountains, there are several things to be cautious about. First, pay attention to the weather conditions, as sudden changes in weather, especially strong winds or storms, can cause the water to become rough and pose risks to the stability of the pier. Second, be mindful of the structural integrity of the pier; avoid overloading it with too many people or excessive weight, as this could potentially damage the pier or cause it to collapse. Additionally, be aware of the surrounding wildlife and respect their space; do not harm or disturb them in any way. Finally, if you decide to go fishing, ensure that you adhere to any local fishing regulations and use appropriate fishing techniques to keep the lake ecosystem healthy and maintain the balance of the food chain.​​"],
["User", "Compose an engaging travel blog post about my trip to this place, highlighting the beautiful sceneries, joyful experiences, and include possible anecdotes.​​"],
["LLaVA",
`Traveling to this picturesque location with a serene lake and mountains in the background was a refreshing and rejuvenating experience. The pier extending over the calm water provided an excellent vantage point from which to appreciate the breathtaking scenery. As I stood on the pier, I couldn't help but feel a deep connection with nature, taking a moment to soak in the peaceful atmosphere.​
One of the highlights of my trip was undoubtedly the fishing experience. The area is teeming with life, and I was thrilled to land a sizeable catch, despite the modest size of the lake. The joy of fishing in such a beautiful setting was truly rewarding, and it allowed me to take a break from my busy life and truly immerse myself in the tranquility of nature.​
While exploring the surrounding area, I came across a family of birds that had built a nest close to the pier, adding to the charm of the location. Interacting with the wildlife and observing their behavior was an enriching experience that I'll cherish for years to come.​
In conclusion, this idyllic destination provided me with an opportunity to unwind and recharge, surrounded by mesmerizing natural beauty and the gentle sounds of water and wildlife. If you're looking for a serene escape from the hustle and bustle of everyday life, this pier overlooking the lake is the perfect place to find solace and reconnect with nature.`
]
]
},
];

// The current image index
let currentIndex = 0;
Expand Down

0 comments on commit f1b21a8

Please sign in to comment.