diff --git a/gemini-2/video_understanding.ipynb b/gemini-2/video_understanding.ipynb index 9066798a8..3b7413ead 100644 --- a/gemini-2/video_understanding.ipynb +++ b/gemini-2/video_understanding.ipynb @@ -1,11 +1,20 @@ { "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "lb5yiH5h8x3h" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "metadata": { "cellView": "form", - "id": "VGj9OhLBPVD8" + "id": "906e07f6e562" }, "outputs": [], "source": [ @@ -50,7 +59,7 @@ "id": "3w14yjWnPVD-" }, "source": [ - "Gemini has from the begining been a multidal model, capable of analyzing all sorts of medias using its [long context window](https://developers.googleblog.com/en/new-features-for-the-gemini-api-and-google-ai-studio/).\n", + "Gemini has from the begining been a multimodal model, capable of analyzing all sorts of medias using its [long context window](https://developers.googleblog.com/en/new-features-for-the-gemini-api-and-google-ai-studio/).\n", "\n", "[Gemini 2.0](https://ai.google.dev/gemini-api/docs/models/gemini-v2) bring video analysis to a whole new level as illustrated in [this video](https://www.youtube.com/watch?v=Mot-JEU26GQ):\n" ] @@ -103,7 +112,7 @@ "\n", "This section install the SDK, set it up using your [API key](../quickstarts/Authentication.ipynb), imports the relevant libs, downloads the sample videos and upload them to Gemini.\n", "\n", - "Expand the section if you are curious, but you can also just run it (ity should take a couple of minutes since tehre are large files) and go straight to the examples." + "Expand the section if you are curious, but you can also just run it (it should take a couple of minutes since tehre are large files) and go straight to the examples." ] }, { @@ -306,7 +315,6 @@ } ], "source": [ - "from ctypes import pointer\n", "import time\n", "\n", "def upload_video(video_file_name):\n", @@ -373,80 +381,7 @@ "outputs": [ { "data": { - "text/markdown": [ - "```json\n", - "[\n", - " {\n", - " \"timecode\": \"00:00:00\",\n", - " \"caption\": \"A close up of brown fur with visible plant life in the background.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:00:00\",\n", - " \"caption\": \"A gray fox walks into the scene, followed by another fox. The background is a rocky area with some undergrowth. A high pitched sound effect can be heard.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:00:16\",\n", - " \"caption\": \"A grayscale shot of a mountain lion in the forest, walking on some leaves.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:00:35\",\n", - " \"caption\": \"A nighttime grayscale shot of two foxes in the forest. They appear to be digging.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:00:50\",\n", - " \"caption\": \"An overexposed, blurry shot that is most of one solid color, followed by\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:00:51\",\n", - " \"caption\": \"A nighttime shot of some foxes by rocks in a forest. A sound effect can be heard.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:01:04\",\n", - " \"caption\": \"A nighttime shot of a mountain lion in the forest with visible rock formations.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:01:28\",\n", - " \"caption\": \"A night time shot of a bobcat standing by a tree in the forest.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:01:51\",\n", - " \"caption\": \"A medium shot of a black bear in the forest during the day. It turns to its left and moves past the camera.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:01:56\",\n", - " \"caption\": \"A grayscale shot of a mountain lion moving quickly across the screen in a forest.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:02:05\",\n", - " \"caption\":\"A close up of brown fur. The camera moves to reveal a black bear.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:02:22\",\n", - " \"caption\": \"A night time grayscale shot of a fox looking at a lit up city in the background.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:02:35\",\n", - " \"caption\": \"A nighttime grayscale shot of a bear walking near the edge of a cliff.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:02:51\",\n", - " \"caption\": \"A nighttime shot of a mountain lion in the forest.\"\n", - " },\n", - "{\n", - " \"timecode\": \"00:03:04\",\n", - " \"caption\":\"A black bear is standing in the forest looking directly at the camera. The bear is moving. Then a lighter brown bear enters the frame from the right.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:03:22\",\n", - " \"caption\": \"A lighter brown bear stands in the forest, looking to the left.\"\n", - " },\n", - " {\n", - " \"timecode\": \"00:04:21\",\n", - " \"caption\": \"A nighttime shot of a bobcat with glowing eyes standing near a fallen tree limb.\"\n", - " }\n", - "]\n", - "```" - ], + "text/markdown": "```json\n[\n {\n \"timecode\": \"00:00:00\",\n \"caption\": \"A close up of brown fur with visible plant life in the background.\"\n },\n {\n \"timecode\": \"00:00:00\",\n \"caption\": \"A gray fox walks into the scene, followed by another fox. The background is a rocky area with some undergrowth. A high pitched sound effect can be heard.\"\n },\n {\n \"timecode\": \"00:00:16\",\n \"caption\": \"A grayscale shot of a mountain lion in the forest, walking on some leaves.\"\n },\n {\n \"timecode\": \"00:00:35\",\n \"caption\": \"A nighttime grayscale shot of two foxes in the forest. They appear to be digging.\"\n },\n {\n \"timecode\": \"00:00:50\",\n \"caption\": \"An overexposed, blurry shot that is most of one solid color, followed by\"\n },\n {\n \"timecode\": \"00:00:51\",\n \"caption\": \"A nighttime shot of some foxes by rocks in a forest. A sound effect can be heard.\"\n },\n {\n \"timecode\": \"00:01:04\",\n \"caption\": \"A nighttime shot of a mountain lion in the forest with visible rock formations.\"\n },\n {\n \"timecode\": \"00:01:28\",\n \"caption\": \"A night time shot of a bobcat standing by a tree in the forest.\"\n },\n {\n \"timecode\": \"00:01:51\",\n \"caption\": \"A medium shot of a black bear in the forest during the day. It turns to its left and moves past the camera.\"\n },\n {\n \"timecode\": \"00:01:56\",\n \"caption\": \"A grayscale shot of a mountain lion moving quickly across the screen in a forest.\"\n },\n {\n \"timecode\": \"00:02:05\",\n \"caption\":\"A close up of brown fur. The camera moves to reveal a black bear.\"\n },\n {\n \"timecode\": \"00:02:22\",\n \"caption\": \"A night time grayscale shot of a fox looking at a lit up city in the background.\"\n },\n {\n \"timecode\": \"00:02:35\",\n \"caption\": \"A nighttime grayscale shot of a bear walking near the edge of a cliff.\"\n },\n {\n \"timecode\": \"00:02:51\",\n \"caption\": \"A nighttime shot of a mountain lion in the forest.\"\n },\n{\n \"timecode\": \"00:03:04\",\n \"caption\":\"A black bear is standing in the forest looking directly at the camera. The bear is moving. Then a lighter brown bear enters the frame from the right.\"\n },\n {\n \"timecode\": \"00:03:22\",\n \"caption\": \"A lighter brown bear stands in the forest, looking to the left.\"\n },\n {\n \"timecode\": \"00:04:21\",\n \"caption\": \"A nighttime shot of a bobcat with glowing eyes standing near a fallen tree limb.\"\n }\n]\n```", "text/plain": [ "" ] @@ -486,7 +421,7 @@ "source": [ "The prompt used is quite a generic one, but you can get even better results if you cutomize it to your needs (like asking specifically for foxes).\n", "\n", - "The [live demo on AI Studio](https://aistudio.google.com/starter-apps/video) shows how you can postprocess this output to just directly to the the specific part of the video by clicking on the timecodes. If you are interested, you can check the [code of that demo on Github](https://github.com/google-gemini/starter-applets/tree/main/video)." + "The [live demo on AI Studio](https://aistudio.google.com/starter-apps/video) shows how you can postprocess this output to jump directly to the the specific part of the video by clicking on the timecodes. If you are interested, you can check the [code of that demo on Github](https://github.com/google-gemini/starter-applets/tree/main/video)." ] }, { @@ -511,34 +446,7 @@ "outputs": [ { "data": { - "text/markdown": [ - "Certainly! Here's a table of the sticky notes, organized into categories based on their subject matter, along with a few ideas for additional considerations:\n", - "\n", - "**Project Name Brainstorm Sticky Notes**\n", - "\n", - "| Category | Sticky Note Names |\n", - "|--------------------|--------------------|\n", - "| **Constellations** | Canis Major, Leo Minor, Lynx, Delphinus, Serpens, Centaurus, Lyra |\n", - "| **Celestial Objects**| Lunar Eclipse, Stellar Nexus, Comets Tail, Supernova Echo, Orion's Belt, Celestial Drift|\n", - "| **Mythological** | Prometheus Rising, Chimera Dream, Perseus Shield, Zephyr, Odin, Aether, Athena's Eye, Athena, Medusa, Hera, Cerberus, |\n", - "| **Scientific Concepts** | Convergence, Astral Forge, Draco, Euler's Path, Equilibrium, Chaos Theory, Golden Ratio, Infinity Loop, Taylor Series, Stokes Theorem, Bayes Theorem, Riemann's Hypothesis, Fractal, Symmetry, Vector |\n", - "| **Other** | Galactic Core, Titan, Echo, Phoenix, Pandora's Box, Sagitta|\n", - "\n", - "\n", - "**Ideas for What We May Have Missed**\n", - "\n", - "1. **Themes & Subcategories**: Is there a desired theme or subcategory (e.g., \"geometric\", \"mythological\", \"technical\", etc.)? If so, some additional terms or more targeted brainstorming in that area might be beneficial.\n", - " * **Greek Mythology**: Expand on Greek/Roman myths. Consider names like: *Icarus's Fall*, *Pegasus*, *Hades' Gate*, *Poseidon's Trident*.\n", - " * **Astronomy**: More technical names from astronomy like *Solar Flare*, *Accretion Disk*, *Gravitational Lens*, *Quasar Jet*, *Event Horizon*, etc.\n", - "2. **Style**:\n", - " * **Simplicity vs. Complexity**: Do you want a name that is straightforward or something more complex and evocative? Consider a combination of scientific and mythological names.\n", - " * **Acronyms/Initialisms**: Some project names are short acronyms (e.g., ALICE, ATLAS). Perhaps brainstorm some of those if you need a shorter title.\n", - "3. **Audience**: Who is the audience for this project? The name could be more or less technical, depending on whether it's an expert audience or a general public one.\n", - "4. **Uniqueness**: It's often valuable to research if any of these project names are already in use to avoid confusion.\n", - "5. **Team Name/Internal vs. External Name**: Is this a team name or is this a name for a product that the public will see?\n", - "\n", - "Let me know if you’d like me to explore any of these ideas or categories in more detail!" - ], + "text/markdown": "Certainly! Here's a table of the sticky notes, organized into categories based on their subject matter, along with a few ideas for additional considerations:\n\n**Project Name Brainstorm Sticky Notes**\n\n| Category | Sticky Note Names |\n|--------------------|--------------------|\n| **Constellations** | Canis Major, Leo Minor, Lynx, Delphinus, Serpens, Centaurus, Lyra |\n| **Celestial Objects**| Lunar Eclipse, Stellar Nexus, Comets Tail, Supernova Echo, Orion's Belt, Celestial Drift|\n| **Mythological** | Prometheus Rising, Chimera Dream, Perseus Shield, Zephyr, Odin, Aether, Athena's Eye, Athena, Medusa, Hera, Cerberus, |\n| **Scientific Concepts** | Convergence, Astral Forge, Draco, Euler's Path, Equilibrium, Chaos Theory, Golden Ratio, Infinity Loop, Taylor Series, Stokes Theorem, Bayes Theorem, Riemann's Hypothesis, Fractal, Symmetry, Vector |\n| **Other** | Galactic Core, Titan, Echo, Phoenix, Pandora's Box, Sagitta|\n\n\n**Ideas for What We May Have Missed**\n\n1. **Themes & Subcategories**: Is there a desired theme or subcategory (e.g., \"geometric\", \"mythological\", \"technical\", etc.)? If so, some additional terms or more targeted brainstorming in that area might be beneficial.\n * **Greek Mythology**: Expand on Greek/Roman myths. Consider names like: *Icarus's Fall*, *Pegasus*, *Hades' Gate*, *Poseidon's Trident*.\n * **Astronomy**: More technical names from astronomy like *Solar Flare*, *Accretion Disk*, *Gravitational Lens*, *Quasar Jet*, *Event Horizon*, etc.\n2. **Style**:\n * **Simplicity vs. Complexity**: Do you want a name that is straightforward or something more complex and evocative? Consider a combination of scientific and mythological names.\n * **Acronyms/Initialisms**: Some project names are short acronyms (e.g., ALICE, ATLAS). Perhaps brainstorm some of those if you need a shorter title.\n3. **Audience**: Who is the audience for this project? The name could be more or less technical, depending on whether it's an expert audience or a general public one.\n4. **Uniqueness**: It's often valuable to research if any of these project names are already in use to avoid confusion.\n5. **Team Name/Internal vs. External Name**: Is this a team name or is this a name for a product that the public will see?\n\nLet me know if you’d like me to explore any of these ideas or categories in more detail!", "text/plain": [ "" ] @@ -549,7 +457,7 @@ } ], "source": [ - "prompt = \"Transcribe the sticky notes, organize them and put it in a table. Do you think we missed anything?\" # @param {type:\"string\"}\n", + "prompt = \"Transcribe the sticky notes, organize them and put it in a table. Do you think anything is missing?\" # @param {type:\"string\"}\n", "\n", "video = post_its_video # @param [\"trailcam_video\", \"pottery_video\", \"post_its_video\", \"user_study_video\"] {\"type\":\"raw\",\"allow-input\":true}\n", "\n",