|
| 1 | + |
| 2 | +<!--Copyright 2025 The HuggingFace Team. All rights reserved. |
| 3 | +
|
| 4 | +Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with |
| 5 | +the License. You may obtain a copy of the License at |
| 6 | +
|
| 7 | +http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | +
|
| 9 | +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on |
| 10 | +an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the |
| 11 | +specific language governing permissions and limitations under the License. |
| 12 | +
|
| 13 | +⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be |
| 14 | +rendered properly in your Markdown viewer. |
| 15 | +
|
| 16 | +--> |
| 17 | + |
| 18 | +# Gemma3 |
| 19 | + |
| 20 | +## Overview |
| 21 | + |
| 22 | +The Gemma 3 model was proposed in the [Gemma 3 Techncial Report](https://goo.gle/Gemma3Report) by Google. It is a vision-language model composed by a [SigLIP](siglip) vision encoder and a [Gemma 2](gemma_2) language decoder, linked by a multimodal linear projection. It cuts an image into a fixed number of tokens, in the same way as SigLIP, as long as the image does not exceed certain aspect ratio. For images that exceed the given aspect ratio, it crops the image into multiple smaller patches and concatenates them with the base image embedding. One particularity is that the model uses bidirectional attention on all the image tokens. In addition, the model interleaves sliding window local attention with full causal attention in the language backbone, where each sixth layer is a full causal attention layer. |
| 23 | + |
| 24 | +This model was contributed by [Ryan Mullins](https://huggingface.co/RyanMullins), [Raushan Turganbay](https://huggingface.co/RaushanTurganbay) [Arthur Zucker](https://huggingface.co/ArthurZ), and [Pedro Cuenca](https://huggingface.co/pcuenq). |
| 25 | + |
| 26 | + |
| 27 | +## Usage tips |
| 28 | + |
| 29 | + |
| 30 | +- For image+text and image-only inputs use `Gemma3ForConditionalGeneration`. |
| 31 | +- For text-only inputs use `Gemma3ForCausalLM` for generation to avoid loading the vision tower. |
| 32 | +- Each sample can contain multiple images, and the number of images can vary between samples. However, make sure to pass correctly batched images to the processor, where each batch is a list of one or more images. |
| 33 | +- The text passed to the processor should have a `<start_of_image>` token wherever an image should be inserted. |
| 34 | +- The processor has its own `apply_chat_template` method to convert chat messages to model inputs. See the examples below for more details on how to use it. |
| 35 | + |
| 36 | + |
| 37 | +### Image cropping for high resolution images |
| 38 | + |
| 39 | +The model supports cropping images into smaller patches when the image aspect ratio exceeds a certain value. By default the images are not cropped and only the base image is forwarded to the model. Users can set `do_pan_and_scan=True` to obtain several crops per image along with the base image to improve the quality in DocVQA or similar tasks requiring higher resolution images. |
| 40 | + |
| 41 | +Pan and scan is an inference time optimization to handle images with skewed aspect ratios. When enabled, it improves performance on tasks related to document understanding, infographics, OCR, etc. |
| 42 | + |
| 43 | +```python |
| 44 | + |
| 45 | +processor = AutoProcessor.from_pretrained("google/gemma-3-4b-it", padding_side="left") |
| 46 | + |
| 47 | +url = "https://media.istockphoto.com/id/1192867753/photo/cow-in-berchida-beach-siniscola.jpg?s=612x612&w=0&k=20&c=v0hjjniwsMNfJSuKWZuIn8pssmD5h5bSN1peBd1CmH4=" |
| 48 | +messages = [ |
| 49 | + { |
| 50 | + "role": "system", |
| 51 | + "content": [ |
| 52 | + {"type": "text", "text": "You are a helpful assistant."} |
| 53 | + ] |
| 54 | + }, |
| 55 | + { |
| 56 | + "role": "user", "content": [ |
| 57 | + {"type": "image", "url": url}, |
| 58 | + {"type": "text", "text": "What is shown in this image?"}, |
| 59 | + ] |
| 60 | + }, |
| 61 | +] |
| 62 | +inputs = processor.apply_chat_template( |
| 63 | + messages, |
| 64 | + tokenize=True, |
| 65 | + return_dict=True, |
| 66 | + return_tensors="pt", |
| 67 | + add_generation_prompt=True, |
| 68 | + do_pan_and_scan=True, |
| 69 | +).to(model.device) |
| 70 | + |
| 71 | +``` |
| 72 | + |
| 73 | + |
| 74 | +## Usage Example |
| 75 | + |
| 76 | +### Single-image Inference |
| 77 | + |
| 78 | +```python |
| 79 | +from transformers import AutoProcessor, Gemma3ForConditionalGeneration |
| 80 | + |
| 81 | +model_id = "google/gemma-3-4b-it" |
| 82 | +model = Gemma3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") |
| 83 | +processor = AutoProcessor.from_pretrained(model_id, padding_side="left") |
| 84 | + |
| 85 | +url = "https://media.istockphoto.com/id/1192867753/photo/cow-in-berchida-beach-siniscola.jpg?s=612x612&w=0&k=20&c=v0hjjniwsMNfJSuKWZuIn8pssmD5h5bSN1peBd1CmH4=" |
| 86 | +messages = [ |
| 87 | + { |
| 88 | + "role": "system", |
| 89 | + "content": [ |
| 90 | + {"type": "text", "text": "You are a helpful assistant."} |
| 91 | + ] |
| 92 | + }, |
| 93 | + { |
| 94 | + "role": "user", "content": [ |
| 95 | + {"type": "image", "url": url}, |
| 96 | + {"type": "text", "text": "What is shown in this image?"}, |
| 97 | + ] |
| 98 | + }, |
| 99 | +] |
| 100 | +inputs = processor.apply_chat_template( |
| 101 | + messages, |
| 102 | + tokenize=True, |
| 103 | + return_dict=True, |
| 104 | + return_tensors="pt", |
| 105 | + add_generation_prompt=True, |
| 106 | +).to(model.device) |
| 107 | + |
| 108 | +output = model.generate(**inputs, max_new_tokens=50) |
| 109 | +print(processor.decode(output[0], skip_special_tokens=True)[inputs.input_ids.shape[1]: ]) |
| 110 | +``` |
| 111 | + |
| 112 | +### Multi-image Inference |
| 113 | + |
| 114 | +```python |
| 115 | +model_id = "google/gemma-3-4b-it" |
| 116 | +model = Gemma3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") |
| 117 | +processor = AutoProcessor.from_pretrained(model_id, padding_side="left") |
| 118 | + |
| 119 | +url_cow = "https://media.istockphoto.com/id/1192867753/photo/cow-in-berchida-beach-siniscola.jpg?s=612x612&w=0&k=20&c=v0hjjniwsMNfJSuKWZuIn8pssmD5h5bSN1peBd1CmH4=" |
| 120 | +url_stop = "https://www.ilankelman.org/stopsigns/australia.jpg" |
| 121 | +messages = [ |
| 122 | + { |
| 123 | + "role": "system", |
| 124 | + "content": [ |
| 125 | + {"type": "text", "text": "You are a helpful assistant."} |
| 126 | + ] |
| 127 | + }, |
| 128 | + { |
| 129 | + "role": "user", "content": [ |
| 130 | + {"type": "image", "url": url_cow}, |
| 131 | + {"type": "image", "url": url_stop}, |
| 132 | + {"type": "text", "text": "Are these two images identical?"}, |
| 133 | + ] |
| 134 | + }, |
| 135 | +] |
| 136 | +inputs = processor.apply_chat_template( |
| 137 | + messages, |
| 138 | + tokenize=True, |
| 139 | + return_dict=True, |
| 140 | + return_tensors="pt", |
| 141 | + add_generation_prompt=True, |
| 142 | +).to(model.device) |
| 143 | + |
| 144 | +output = model.generate(**inputs, max_new_tokens=50) |
| 145 | +print(processor.decode(output[0], skip_special_tokens=True)[inputs.input_ids.shape[1]: ]) |
| 146 | + |
| 147 | +``` |
| 148 | + |
| 149 | +### Text-only inference |
| 150 | + |
| 151 | +You can use the VLMs for text-only generation by omitting images in your input. However, you can also load the models in text-only mode as shown below. This will skip loading the vision tower and will save resources when you just need the LLM capabilities. |
| 152 | +```python |
| 153 | +from transformers import AutoTokenizer, Gemma3ForCausalLM |
| 154 | + |
| 155 | +model_id = "google/gemma-3-1b-it" |
| 156 | + |
| 157 | +tokenizer = AutoTokenizer.from_pretrained(model_id) |
| 158 | +model = Gemma3ForCausalLM.from_pretrained(model_id, device_map="auto") |
| 159 | + |
| 160 | +input_ids = tokenizer("Write me a poem about Machine Learning.", return_tensors="pt").to(model.device) |
| 161 | + |
| 162 | +outputs = model.generate(**input_ids, max_new_tokens=100) |
| 163 | +text = tokenizer.batch_decode(outputs, skip_special_tokens=True) |
| 164 | + |
| 165 | +print(text) |
| 166 | + |
| 167 | +``` |
| 168 | + |
| 169 | + |
| 170 | +## Gemma3ImageProcessor |
| 171 | + |
| 172 | +[[autodoc]] Gemma3ImageProcessor |
| 173 | + |
| 174 | +## Gemma3ImageProcessorFast |
| 175 | + |
| 176 | +[[autodoc]] Gemma3ImageProcessorFast |
| 177 | + |
| 178 | +## Gemma3Processor |
| 179 | + |
| 180 | +[[autodoc]] Gemma3Processor |
| 181 | + |
| 182 | +## Gemma3TextConfig |
| 183 | + |
| 184 | +[[autodoc]] Gemma3TextConfig |
| 185 | + |
| 186 | +## Gemma3Config |
| 187 | + |
| 188 | +[[autodoc]] Gemma3Config |
| 189 | + |
| 190 | +## Gemma3TextModel |
| 191 | + |
| 192 | +[[autodoc]] Gemma3TextModel |
| 193 | + - forward |
| 194 | + |
| 195 | +## Gemma3ForCausalLM |
| 196 | + |
| 197 | +[[autodoc]] Gemma3ForCausalLM |
| 198 | + - forward |
| 199 | + |
| 200 | +## Gemma3ForConditionalGeneration |
| 201 | + |
| 202 | +[[autodoc]] Gemma3ForConditionalGeneration |
| 203 | + - forward |
0 commit comments