Hugging Face Transformers: Gemma 3 (Based on v4.49.0)

A new model is added to transformers: Gemma 3. It is added on top of the v4.49.0 release, and can be installed from the following tag: v4.49.0-Gemma-3. In order to install this version, please install with the following command: ``` pip install git+https://github.com/huggingface/[email protected] ``` If fixes are needed, they will be applied to this release; this installation may therefore be considered as stable and improving. # Gemma 3  The model is detailed in the following [blog post](https://huggingface.co/blog/gemma3). The models and demos using the model are available in the following [collection](https://huggingface.co/collections/google/gemma-3-release-67c6c6f89c4f76621268bb6d). A Space to play around with the [12B-it flavor is available here](https://huggingface.co/spaces/huggingface-projects/gemma-3-12b-it). ## Overview The Gemma 3 model was proposed by Google. It is a vision-language model composed by a [SigLIP](https://huggingface.co/docs/transformers/model_doc/siglip) vision encoder and a [Gemma 2](https://huggingface.co/docs/transformers/model_doc/gemma_2) language decoder linked by a multimodal linear projection. It cuts an image into a fixed number of tokens same way as Siglip if the image does not exceed certain aspect ratio. For images that exceed the given aspect ratio, it crops the image into multiple smaller pacthes and concatenates them with the base image embedding. One particularity is that the model uses bidirectional attention on all the image tokens. Also, the model interleaves sliding window local attention with full causal attention in the language backbone, where each sixth layer is a full causal attention layer. ## Usage tips - For image+text and image-only inputs use `Gemma3ForConditionalGeneration`. - For text-only inputs use `Gemma3ForCausalLM` for generation to avoid loading the vision tower. - Each sample can contain multiple images, and the number of images can vary between samples. However make sure to pass correctly batched images to the processor, where each batch is a list of one or more images. - The text passed to the processor should have the `"<start_of_image_>"` token where the images should be inserted. - The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as text to the processor. You can also get a vectorized output from `apply_chat_template`. See the examples below for more details on how to use it. ### Image cropping for high resolution images The model supports cropping images into smaller patches when the image aspect ratio exceeds a certain value. By default the images are not cropped and only the base image is forwarded to the model. Users can set `do_pan_and_scan=True` to obtain several crops per image along with the base image to improve the quality in DocVQA or similar tasks requiring higher resolution images. Pan and scan is an inference time optimization to handle images with skewed aspect ratios. When enabled, it improves performance on tasks related to document understanding, infographics, OCR, etc. ```python from transformers import AutoProcessor processor = AutoProcessor.from_pretrained("google/gemma-3-4b-it", padding_side="left") url = "https://media.istockphoto.com/id/1192867753/photo/cow-in-berchida-beach-siniscola.jpg?s=612x612&w=0&k=20&c=v0hjjniwsMNfJSuKWZuIn8pssmD5h5bSN1peBd1CmH4=" messages = [ { "role": "system", "content": [ {"type": "text", "text": "You are a helpful assistant."} ] }, { "role": "user", "content": [ {"type": "image", "url": url}, {"type": "text", "text": "What is shown in this image?"}, ] }, ] inputs = processor.apply_chat_template( messages, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True, do_pan_and_scan=True, ).to(model.device) ``` ## Usage Example ### Single-image Inference ```python from transformers import AutoProcessor, Gemma3ForConditionalGeneration model_id = "google/gemma-3-4b-it" model = Gemma3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") processor = AutoProcessor.from_pretrained(model_id, padding_side="left") url = "https://media.istockphoto.com/id/1192867753/photo/cow-in-berchida-beach-siniscola.jpg?s=612x612&w=0&k=20&c=v0hjjniwsMNfJSuKWZuIn8pssmD5h5bSN1peBd1CmH4=" messages = [ { "role": "system", "content": [ {"type": "text", "text": "You are a helpful assistant."} ] }, { "role": "user", "content": [ {"type": "image", "url": url}, {"type": "text", "text": "What is shown in this image?"}, ] }, ] inputs = processor.apply_chat_template( messages, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True, ).to(model.device) output = model.generate(**inputs, max_new_tokens=50) print(processor.decode(output[0], skip_special_tokens=True)[inputs.input_ids.shape[1]: ]) ``` ### Multi-image Inference ```python from transformers import AutoTokenizer, Gemma3ForCausalLM model_id = "google/gemma-3-4b-it" model = Gemma3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") processor = AutoProcessor.from_pretrained(model_id, padding_side="left") url_cow = "https://media.istockphoto.com/id/1192867753/photo/cow-in-berchida-beach-siniscola.jpg?s=612x612&w=0&k=20&c=v0hjjniwsMNfJSuKWZuIn8pssmD5h5bSN1peBd1CmH4=" url_stop = "https://www.ilankelman.org/stopsigns/australia.jpg" messages = [ { "role": "system", "content": [ {"type": "text", "text": "You are a helpful assistant."} ] }, { "role": "user", "content": [ {"type": "image", "url": url_cow}, {"type": "image", "url": url_stop}, {"type": "text", "text": "Are these two images identical?"}, ] }, ] inputs = processor.apply_chat_template( messages, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True, ).to(model.device) output = model.generate(**inputs, max_new_tokens=50) print(processor.decode(output[0], skip_special_tokens=True)[inputs.input_ids.shape[1]: ]) ``` ### Text-only inference ```python from transformers import AutoTokenizer, Gemma3ForCausalLM model_id = "google/gemma-3-1b-it" tokenizer = AutoTokenizer.from_pretrained(model_id) model = Gemma3ForCausalLM.from_pretrained(model_id, device_map="auto") input_ids = tokenizer("Write me a poem about Machine Learning.", return_tensors="pt").to(model.device) outputs = model.generate(**input_ids, max_new_tokens=100) text = tokenizer.batch_decode(outputs, skip_special_tokens=True) print(text) ```