Hugging Face Transformers: Aya Vision (Based on v4.49.0)

A new model is added to transformers: Aya Vision. It is added on top of the v4.49.0 release, and can be installed from the following tag: v4.49.0-AyaVision. In order to install this version, please install with the following command: ``` pip install git+https://github.com/huggingface/[email protected] ``` If fixes are needed, they will be applied to this release; this installation may therefore be considered as stable and improving. # Aya Vision  The model is detailed in the following [blog post](https://huggingface.co/blog/aya-vision). ## Overview The Aya Vision 8B and 32B models is a state-of-the-art multilingual multimodal models developed by Cohere For AI. They build on the Aya Expanse recipe to handle both visual and textual information without compromising on the strong multilingual textual performance of the original model. Aya Vision 8B combines the `Siglip2-so400-384-14` vision encoder with the Cohere CommandR-7B language model further post-trained with the Aya Expanse recipe, creating a powerful vision-language model capable of understanding images and generating text across 23 languages. Whereas, Aya Vision 32B uses Aya Expanse 32B as the language model. Key features of Aya Vision include: - Multimodal capabilities in 23 languages - Strong text-only multilingual capabilities inherited from CommandR-7B post-trained with the Aya Expanse recipe and Aya Expanse 32B - High-quality visual understanding using the Siglip2-so400-384-14 vision encoder - Seamless integration of visual and textual information in 23 languages. ## Usage Example Here's an example usage of the Aya Vision model. ```py from transformers import AutoProcessor, AutoModelForImageTextToText import torch model_id = "CohereForAI/aya-vision-32b" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForImageTextToText.from_pretrained( model_id, device_map="auto", torch_dtype=torch.float16 ) # Format message with the aya-vision chat template messages = [ {"role": "user", "content": [ {"type": "image", "url": "https://pbs.twimg.com/media/Fx7YvfQWYAIp6rZ?format=jpg&name=medium"}, {"type": "text", "text": "चित्र में लिखा पाठ क्या कहता है?"}, ]}, ] inputs = processor.apply_chat_template( messages, padding=True, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt" ).to(model.device) gen_tokens = model.generate( **inputs, max_new_tokens=300, do_sample=True, temperature=0.3, ) print(processor.tokenizer.decode(gen_tokens[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)) ```