Hugging Face Transformers: SmolVLM-2 (Based on v4.49.0)

A new model is added to `transformers`: SmolVLM-2. It is added on top of the v4.49.0 release, and can be installed from the following tag: `v4.49.0-SmolVLM-2`. In order to install this version, please install with the following command: ```bash pip install git+https://github.com/huggingface/[email protected] ``` If fixes are needed, they will be applied to this release; this installation may therefore be considered as stable and improving. ## SmolVLM-2  SmolVLM-2 is detailed in the following [blog post](https://huggingface.co/blog/smolvlm2). The models and demos using the model are available in the following [collection](https://huggingface.co/collections/HuggingFaceTB/smolvlm2-smallest-video-lm-ever-67ab6b5e84bf8aaa60cb17c7). ## Overview SmolVLM2 is an adaptation of the Idefics3 model with two main differences: - It uses SmolLM2 for the text model. - It supports multi-image and video inputs ## Usage tips Input images are processed either by upsampling (if resizing is enabled) or at their original resolution. The resizing behavior depends on two parameters: do_resize and size. Videos should not be upsampled. If `do_resize` is set to `True`, the model resizes images so that the longest edge is 4*512 pixels by default. The default resizing behavior can be customized by passing a dictionary to the `size` parameter. For example, `{"longest_edge": 4 * 512}` is the default, but you can change it to a different value if needed. Here’s how to control resizing and set a custom size: ```python image_processor = SmolVLMImageProcessor(do_resize=True, size={"longest_edge": 2 * 512}, max_image_size=512) ``` Additionally, the `max_image_size` parameter, which controls the size of each square patch the image is decomposed into, is set to 512 by default but can be adjusted as needed. After resizing (if applicable), the image processor decomposes the images into square patches based on the `max_image_size` parameter. This model was contributed by [orrzohar](https://huggingface.co/orrzohar). ## Usage example ### Single Media inference The model can accept both images and videos as input, but you should use only one of the modalities at a time. Here's an example code for that. ```python import torch from transformers import AutoProcessor, AutoModelForImageTextToText processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM2-256M-Video-Instruct") model = AutoModelForImageTextToText.from_pretrained( "HuggingFaceTB/SmolVLM2-256M-Video-Instruct", torch_dtype=torch.bfloat16, device_map="cuda" ) conversation = [ { "role": "user", "content":[ {"type": "image", "url": "http://images.cocodataset.org/val2017/000000039769.jpg"}, {"type": "text", "text": "Describe this image."} ] } ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device, dtype=torch.bfloat16) output_ids = model.generate(**inputs, max_new_tokens=128) generated_texts = processor.batch_decode(output_ids, skip_special_tokens=True) print(generated_texts) # Video conversation = [ { "role": "user", "content": [ {"type": "video", "path": "/path/to/video.mp4"}, {"type": "text", "text": "Describe this video in detail"} ] }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device, dtype=torch.bfloat16) generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=100) generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True) print(generated_texts[0]) ```