Transformers documentation

LLaVA-OneVision

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.1).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

该模型于 2024-08-06 在 HF papers 上发布，并于 2024-09-05 贡献至 Hugging Face Transformers。

LLaVA-OneVision

概述

LLaVA-OneVision 模型由 Bo Li、Yuanhan Zhang、Dong Guo、Renrui Zhang、Feng Li、Hao Zhang、Kaichen Zhang、Yanwei Li、Ziwei Liu 和 Chunyuan Li 在 LLaVA-OneVision: Easy Visual Task Transfer 中提出。

LLaVA-OneVision 是一个视觉语言模型，能够根据一张或多张图像/视频生成文本。该模型由 SigLIP 视觉编码器和 Qwen2 语言骨干网络组成。图像采用 anyres-9 技术进行处理，即将图像分割为 9 个块，以更好地处理高分辨率图像并尽可能多地捕捉细节。然而，为了更节省内存，视频每帧会被池化（pooling）为总长度为 196 个 token 的序列。LLaVA-OneVision 提供 0.5B、7B 和 72B 三种规模，并在基准评估中取得了卓越的性能。

该论文的摘要如下：

我们提出了 LLaVA-OneVision，这是一个开放的大型多模态模型（LMMs）系列，它整合了我们在 LLaVA-NeXT 博客系列中关于数据、模型和视觉表示的见解。实验结果表明，LLaVA-OneVision 是首个能够同时在三个重要的计算机视觉场景——单图像、多图像和视频场景——中突破开放 LMMs 性能边界的单一模型。重要的是，LLaVAOneVision 的设计允许在不同模态/场景之间进行强大的迁移学习，从而产生新的涌现能力。特别是，通过从图像到视频的任务迁移，展示了强大的视频理解和跨场景能力。

LLaVA-OneVision architecture. Taken from the original paper.

提示：

我们建议用户在计算批量生成时使用 padding_side="left"，因为这能带来更准确的结果。只需确保在生成之前调用 processor.tokenizer.padding_side = "left" 即可。

Llava-OneVision 针对图像使用不同数量的 patch，因此除了在处理输入时执行的填充外，还必须在建模代码内部对输入进行填充。如果模型处于 eval() 模式，默认设置为“左侧填充”，否则为“右侧填充”。

使用聊天模板格式化提示

每个 checkpoint 都使用特定的提示格式进行训练，具体取决于底层的大语言模型骨干网络。为确保格式正确，请使用处理器的 apply_chat_template 方法。

重要：

您必须构建对话历史——直接传入普通字符串将无法正常工作。
每条消息应为一个字典，包含 "role" 和 "content" 键。
"content" 应为包含不同模态（如 "text" 和 "image"）的字典列表。

以下是一个输入结构的示例。我们将使用 llava-onevision-qwen2-7b-si-hf 以及包含文本和图像的对话历史。每个 content 字段都必须是一个字典列表，如下所示：

from transformers import AutoProcessor


processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-7b-si-hf")

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What’s shown in this image?"},
        ],
    },
    {
        "role": "assistant",
        "content": [{"type": "text", "text": "This image shows a red stop sign."},]
    },
    {

        "role": "user",
        "content": [
            {"type": "text", "text": "Describe the image in more details."},
        ],
    },
]

text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images
print(text_prompt)
'<|im_start|>user\n<image>What is shown in this image?<|im_end|>\n<|im_start|>assistant\nPage showing the list of options.<|im_end|>'

🚀 额外提示： 如果你使用的是 transformers>=4.49.0，还可以从 apply_chat_template 获取向量化输出。有关如何使用它的更多详细信息，请参阅下方的 使用示例。

该模型由 RaushanTurganbay 贡献。原始代码可在此处找到。

使用示例

单图像推理

以下是加载模型并以半精度（torch.float16）执行推理的方法：

import torch

from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration


processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf")
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    "llava-hf/llava-onevision-qwen2-7b-ov-hf",
    device_map="auto"
)

# prepare image and text prompt, using the appropriate prompt template
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": url},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    },
]
inputs = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
inputs = inputs.to(model.device, torch.float16)

# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
'user\n\nWhat is shown in this image?\nassistant\nThe image shows a radar chart, also known as a spider chart or a star chart, which is used to compare multiple quantitative variables. Each axis represents a different variable, and the chart is filled with'

多图像推理

LLaVa-OneVision 支持以多张图片作为输入进行推理，这些图片可以属于同一个提示词，也可以属于不同的提示词（在批量推理中）。为此，您必须使用带有 “ov” 后缀的检查点。对于多张图片的情况，我们建议使用嵌套的图片列表作为输入。否则，每张图片都会被分块处理，从而消耗大量内存。具体操作如下：

import torch

from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration


# Load the model in half-precision
model = LlavaOnevisionForConditionalGeneration.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf", device_map="auto")
processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf")

# Prepare a batch of two prompts, where the first one is a multi-turn conversation and the second is not
conversation_1 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "There is a red stop sign in the image."},
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "http://images.cocodataset.org/val2017/000000039769.jpg"},
            {"type": "text", "text": "What about this image? How many cats do you see?"},
        ],
    },
]

conversation_2 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg"},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    },
]

inputs = processor.apply_chat_template(
    [conversation_1, conversation_2],
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    padding=True,
    padding_side="left",
    return_tensors="pt",
).to(model.device, torch.float16)

# Generate
generate_ids = model.generate(**inputs, max_new_tokens=30)
processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
['user\n\nWhat is shown in this image?\nassistant\nThere is a red stop sign in the image.\nuser\n\nWhat about this image? How many cats do you see?\nassistant\ntwo', 'user\n\nWhat is shown in this image?\nassistant\nThe image shows a whimsical scene of a snowman sitting by a campfire. The snowman is anthropomorphized, wearing a hat and']

视频推理

LLaVa-OneVision 还支持以视频作为输入进行推理，其中视频帧被视为多张图片。以下是具体操作方法：

from huggingface_hub import hf_hub_download
import torch
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration

# Load the model in half-precision
model = LlavaOnevisionForConditionalGeneration.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf", device_map="auto")
processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf")

video_path = hf_hub_download(repo_id="raushan-testing-hf/videos-test", filename="sample_demo_1.mp4", repo_type="dataset")
conversation = [
    {

        "role": "user",
        "content": [
            {"type": "video", "path": video_path},
            {"type": "text", "text": "Why is this video funny?"},
            ],
    },
]

inputs = processor.apply_chat_template(
    conversation,
    num_frames=8
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device, torch.float16)

out = model.generate(**inputs, max_new_tokens=60)
processor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)
["user\n\nWhy is this video funny?\nassistant\nThe video appears to be humorous because it shows a young child, who is wearing glasses and holding a book, seemingly reading with a serious and focused expression. The child's glasses are a bit oversized for their face, which adds a comical touch, as it's a common trope to see children wearing"]

模型优化

使用 bitsandbytes 进行量化

该模型可以以 8 位或 4 位精度加载，在保持原始模型性能的同时大幅降低内存需求。首先请确保已安装 bitsandbytes，pip install bitsandbytes，并确保拥有该库支持的 GPU/加速器。

bitsandbytes 正在重构以支持 CUDA 之外的多种后端。目前，ROCm（AMD GPU）和 Intel CPU 的实现已趋于成熟，Intel XPU 正在开发中，预计将在第四季度/第一季度支持 Apple Silicon。有关安装说明和最新的后端更新，请访问此链接。

我们非常重视您的反馈，以便在正式发布前发现并修复问题！请查阅这些文档以获取更多详情和反馈链接。

只需将上述代码片段更改为：

from transformers import BitsAndBytesConfig, LlavaOnevisionForConditionalGeneration


# specify how to quantize the model
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = LlavaOnevisionForConditionalGeneration.from_pretrained(model_id, quantization_config=quantization_config, device_map="auto")

使用 Flash-Attention 2 进一步加快生成速度

首先，请确保已安装 flash-attn。有关该包的安装方法，请参阅 Flash Attention 的原始仓库。只需将上述代码片段更改为：

from transformers import LlavaOnevisionForConditionalGeneration


model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    model_id,
    use_flash_attention_2=True
).to(0, device_map="auto")

Transformers

LLaVA-OneVision

概述

使用聊天模板格式化提示

使用示例

单图像推理

多图像推理

视频推理

模型优化

使用 bitsandbytes 进行量化

使用 Flash-Attention 2 进一步加快生成速度

LlavaOnevisionConfig

class transformers.LlavaOnevisionConfig

LlavaOnevisionProcessor

class transformers.LlavaOnevisionProcessor

__call__

LlavaOnevisionImageProcessor

class transformers.LlavaOnevisionImageProcessor

预处理

LlavaOnevisionImageProcessorPil

class transformers.LlavaOnevisionImageProcessorPil

预处理

LlavaOnevisionVideoProcessor

class transformers.LlavaOnevisionVideoProcessor

LlavaOnevisionModel

class transformers.LlavaOnevisionModel

forward

get_image_features

get_placeholder_mask

get_video_features

pack_image_features

LlavaOnevisionForConditionalGeneration

类 transformers.LlavaOnevisionForConditionalGeneration

forward

get_image_features

get_video_features

call