# Qwen2.5-Omni ## Overview The [Qwen2.5-Omni](https://qwenlm.github.io/blog/qwen2.5-omni/) model is a unified multiple modalities model proposed in [Qwen2.5-Omni Technical Report](https://huggingface.co/papers/2503.20215) from Qwen team, Alibaba Group. The abstract from the technical report is the following: *We present Qwen2.5-Omni, an end-to-end multimodal model designed to perceive diverse modalities, including text, images, audio, and video, while simultaneously generating text and natural speech responses in a streaming manner. To enable the streaming of multimodal information inputs, both audio and visual encoders utilize a block-wise processing approach. This strategy effectively decouples the handling of long sequences of multimodal data, assigning the perceptual responsibilities to the multimodal encoder and entrusting the modeling of extended sequences to a large language model. Such a division of labor enhances the fusion of different modalities via the shared attention mechanism. To synchronize the timestamps of video inputs with audio, we organized the audio and video sequentially in an interleaved manner and propose a novel position embedding approach, named TMRoPE (Time-aligned Multimodal RoPE). To concurrently generate text and speech while avoiding interference between the two modalities, we propose Thinker-Talker architecture. In this framework, Thinker functions as a large language model tasked with text generation, while Talker is a dual-track autoregressive model that directly utilizes the hidden representations from the Thinker to produce audio tokens as output. Both the Thinker and Talker models are designed to be trained and inferred in an end-to-end manner. For decoding audio tokens in a streaming manner, we introduce a sliding-window DiT that restricts the receptive field, aiming to reduce the initial package delay. Qwen2.5-Omni outperforms the similarly sized Qwen2-VL and Qwen2-Audio in both image and audio capabilities. Furthermore, Qwen2.5-Omni achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench. Notably, Qwen2.5-Omni is the first open-source model to achieve a level of performance in end-to-end speech instruction following that is comparable to its capabilities with text inputs, as evidenced by benchmarks such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming Talker outperform most existing streaming and non-streaming alternatives in robustness and naturalness.* ## Notes - Use [Qwen2_5OmniForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniForConditionalGeneration) to generate audio and text output. To generate only one output type, use [Qwen2_5OmniThinkerForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniThinkerForConditionalGeneration) for text-only and `Qwen2_5OmniTalkersForConditionalGeneration` for audio-only outputs. - Audio generation with [Qwen2_5OmniForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniForConditionalGeneration) supports only single batch size at the moment. - In case out out-of-memory errors hwen working with video input, decrease `processor.max_pixels`. By default the maximum is set to a very arge value and high resolution visuals will not be resized, unless resolution exceeds `processor.max_pixels`. - The processor has its own [apply_chat_template()](/docs/transformers/v5.3.0/en/main_classes/processors#transformers.ProcessorMixin.apply_chat_template) method to convert chat messages to model inputs. ## Usage example `Qwen2.5-Omni` can be found on the [Huggingface Hub](https://huggingface.co/Qwen). ### Single Media inference The model can accept text, images, audio and videos as input. Here's an example code for inference. ```python import soundfile as sf from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor model = Qwen2_5OmniForConditionalGeneration.from_pretrained( "Qwen/Qwen2.5-Omni-7B", dtype="auto", device_map="auto" ) processor = Qwen2_5OmniProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B") conversations = [ { "role": "system", "content": [ {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."} ], }, { "role": "user", "content": [ {"type": "video", "video": "/path/to/video.mp4"}, {"type": "text", "text": "What cant you hear and see in this video?"}, ], }, ] inputs = processor.apply_chat_template( conversations, load_audio_from_video=True, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", fps=1, # kwargs to be passed to `Qwen2-5-OmniProcessor` padding=True, use_audio_in_video=True, ).to(model.device) # Generation params for audio or text can be different and have to be prefixed with `thinker_` or `talker_` text_ids, audio = model.generate(**inputs, use_audio_in_video=True, thinker_do_sample=False, talker_do_sample=True) text = processor.batch_decode(text_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) sf.write( "output.wav", audio.reshape(-1).detach().cpu().numpy(), samplerate=24000, ) print(text) ``` ### Text-only generation To generate only text output and save compute by not loading the audio generation model, we can use `Qwen2_5OmniThinkerForConditionalGeneration` model. ```python from transformers import Qwen2_5OmniThinkerForConditionalGeneration, Qwen2_5OmniProcessor model = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained( "Qwen/Qwen2.5-Omni-7B", dtype="auto", device_map="auto", ) processor = Qwen2_5OmniProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B") conversations = [ { "role": "system", "content": [ {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."} ], }, { "role": "user", "content": [ {"type": "video", "video": "/path/to/video.mp4"}, {"type": "text", "text": "What cant you hear and see in this video?"}, ], }, ] inputs = processor.apply_chat_template( conversations, load_audio_from_video=True, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", fps=1, # kwargs to be passed to `Qwen2-5-OmniProcessor` padding=True, use_audio_in_video=True, ).to(model.device) text_ids = model.generate(**inputs, use_audio_in_video=True) text = processor.batch_decode(text_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) sf.write( "output.wav", audio.reshape(-1).detach().cpu().numpy(), samplerate=24000, ) print(text) ``` ### Batch Mixed Media Inference The model can batch inputs composed of mixed samples of various types such as text, images, audio and videos as input when using `Qwen2_5OmniThinkerForConditionalGeneration` model. Here is an example. ```python import soundfile as sf from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor model = Qwen2_5OmniForConditionalGeneration.from_pretrained( "Qwen/Qwen2.5-Omni-7B", dtype="auto", device_map="auto" ) processor = Qwen2_5OmniProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B") # Conversation with video only conversation1 = [ { "role": "system", "content": [ {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."} ], }, { "role": "user", "content": [ {"type": "video", "path": "/path/to/video.mp4"}, ] } ] # Conversation with audio only conversation2 = [ { "role": "system", "content": [ {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."} ], }, { "role": "user", "content": [ {"type": "audio", "path": "/path/to/audio.wav"}, ] } ] # Conversation with pure text conversation3 = [ { "role": "system", "content": [ {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."} ], }, { "role": "user", "content": [{"type": "text", "text": "who are you?"}], } ] # Conversation with mixed media conversation4 = [ { "role": "system", "content": [ {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."} ], }, { "role": "user", "content": [ {"type": "image", "path": "/path/to/image.jpg"}, {"type": "video", "path": "/path/to/video.mp4"}, {"type": "audio", "path": "/path/to/audio.wav"}, {"type": "text", "text": "What are the elements can you see and hear in these medias?"}, ], } ] conversations = [conversation1, conversation2, conversation3, conversation4] inputs = processor.apply_chat_template( conversations, load_audio_from_video=True, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", fps=1, # kwargs to be passed to `Qwen2-5-OmniProcessor` padding=True, use_audio_in_video=True, ).to(model.thinker.device) text_ids = model.generate(**inputs, use_audio_in_video=True) text = processor.batch_decode(text_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) print(text) ``` ### Usage Tips #### Image Resolution trade-off The model supports a wide range of resolution inputs. By default, it uses the native resolution for input, but higher resolutions can enhance performance at the cost of more computation. Users can set the minimum and maximum number of pixels to achieve an optimal configuration for their needs. ```python min_pixels = 128*28*28 max_pixels = 768*28*28 processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B", min_pixels=min_pixels, max_pixels=max_pixels) ``` #### Prompt for audio output If users need audio output, the system prompt must be set as "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.", otherwise the audio output may not work as expected. ```python { "role": "system", "content": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.", } ``` #### Use audio output or not The model supports both text and audio outputs, if users do not need audio outputs, they can set `enable_audio_output` in the `from_pretrained` function. This option will save about `~2GB` of GPU memory but the `return_audio` option for `generate` function will only allow to be set at `False`. ```python model = Qwen2_5OmniForConditionalGeneration.from_pretrained( "Qwen/Qwen2.5-Omni-7B", dtype="auto", device_map="auto", enable_audio_output=False, ) ``` In order to obtain a flexible experience, we recommend that users set `enable_audio_output` at `True` when initializing the model through `from_pretrained` function, and then decide whether to return audio when `generate` function is called. When `return_audio` is set to `False`, the model will only return text outputs to get text responses faster. ```python model = Qwen2_5OmniForConditionalGeneration.from_pretrained( "Qwen/Qwen2.5-Omni-7B", dtype="auto", device_map="auto", enable_audio_output=True, ) ... text_ids = model.generate(**inputs, return_audio=False) ``` #### Change voice type of output audio Qwen2.5-Omni supports the ability to change the voice of the output audio. Users can use the `spk` parameter of `generate` function to specify the voice type. The `"Qwen/Qwen2.5-Omni-7B"` checkpoint support two voice types: `Chelsie` and `Ethan`, while `Chelsie` is a female voice and `Ethan` is a male voice. By default, if `spk` is not specified, the default voice type is `Chelsie`. ```python text_ids, audio = model.generate(**inputs, spk="Chelsie") ``` ```python text_ids, audio = model.generate(**inputs, spk="Ethan") ``` #### Flash-Attention 2 to speed up generation First, make sure to install the latest version of Flash Attention 2: ```bash pip install -U flash-attn --no-build-isolation ``` Also, you should have hardware that is compatible with FlashAttention 2. Read more about it in the official documentation of the [flash attention repository](https://github.com/Dao-AILab/flash-attention). FlashAttention-2 can only be used when a model is loaded in `torch.float16` or `torch.bfloat16`. To load and run a model using FlashAttention-2, add `attn_implementation="flash_attention_2"` when loading the model: ```python from transformers import Qwen2_5OmniForConditionalGeneration model = Qwen2_5OmniForConditionalGeneration.from_pretrained( "Qwen/Qwen2.5-Omni-7B", device_map="auto", dtype=torch.bfloat16, attn_implementation="flash_attention_2", ) ``` ## Qwen2_5OmniConfig[[transformers.Qwen2_5OmniConfig]] #### transformers.Qwen2_5OmniConfig[[transformers.Qwen2_5OmniConfig]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/configuration_qwen2_5_omni.py#L935) This is the configuration class to store the configuration of a [Qwen2_5OmniForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniForConditionalGeneration). It is used to instantiate a Qwen2.5Omni model according to the specified sub-models configurations, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the [Qwen/Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) architecture. Configuration objects inherit from [PreTrainedConfig](/docs/transformers/v5.3.0/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the documentation from [PreTrainedConfig](/docs/transformers/v5.3.0/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. Example: ```python >>> from transformers import ( ... Qwen2_5OmniThinkerConfig, ... Qwen2_5OmniTalkerConfig, ... Qwen2_5OmniToken2WavConfig, ... Qwen2_5OmniForConditionalGeneration, ... Qwen2_5OmniConfig, ... ) >>> # Initializing sub-modules configurations. >>> thinker_config = Qwen2_5OmniThinkerConfig() >>> talker_config = Qwen2_5OmniTalkerConfig() >>> token2wav_config = Qwen2_5OmniToken2WavConfig() >>> # Initializing a module style configuration >>> configuration = Qwen2_5OmniConfig( ... thinker_config, talker_config, token2wav_config ... ) >>> # Initializing a model (with random weights) >>> model = Qwen2_5OmniForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` get_text_configtransformers.Qwen2_5OmniConfig.get_text_confighttps://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/configuration_qwen2_5_omni.py#L1016[{"name": "*args", "val": ""}, {"name": "**kwargs", "val": ""}]- **decoder** (`Optional[bool]`, *optional*, defaults to `False`) -- If set to `True`, then only search for decoder config names.0 Returns the config that is meant to be used with text IO. On most models, it is the original config instance itself. On specific composite models, it is under a set of valid names. **Parameters:** thinker_config (`dict`, *optional*) : Configuration of the underlying thinker sub-model. talker_config (`dict`, *optional*) : Configuration of the underlying talker sub-model. token2wav_config (`dict`, *optional*) : Configuration of the underlying codec sub-model. enable_audio_output (`bool`, *optional*, defaults to `True`) : Whether enable audio output and load talker and token2wav module. ## Qwen2_5OmniProcessor[[transformers.Qwen2_5OmniProcessor]] #### transformers.Qwen2_5OmniProcessor[[transformers.Qwen2_5OmniProcessor]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py#L106) Constructs a Qwen2_5OmniProcessor which wraps a image processor, a video processor, a feature extractor, and a tokenizer into a single processor. [Qwen2_5OmniProcessor](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniProcessor) offers all the functionalities of [Qwen2VLImageProcessorFast](/docs/transformers/v5.3.0/en/model_doc/qwen2_vl#transformers.Qwen2VLImageProcessorFast), [Qwen2VLVideoProcessor](/docs/transformers/v5.3.0/en/model_doc/qwen2_vl#transformers.Qwen2VLVideoProcessor), [WhisperFeatureExtractor](/docs/transformers/v5.3.0/en/model_doc/whisper#transformers.WhisperFeatureExtractor), and [Qwen2Tokenizer](/docs/transformers/v5.3.0/en/model_doc/qwen2#transformers.Qwen2Tokenizer). See the [~Qwen2VLImageProcessorFast](/docs/transformers/v5.3.0/en/model_doc/qwen2_vl#transformers.Qwen2VLImageProcessorFast), [~Qwen2VLVideoProcessor](/docs/transformers/v5.3.0/en/model_doc/qwen2_vl#transformers.Qwen2VLVideoProcessor), [~WhisperFeatureExtractor](/docs/transformers/v5.3.0/en/model_doc/whisper#transformers.WhisperFeatureExtractor), and [~Qwen2Tokenizer](/docs/transformers/v5.3.0/en/model_doc/qwen2#transformers.Qwen2Tokenizer) for more information. __call__transformers.Qwen2_5OmniProcessor.__call__https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py#L119[{"name": "text", "val": ": str | list[str] | list[list[str]] = None"}, {"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None"}, {"name": "videos", "val": ": typing.Union[list['PIL.Image.Image'], numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], list[list['PIL.Image.Image']], list[list[numpy.ndarray]], list[list['torch.Tensor']], transformers.video_utils.URL, list[transformers.video_utils.URL], list[list[transformers.video_utils.URL]], transformers.video_utils.Path, list[transformers.video_utils.Path], list[list[transformers.video_utils.Path]], NoneType] = None"}, {"name": "audio", "val": ": typing.Union[numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], NoneType] = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.models.qwen2_5_omni.processing_qwen2_5_omni.Qwen2_5OmniProcessorKwargs]"}]- **text** (`Union[str, list[str], list[list[str]]]`, *optional*) -- The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If you pass a pretokenized input, set `is_split_into_words=True` to avoid ambiguity with batched inputs. - **images** (`Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor]]`, *optional*) -- Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. - **videos** (`Union[list[PIL.Image.Image], numpy.ndarray, torch.Tensor, list[numpy.ndarray], list[torch.Tensor], list[list[PIL.Image.Image]], list[list[numpy.ndarray]], list[list[torch.Tensor]], ~video_utils.URL, list[~video_utils.URL], list[list[~video_utils.URL]], ~video_utils.Path, list[~video_utils.Path], list[list[~video_utils.Path]]]`, *optional*) -- Video to preprocess. Expects a single or batch of videos with pixel values ranging from 0 to 255. If passing in videos with pixel values between 0 and 1, set `do_rescale=False`. - **audio** (`Union[numpy.ndarray, torch.Tensor, list[numpy.ndarray], list[torch.Tensor]]`, *optional*) -- The audio or batch of audios to be prepared. Each audio can be a NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each audio should be of shape (C, T), where C is a number of channels, and T is the sample length of the audio. - **min_pixels** (`int`, *kwargs*, *optional*) -- Minimum number of pixels (height × width) for video frames after resizing. Frames smaller than this threshold will be upscaled to meet the minimum requirement. - **max_pixels** (`int`, *kwargs*, *optional*) -- Maximum number of pixels (height × width) for video frames after resizing. Frames larger than this threshold will be downscaled to fit within the limit. - **patch_size** (`int`, *kwargs*, *optional*) -- The spatial patch size used by the vision encoder. Video frames are divided into patches of this size in both height and width dimensions. - **temporal_patch_size** (`int`, *kwargs*, *optional*) -- The temporal patch size used by the vision encoder. This determines how many consecutive frames are grouped together as a single temporal patch. - **merge_size** (`int`, *kwargs*, *optional*) -- The merge size used for combining spatial patches. Multiple patches are merged together to reduce the sequence length while maintaining spatial information. - **min_frames** (`int`, *kwargs*, *optional*) -- Minimum number of frames to extract from the video. Videos with fewer frames will be padded or repeated to meet this requirement. - **max_frames** (`int`, *kwargs*, *optional*) -- Maximum number of frames to extract from the video. Longer videos will be truncated or sampled to fit within this limit. - **use_audio_in_video** (`bool`, *kwargs*, *optional*, defaults to `False`) -- Whether to incorporate audio information when processing videos. When enabled, audio tokens are interleaved with video tokens based on temporal alignment, creating a unified multimodal representation. - **seconds_per_chunk** (`float`, *kwargs*, *optional*, defaults to `2.0`) -- The duration (in seconds) of each video chunk when splitting long videos. This parameter controls how videos are divided into temporal segments for processing. - **position_id_per_seconds** (`int`, *kwargs* or `float`, *optional*, defaults to `25`) -- The number of position IDs allocated per second of video. This parameter controls the temporal resolution of position embeddings and is used to align video tokens with audio tokens when `use_audio_in_video=True`. - **return_tensors** (`str` or [TensorType](/docs/transformers/v5.3.0/en/internal/file_utils#transformers.TensorType), *optional*) -- If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - ****kwargs** ([ProcessingKwargs](/docs/transformers/v5.3.0/en/main_classes/processors#transformers.ProcessingKwargs), *optional*) -- Additional processing options for each modality (text, images, videos, audio). Model-specific parameters are listed above; see the TypedDict class for the complete list of supported arguments.0`~feature_extraction_utils.BatchFeature`- **data** (`dict`, *optional*) -- Dictionary of lists/arrays/tensors returned by the __call__/pad methods ('input_values', 'attention_mask', etc.). - **tensor_type** (`Union[None, str, TensorType]`, *optional*) -- You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at initialization. - **skip_tensor_conversion** (`list[str]` or `set[str]`, *optional*) -- List or set of keys that should NOT be converted to tensors, even when `tensor_type` is specified. **Parameters:** image_processor (`Qwen2VLImageProcessorFast`) : The image processor is a required input. video_processor (`Qwen2VLVideoProcessor`) : The video processor is a required input. feature_extractor (`WhisperFeatureExtractor`) : The feature extractor is a required input. tokenizer (`Qwen2Tokenizer`) : The tokenizer is a required input. chat_template (`str`) : A Jinja template to convert lists of messages in a chat into a tokenizable string. **Returns:** ``~feature_extraction_utils.BatchFeature`` - **data** (`dict`, *optional*) -- Dictionary of lists/arrays/tensors returned by the __call__/pad methods ('input_values', 'attention_mask', etc.). - **tensor_type** (`Union[None, str, TensorType]`, *optional*) -- You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at initialization. - **skip_tensor_conversion** (`list[str]` or `set[str]`, *optional*) -- List or set of keys that should NOT be converted to tensors, even when `tensor_type` is specified. ## Qwen2_5OmniForConditionalGeneration[[transformers.Qwen2_5OmniForConditionalGeneration]] #### transformers.Qwen2_5OmniForConditionalGeneration[[transformers.Qwen2_5OmniForConditionalGeneration]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L3801) The full Qwen2.5Omni model, a multimodal model composed of 3 sub-models: - [Qwen2_5OmniThinkerForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniThinkerForConditionalGeneration): a causal auto-regressive transformer takes text, audio, image, video as input and predict text tokens. - [Qwen2_5OmniTalkerForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniTalkerForConditionalGeneration): a causal auto-regressive transformer takes thinker hidden states and response as input and predict speech tokens. - [Qwen2_5OmniToken2WavModel](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniToken2WavModel): a DiT model take speech tokens as input and predict mel spectrogram and a BigVGAN vocoder take mel spectrogram as input and predict waveform. This model inherits from [PreTrainedModel](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. _forward_unimplementedtransformers.Qwen2_5OmniForConditionalGeneration.forwardhttps://github.com/huggingface/transformers/blob/v5.3.0/src/torch/nn/modules/module.py#L391[{"name": "*input", "val": ": typing.Any"}] Define the computation performed at every call. Should be overridden by all subclasses. Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them. **Parameters:** config ([Qwen2_5OmniForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniForConditionalGeneration)) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. ## Qwen2_5OmniPreTrainedModelForConditionalGeneration[[transformers.Qwen2_5OmniPreTrainedModelForConditionalGeneration]] #### transformers.Qwen2_5OmniPreTrainedModelForConditionalGeneration[[transformers.Qwen2_5OmniPreTrainedModelForConditionalGeneration]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L150) get_chunked_indextransformers.Qwen2_5OmniPreTrainedModelForConditionalGeneration.get_chunked_indexhttps://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L173[{"name": "token_indices", "val": ": Tensor"}, {"name": "tokens_per_chunk", "val": ": int"}, {"name": "remove_index", "val": ": int"}]- **token_indices** (`torch.Tensor` of shape `(seq_len, )`) -- A monotonically increasing list of token index values. - **t_ntoken_per_chunk** (`int`) -- Number of tokens per chunk (used as the chunk size threshold). - **remove_index** (`int`) An index id to subtract from `token_indices` before chunking --0`list[tuple[int, int]]`A list of tuples, each representing the start (inclusive) and end (exclusive) indices of a chunk in `token_indices`. Splits token index list into chunks based on token value ranges. Given a list of token indices, returns a list of (start, end) index tuples representing slices of the list where the token values fall within successive ranges of `t_ntoken_per_chunk`. For example, if `t_ntoken_per_chunk` is 1000, the function will create chunks such that: - the first chunk contains token values = 1000 and >> from transformers import Qwen2_5OmniThinkerForConditionalGeneration, Qwen2_5OmniThinkerConfig, Qwen2_5OmniAudioEncoderConfig, Qwen2_5OmniVisionEncoderConfig >>> # Initializing a Qwen2_5OmniAudioEncoder config >>> audio_config = Qwen2_5OmniAudioEncoderConfig() >>> # Initializing a Qwen2_5OmniVisionEncoder config >>> vision_config = Qwen2_5OmniVisionEncoderConfig() >>> # Initializing a Qwen2_5OmniTextConfig config >>> text_config = Qwen2_5OmniTextConfig() >>> # Initializing a Qwen2.5OmniThinker configuration >>> configuration = Qwen2_5OmniThinkerConfig(audio_config, vision_config, text_config) >>> # Initializing a model from the Qwen-Omni style configuration >>> model = Qwen2_5OmniThinkerForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` **Parameters:** audio_config (`dict`, *optional*) : The config dictionary of the audio backbone. vision_config (`dict`, *optional*) : The config dictionary of the vision backbone. text_config (`dict`, *optional*) : The config dictionary of the text backbone. audio_token_index (`int`, *optional*, defaults to 151646) : The audio token index to encode the audio prompt. image_token_index (`int`, *optional*, defaults to 151655) : The image token index to encode the image prompt. video_token_index (`int`, *optional*, defaults to 151656) : The video token index to encode the video prompt. position_id_per_seconds (`int`, *optional*, defaults to 25) : The increment of position id per second. seconds_per_chunk (`int`, *optional*, defaults to 2) : The duration in seconds of the chunk of audio and video data. audio_start_token_id (`int`, *optional*, defaults to 151647) : The audio start token index to encode the audio prompt. audio_end_token_id (`int`, *optional*, defaults to 151648) : The audio end token index to encode the audio prompt. user_token_id (`int, *optional*, defaults to 872) : The user token index to encode the user token. initializer_range (`float`, *optional*, defaults to 0.02) : The standard deviation of the truncated_normal_initializer for initializing all weight matrices. tie_word_embeddings (`bool`, *optional*, defaults to `False`) : Whether the model's input and output word embeddings should be tied. ## Qwen2_5OmniThinkerForConditionalGeneration[[transformers.Qwen2_5OmniThinkerForConditionalGeneration]] #### transformers.Qwen2_5OmniThinkerForConditionalGeneration[[transformers.Qwen2_5OmniThinkerForConditionalGeneration]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L1748) The Qwen2.5OmniThinker model which consists of a audio backbone and a language model. This model inherits from [PreTrainedModel](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. forwardtransformers.Qwen2_5OmniThinkerForConditionalGeneration.forwardhttps://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L1894[{"name": "input_ids", "val": ": torch.LongTensor | None = None"}, {"name": "input_features", "val": ": torch.FloatTensor | None = None"}, {"name": "pixel_values", "val": ": torch.FloatTensor | None = None"}, {"name": "pixel_values_videos", "val": ": torch.FloatTensor | None = None"}, {"name": "image_grid_thw", "val": ": torch.LongTensor | None = None"}, {"name": "video_grid_thw", "val": ": torch.LongTensor | None = None"}, {"name": "attention_mask", "val": ": torch.Tensor | None = None"}, {"name": "feature_attention_mask", "val": ": torch.Tensor | None = None"}, {"name": "audio_feature_lengths", "val": ": torch.LongTensor | None = None"}, {"name": "position_ids", "val": ": torch.LongTensor | None = None"}, {"name": "past_key_values", "val": ": transformers.cache_utils.Cache | None = None"}, {"name": "inputs_embeds", "val": ": torch.FloatTensor | None = None"}, {"name": "rope_deltas", "val": ": torch.LongTensor | None = None"}, {"name": "labels", "val": ": torch.LongTensor | None = None"}, {"name": "use_cache", "val": ": bool | None = None"}, {"name": "output_attentions", "val": ": bool | None = None"}, {"name": "output_hidden_states", "val": ": bool | None = None"}, {"name": "return_dict", "val": ": bool | None = None"}, {"name": "use_audio_in_video", "val": ": bool | None = None"}, {"name": "cache_position", "val": ": torch.LongTensor | None = None"}, {"name": "video_second_per_grid", "val": ": torch.LongTensor | None = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.utils.generic.TransformersKwargs]"}]- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. Indices can be obtained using [AutoTokenizer](/docs/transformers/v5.3.0/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.__call__()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. [What are input IDs?](../glossary#input-ids) - **input_features** (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_dim)`, *optional*) -- The tensors corresponding to the input audio features. Audio features can be obtained using [WhisperFeatureExtractor](/docs/transformers/v5.3.0/en/model_doc/whisper#transformers.WhisperFeatureExtractor). See [WhisperFeatureExtractor.__call__()](/docs/transformers/v5.3.0/en/model_doc/whisper#transformers.WhisperFeatureExtractor.__call__) for details ([Qwen2_5OmniProcessor](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniProcessor) uses [WhisperFeatureExtractor](/docs/transformers/v5.3.0/en/model_doc/whisper#transformers.WhisperFeatureExtractor) for processing audios). - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- The tensors corresponding to the input images. Pixel values can be obtained using [Qwen2VLImageProcessorFast](/docs/transformers/v5.3.0/en/model_doc/qwen2_vl#transformers.Qwen2VLImageProcessorFast). See [Qwen2VLImageProcessorFast.__call__()](/docs/transformers/v5.3.0/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([Qwen2_5OmniProcessor](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniProcessor) uses [Qwen2VLImageProcessorFast](/docs/transformers/v5.3.0/en/model_doc/qwen2_vl#transformers.Qwen2VLImageProcessorFast) for processing images). - **pixel_values_videos** (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, frame_size, frame_size)`, *optional*) -- The tensors corresponding to the input video. Pixel values for videos can be obtained using [Qwen2VLVideoProcessor](/docs/transformers/v5.3.0/en/model_doc/qwen2_vl#transformers.Qwen2VLVideoProcessor). See [Qwen2VLVideoProcessor.__call__()](/docs/transformers/v5.3.0/en/model_doc/pe_video#transformers.PeVideoVideoProcessor.__call__) for details ([Qwen2_5OmniProcessor](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniProcessor) uses [Qwen2VLVideoProcessor](/docs/transformers/v5.3.0/en/model_doc/qwen2_vl#transformers.Qwen2VLVideoProcessor) for processing videos). - **image_grid_thw** (`torch.LongTensor` of shape `(num_images, 3)`, *optional*) -- The temporal, height and width of feature shape of each image in LLM. - **video_grid_thw** (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*) -- The temporal, height and width of feature shape of each video in LLM. - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) - **feature_attention_mask** (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`, *optional*) -- Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. - **audio_feature_lengths** (`torch.LongTensor` of shape `(num_audios)`, *optional*) -- The length of feature shape of each audio in LLM. - **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) - **past_key_values** (`~cache_utils.Cache`, *optional*) -- Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Only [Cache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). If no `past_key_values` are passed, [DynamicCache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default. The model will output the same cache format that is fed as input. If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. - **rope_deltas** (`torch.LongTensor` of shape `(batch_size, )`, *optional*) -- The rope index difference between sequence length and multimodal rope. - **labels** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. - **use_cache** (`bool`, *optional*) -- If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). - **output_attentions** (`bool`, *optional*) -- Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. - **output_hidden_states** (`bool`, *optional*) -- Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. - **return_dict** (`bool`, *optional*) -- Whether or not to return a [ModelOutput](/docs/transformers/v5.3.0/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. - **use_audio_in_video** (`bool`, *optional*) -- Whether or not use audio track in video, should same as the parameter in `process_audio_info`. - **cache_position** (`torch.LongTensor` of shape `(sequence_length)`, *optional*) -- Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. - **video_second_per_grid** (`torch.LongTensor` of shape `(num_videos)`, *optional*) -- Number of seconds per grid for each video, used for temporal feature mapping.0`Qwen2_5OmniThinkerCausalLMOutputWithPast` or `tuple(torch.FloatTensor)`A `Qwen2_5OmniThinkerCausalLMOutputWithPast` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. The [Qwen2_5OmniThinkerForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniThinkerForConditionalGeneration) forward method, overrides the `__call__` special method. Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them. - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Language modeling loss (for next-token prediction). - **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`, *optional*) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). - **past_key_values** (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`) -- It is a [Cache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.Cache) instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. - **hidden_states** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. - **rope_deltas** (`torch.LongTensor` of shape `(batch_size, )`, *optional*) -- The rope index difference between sequence length and multimodal rope. Example: ```python >>> from io import BytesIO >>> from urllib.request import urlopen >>> import librosa >>> from qwen_vl_utils import process_vision_info >>> from transformers import Qwen2_5OmniProcessor, Qwen2_5OmniThinkerForConditionalGeneration >>> thinker = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained("Qwen/Qwen2.5-Omni-7B") >>> processor = Qwen2_5OmniProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B") >>> conversations = [ >>> {'role': 'system', 'content': 'You are a helpful voice chat bot, and please respond to me in a casual conversation manner using random voice.'}, >>> {"role": "user", "content": [ >>> {"type": "image", "image_url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, >>> {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/glass-breaking-151256.mp3"}, >>> ]}, >>> ] >>> text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False) >>> audios = [ librosa.load(BytesIO(urlopen( conversations[1]['content'][1]['audio_url'] ).read()), sr=self.processor.feature_extractor.sampling_rate) ] >>> images, videos = process_vision_info(conversations) >>> inputs = processor(text=text, audio=audios, images=images, videos=videos, return_tensors="pt", padding=True) >>> # Generate >>> inputs['use_audio_in_video'] = `True` or `False` >>> generation = thinker.generate(**inputs, max_new_tokens=2048) >>> generate_ids = generation[:, inputs.input_ids.size(1):] >>> response = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] ``` **Parameters:** config ([Qwen2_5OmniThinkerConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniThinkerConfig)) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. **Returns:** ``Qwen2_5OmniThinkerCausalLMOutputWithPast` or `tuple(torch.FloatTensor)`` A `Qwen2_5OmniThinkerCausalLMOutputWithPast` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. #### get_audio_features[[transformers.Qwen2_5OmniThinkerForConditionalGeneration.get_audio_features]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L1805) - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining. - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. **Parameters:** input_features (`torch.FloatTensor`) : The tensors corresponding to the input audios. feature_attention_mask (`torch.LongTensor`, *optional*) : Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: audio_feature_lengths (`torch.LongTensor` of shape `(num_audios)`, *optional*) : The length of feature shape of each audio in LLM. **Returns:** `[BaseModelOutputWithPooling](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`` A [BaseModelOutputWithPooling](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. #### get_image_features[[transformers.Qwen2_5OmniThinkerForConditionalGeneration.get_image_features]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L1788) - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining. - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. **Parameters:** pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`) : The tensors corresponding to the input images. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*) : The temporal, height and width of feature shape of each image in LLM. **Returns:** `[BaseModelOutputWithPooling](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`` A [BaseModelOutputWithPooling](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. #### get_placeholder_mask[[transformers.Qwen2_5OmniThinkerForConditionalGeneration.get_placeholder_mask]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L1844) Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. #### get_video_features[[transformers.Qwen2_5OmniThinkerForConditionalGeneration.get_video_features]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L1771) - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining. - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. **Parameters:** pixel_values_videos (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`) : The tensors corresponding to the input videos. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*) : The temporal, height and width of feature shape of each video in LLM. **Returns:** `[BaseModelOutputWithPooling](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`` A [BaseModelOutputWithPooling](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. ## Qwen2_5OmniThinkerTextModel[[transformers.Qwen2_5OmniThinkerTextModel]] #### transformers.Qwen2_5OmniThinkerTextModel[[transformers.Qwen2_5OmniThinkerTextModel]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L1586) The bare Qwen2 5 Omni Text Model outputting raw hidden-states without any specific head on to. This model inherits from [PreTrainedModel](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. forwardtransformers.Qwen2_5OmniThinkerTextModel.forwardhttps://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L1609[{"name": "input_ids", "val": ": torch.LongTensor | None = None"}, {"name": "attention_mask", "val": ": torch.Tensor | None = None"}, {"name": "position_ids", "val": ": torch.LongTensor | None = None"}, {"name": "past_key_values", "val": ": transformers.cache_utils.Cache | None = None"}, {"name": "inputs_embeds", "val": ": torch.FloatTensor | None = None"}, {"name": "use_cache", "val": ": bool | None = None"}, {"name": "output_attentions", "val": ": bool | None = None"}, {"name": "output_hidden_states", "val": ": bool | None = None"}, {"name": "return_dict", "val": ": bool | None = None"}, {"name": "cache_position", "val": ": torch.LongTensor | None = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.modeling_flash_attention_utils.FlashAttentionKwargs]"}]- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. Indices can be obtained using [AutoTokenizer](/docs/transformers/v5.3.0/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.__call__()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. [What are input IDs?](../glossary#input-ids) - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) - **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) - **past_key_values** (`~cache_utils.Cache`, *optional*) -- Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Only [Cache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). If no `past_key_values` are passed, [DynamicCache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default. The model will output the same cache format that is fed as input. If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. - **use_cache** (`bool`, *optional*) -- If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). - **output_attentions** (`bool`, *optional*) -- Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. - **output_hidden_states** (`bool`, *optional*) -- Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. - **return_dict** (`bool`, *optional*) -- Whether or not to return a [ModelOutput](/docs/transformers/v5.3.0/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. - **cache_position** (`torch.LongTensor` of shape `(sequence_length)`, *optional*) -- Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length.0[BaseModelOutputWithPast](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or `tuple(torch.FloatTensor)`A [BaseModelOutputWithPast](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. The [Qwen2_5OmniThinkerTextModel](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniThinkerTextModel) forward method, overrides the `__call__` special method. Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them. - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. - **past_key_values** (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`) -- It is a [Cache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.Cache) instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. **Parameters:** config (`Qwen2_5OmniTextConfig`) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. **Returns:** `[BaseModelOutputWithPast](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or `tuple(torch.FloatTensor)`` A [BaseModelOutputWithPast](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. ## Qwen2_5OmniTalkerConfig[[transformers.Qwen2_5OmniTalkerConfig]] #### transformers.Qwen2_5OmniTalkerConfig[[transformers.Qwen2_5OmniTalkerConfig]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/configuration_qwen2_5_omni.py#L502) This is the configuration class to store the configuration of a [Qwen2_5OmniTalkerForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniTalkerForConditionalGeneration). It is used to instantiate an Qwen2.5-Omni-Talker model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Qwen2.5-Omni-Thinker. e.g. [Qwen/Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) Configuration objects inherit from [PreTrainedConfig](/docs/transformers/v5.3.0/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the documentation from [PreTrainedConfig](/docs/transformers/v5.3.0/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. Example: ```python >>> from transformers import Qwen2_5OmniTalkerForConditionalGeneration, Qwen2_5OmniThinkerConfig, Qwen2_5OmniAudioEncoderConfig, Qwen2_5OmniVisionEncoderConfig >>> # Initializing a Qwen2_5OmniAudioEncoder config >>> audio_config = Qwen2_5OmniAudioEncoderConfig() >>> # Initializing a Qwen2 config >>> text_config = Qwen2Config() >>> # Initializing a Qwen2_5Omni configuration >>> configuration = Qwen2_5OmniThinkerConfig(audio_config, text_config) >>> # Initializing a model from the qwen2-audio style configuration >>> model = Qwen2_5OmniTalkerForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` **Parameters:** audio_token_index (`int`, *optional*, defaults to 151646) : The audio token index to encode the audio prompt. image_token_index (`int`, *optional*, defaults to 151655) : The image token index to encode the image prompt. video_token_index (`int`, *optional*, defaults to 151656) : The video token index to encode the video prompt. vocab_size (`int`, *optional*, defaults to 8448) : Vocabulary size of the QwenOmni model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [Qwen2VLModel](/docs/transformers/v5.3.0/en/model_doc/qwen2_vl#transformers.Qwen2VLModel) tts_text_start_token_id (`int`, *optional*, defaults to 151860) : The tts text start token index to encode the start of tts text. tts_text_end_token_id (`int`, *optional*, defaults to 151861) : The tts text end token index to encode the end of tts text. tts_text_pad_token_id (`int`, *optional*, defaults to 151859) : The tts text pad token index to encode the pad of tts text. tts_codec_start_token_id (`int`, *optional*, defaults to 8293) : The tts codec start token index to encode the start of tts codec. tts_codec_end_token_id (`int`, *optional*, defaults to 8294) : The tts codec end token index to encode the end of tts codec. tts_codec_pad_token_id (`int`, *optional*, defaults to 8292) : The tts codec pad token index to encode the pad of tts codec. tts_codec_mask_token_id (`int`, *optional*, defaults to 8296) : The tts codec mask token index to encode the mask of tts codec. vision_start_token_id (`int`, *optional*, defaults to 151652) : The tts vision start token index to encode the start of vision. vision_end_token_id (`int`, *optional*, defaults to 151653) : The tts vision end token index to encode the end of vision. embedding_size (`int`, *optional*, defaults to 3584) : Dimension of the embedding representations. hidden_size (`int`, *optional*, defaults to 3584) : Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 18944) : Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 28) : Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 28) : Number of attention heads for each attention layer in the Transformer encoder. num_key_value_heads (`int`, *optional*, defaults to 4) : This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `32`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`) : The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 32768) : The maximum sequence length that this model might ever be used with. rms_norm_eps (`float`, *optional*, defaults to 1e-06) : The epsilon used by the rms normalization layers. head_dim (`int`, *optional*, defaults to 128) : The dimension of each attention head. use_cache (`bool`, *optional*, defaults to `True`) : Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. tie_word_embeddings (`bool`, *optional*, defaults to `False`) : Whether the model's input and output word embeddings should be tied. use_sliding_window (`bool`, *optional*, defaults to `False`) : Whether to use sliding window attention. sliding_window (`int`, *optional*, defaults to 32768) : Sliding window attention (SWA) window size. If not specified, will default to `4096`. max_window_layers (`int`, *optional*, defaults to 28) : The number of layers using full attention. The first `max_window_layers` layers will use full attention, while any additional layer afterwards will use SWA (Sliding Window Attention). attention_dropout (`float`, *optional*, defaults to 0.0) : The dropout ratio for the attention probabilities. rope_parameters (`RopeParameters`, *optional*) : Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. position_id_per_seconds (`int`, *optional*, defaults to 25) : The increment of position id per second. seconds_per_chunk (`int`, *optional*, defaults to 2) : The duration in seconds of the chunk of audio and video data. audio_start_token_id (`int`, *optional*, defaults to 151647) : The audio start token index to encode the audio prompt. audio_end_token_id (`int`, *optional*, defaults to 151648) : The audio end token index to encode the audio prompt. initializer_range (`float`, *optional*, defaults to 0.02) : The standard deviation of the truncated_normal_initializer for initializing all weight matrices. spatial_merge_size (`int`, *optional*, defaults to 2) : The size used for merging spatial dimensions. layer_types (`list`, *optional*) : Attention pattern for each layer. pad_token_id (`int`, *optional*) : The id of the padding token. ## Qwen2_5OmniTalkerForConditionalGeneration[[transformers.Qwen2_5OmniTalkerForConditionalGeneration]] #### transformers.Qwen2_5OmniTalkerForConditionalGeneration[[transformers.Qwen2_5OmniTalkerForConditionalGeneration]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L2323) forwardtransformers.Qwen2_5OmniTalkerForConditionalGeneration.forwardhttps://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L2357[{"name": "input_ids", "val": ": torch.LongTensor | None = None"}, {"name": "attention_mask", "val": ": torch.Tensor | None = None"}, {"name": "position_ids", "val": ": torch.LongTensor | None = None"}, {"name": "past_key_values", "val": ": transformers.cache_utils.Cache | None = None"}, {"name": "thinker_reply_part", "val": ": torch.FloatTensor | None = None"}, {"name": "inputs_embeds", "val": ": torch.FloatTensor | None = None"}, {"name": "rope_deltas", "val": ": torch.LongTensor | None = None"}, {"name": "use_cache", "val": ": bool | None = None"}, {"name": "cache_position", "val": ": torch.LongTensor | None = None"}, {"name": "input_text_ids", "val": ": torch.LongTensor | None = None"}, {"name": "image_grid_thw", "val": ": torch.LongTensor | None = None"}, {"name": "video_grid_thw", "val": ": torch.LongTensor | None = None"}, {"name": "use_audio_in_video", "val": ": bool | None = None"}, {"name": "audio_feature_lengths", "val": ": torch.LongTensor | None = None"}, {"name": "video_second_per_grid", "val": ": torch.LongTensor | None = None"}, {"name": "output_attentions", "val": ": bool | None = None"}, {"name": "output_hidden_states", "val": ": bool | None = None"}, {"name": "return_dict", "val": ": bool | None = None"}, {"name": "**kwargs", "val": ""}]- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. Indices can be obtained using [AutoTokenizer](/docs/transformers/v5.3.0/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.__call__()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. [What are input IDs?](../glossary#input-ids) - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) - **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) - **past_key_values** (`~cache_utils.Cache`, *optional*) -- Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Only [Cache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). If no `past_key_values` are passed, [DynamicCache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default. The model will output the same cache format that is fed as input. If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. - **thinker_reply_part** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- Hidden states from the thinker model's output that represent the text reply part to be processed. - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. - **rope_deltas** (`torch.LongTensor` of shape `(batch_size, )`, *optional*) -- The rope index difference between sequence length and multimodal rope. - **use_cache** (`bool`, *optional*) -- If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). - **cache_position** (`torch.LongTensor` of shape `(sequence_length)`, *optional*) -- Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. - **input_text_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Input token IDs for text-only content, used for position calculation in multimodal contexts. - **image_grid_thw** (`torch.LongTensor` of shape `(num_images, 3)`, *optional*) -- The temporal, height and width of feature shape of each image in LLM. - **video_grid_thw** (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*) -- The temporal, height and width of feature shape of each video in LLM. - **use_audio_in_video** (`bool`, *optional*) -- Whether or not use audio track in video, should same as the parameter in `process_audio_info`. - **audio_feature_lengths** (`torch.LongTensor` of shape `(num_audios)`, *optional*) -- The length of feature shape of each audio in LLM. - **video_second_per_grid** (`torch.LongTensor` of shape `(num_videos)`, *optional*) -- Number of seconds per grid for each video, used for temporal feature mapping. - **output_attentions** (`bool`, *optional*) -- Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. - **output_hidden_states** (`bool`, *optional*) -- Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. - **return_dict** (`bool`, *optional*) -- Whether or not to return a [ModelOutput](/docs/transformers/v5.3.0/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.0`Qwen2_5OmniTalkerCausalLMOutputWithPast` or `tuple(torch.FloatTensor)`A `Qwen2_5OmniTalkerCausalLMOutputWithPast` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. The [Qwen2_5OmniTalkerForConditionalGeneration](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniTalkerForConditionalGeneration) forward method, overrides the `__call__` special method. Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them. - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Language modeling loss (for next-token prediction). - **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). - **past_key_values** (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`) -- It is a [Cache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.Cache) instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. - **hidden_states** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. - **rope_deltas** (`torch.LongTensor` of shape `(batch_size, )`, *optional*) -- The rope index difference between sequence length and multimodal rope. - **thinker_reply_part** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Hidden states from the thinker model that are used as input for the talker model. These represent the encoded response that the talker model will use to generate speech tokens. Example: ```python >>> from io import BytesIO >>> from urllib.request import urlopen >>> import librosa >>> from transformers import AutoProcessor, Qwen2_5OmniTalkerForConditionalGeneration >>> model = Qwen2_5OmniTalkerForConditionalGeneration.from_pretrained("Qwen/Qwen2-Audio-7B") >>> processor = AutoProcessor.from_pretrained("Qwen/Qwen2-Audio-7B") >>> prompt = "Generate the caption in English:" >>> url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/glass-breaking-151256.mp3" >>> audio, _ = librosa.load(BytesIO(urlopen(url).read()), sr=self.processor.feature_extractor.sampling_rate) >>> inputs = processor(text=prompt, audio=audio, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_length=30) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Generate the caption in English: Glass is breaking." ``` **Parameters:** input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) : Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. Indices can be obtained using [AutoTokenizer](/docs/transformers/v5.3.0/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.__call__()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) : Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) : Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`~cache_utils.Cache`, *optional*) : Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Only [Cache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). If no `past_key_values` are passed, [DynamicCache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default. The model will output the same cache format that is fed as input. If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. thinker_reply_part (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) : Hidden states from the thinker model's output that represent the text reply part to be processed. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) : Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*) : The rope index difference between sequence length and multimodal rope. use_cache (`bool`, *optional*) : If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*) : Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. input_text_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) : Input token IDs for text-only content, used for position calculation in multimodal contexts. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*) : The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*) : The temporal, height and width of feature shape of each video in LLM. use_audio_in_video (`bool`, *optional*) : Whether or not use audio track in video, should same as the parameter in `process_audio_info`. audio_feature_lengths (`torch.LongTensor` of shape `(num_audios)`, *optional*) : The length of feature shape of each audio in LLM. video_second_per_grid (`torch.LongTensor` of shape `(num_videos)`, *optional*) : Number of seconds per grid for each video, used for temporal feature mapping. output_attentions (`bool`, *optional*) : Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*) : Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*) : Whether or not to return a [ModelOutput](/docs/transformers/v5.3.0/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. **Returns:** ``Qwen2_5OmniTalkerCausalLMOutputWithPast` or `tuple(torch.FloatTensor)`` A `Qwen2_5OmniTalkerCausalLMOutputWithPast` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. ## Qwen2_5OmniTalkerModel[[transformers.Qwen2_5OmniTalkerModel]] #### transformers.Qwen2_5OmniTalkerModel[[transformers.Qwen2_5OmniTalkerModel]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L2166) The bare Qwen2 5 Omni Model outputting raw hidden-states without any specific head on top. This model inherits from [PreTrainedModel](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. forwardtransformers.Qwen2_5OmniTalkerModel.forwardhttps://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L2189[{"name": "input_ids", "val": ": torch.LongTensor | None = None"}, {"name": "attention_mask", "val": ": torch.Tensor | None = None"}, {"name": "position_ids", "val": ": torch.LongTensor | None = None"}, {"name": "past_key_values", "val": ": transformers.cache_utils.Cache | None = None"}, {"name": "inputs_embeds", "val": ": torch.FloatTensor | None = None"}, {"name": "use_cache", "val": ": bool | None = None"}, {"name": "output_attentions", "val": ": bool | None = None"}, {"name": "output_hidden_states", "val": ": bool | None = None"}, {"name": "return_dict", "val": ": bool | None = None"}, {"name": "cache_position", "val": ": torch.LongTensor | None = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.modeling_flash_attention_utils.FlashAttentionKwargs]"}]- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. Indices can be obtained using [AutoTokenizer](/docs/transformers/v5.3.0/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and [PreTrainedTokenizer.__call__()](/docs/transformers/v5.3.0/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. [What are input IDs?](../glossary#input-ids) - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) - **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) - **past_key_values** (`~cache_utils.Cache`, *optional*) -- Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Only [Cache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). If no `past_key_values` are passed, [DynamicCache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default. The model will output the same cache format that is fed as input. If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. - **use_cache** (`bool`, *optional*) -- If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). - **output_attentions** (`bool`, *optional*) -- Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. - **output_hidden_states** (`bool`, *optional*) -- Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. - **return_dict** (`bool`, *optional*) -- Whether or not to return a [ModelOutput](/docs/transformers/v5.3.0/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. - **cache_position** (`torch.LongTensor` of shape `(sequence_length)`, *optional*) -- Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length.0[BaseModelOutputWithPast](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or `tuple(torch.FloatTensor)`A [BaseModelOutputWithPast](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. The [Qwen2_5OmniTalkerModel](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniTalkerModel) forward method, overrides the `__call__` special method. Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them. - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. - **past_key_values** (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`) -- It is a [Cache](/docs/transformers/v5.3.0/en/internal/generation_utils#transformers.Cache) instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. **Parameters:** config ([Qwen2_5OmniTalkerConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniTalkerConfig)) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. **Returns:** `[BaseModelOutputWithPast](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or `tuple(torch.FloatTensor)`` A [BaseModelOutputWithPast](/docs/transformers/v5.3.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration ([Qwen2_5OmniConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniConfig)) and inputs. ## Qwen2_5OmniToken2WavConfig[[transformers.Qwen2_5OmniToken2WavConfig]] #### transformers.Qwen2_5OmniToken2WavConfig[[transformers.Qwen2_5OmniToken2WavConfig]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/configuration_qwen2_5_omni.py#L876) This is the configuration class to store the configuration of a [Qwen2_5OmniToken2WavModel](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniToken2WavModel). It is used to instantiate the Qwen2.5-Omni-Token2Wav model which combines a Diffusion Transformer (DiT) for mel-spectrogram generation with a BigVGAN model for waveform synthesis. The configuration contains sub-configurations for both components. Configuration objects inherit from [PreTrainedConfig](/docs/transformers/v5.3.0/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the documentation from [PreTrainedConfig](/docs/transformers/v5.3.0/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. Example: ```python >>> from transformers import Qwen2_5OmniToken2WavModel, DiT_Args, BigVGAN_Args >>> # Initialize DiT configuration >>> dit_config = DiT_Args( ... dim=1024, ... depth=22, ... heads=16, ... ff_mult=2 ... ) >>> # Initialize BigVGAN configuration >>> bigvgan_config = BigVGAN_Args( ... mel_dim=80, ... upsample_rates=[5,3,2,2,2,2] ... ) >>> # Initialize main configuration >>> config = Qwen2_5OmniToken2WavConfig(dit_config, bigvgan_config) >>> # Initialize model with config >>> model = Qwen2_5OmniToken2Wav(config) >>> # Accessing the model configuration >>> configuration = model.config ``` **Parameters:** dit_config (`DiT_Args`, *optional*) : Configuration class for the Diffusion Transformer (DiT) module responsible for generating mel-spectrograms. bigvgan_config (`BigVGAN_Args`, *optional*) : Configuration class for the BigVGAN module responsible for converting mel-spectrograms to waveforms. ## Qwen2_5OmniToken2WavModel[[transformers.Qwen2_5OmniToken2WavModel]] #### transformers.Qwen2_5OmniToken2WavModel[[transformers.Qwen2_5OmniToken2WavModel]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L3730) The full Qwen2.5Omni Token2Wav model. Consists a DiT model take speech tokens as input and predict mel spectrogram and a BigVGAN vocoder take mel spectrogram as input and predict waveform. This model inherits from [PreTrainedModel](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. forwardtransformers.Qwen2_5OmniToken2WavModel.forwardhttps://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L3759[{"name": "code", "val": ""}, {"name": "conditioning", "val": ""}, {"name": "reference_mel", "val": ""}, {"name": "num_steps", "val": " = 10"}, {"name": "guidance_scale", "val": " = 0.5"}, {"name": "sway_coefficient", "val": " = -1.0"}, {"name": "**kwargs", "val": ""}] Generates a waveform from input code and conditioning parameters. **Parameters:** config ([Qwen2_5OmniToken2WavConfig](/docs/transformers/v5.3.0/en/model_doc/qwen2_5_omni#transformers.Qwen2_5OmniToken2WavConfig)) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. ## Qwen2_5OmniToken2WavDiTModel[[transformers.Qwen2_5OmniToken2WavDiTModel]] #### transformers.Qwen2_5OmniToken2WavDiTModel[[transformers.Qwen2_5OmniToken2WavDiTModel]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L3559) The full Qwen2.5Omni Token2WavDiT model. Which take speech tokens as input and predict mel spectrogram. This model inherits from [PreTrainedModel](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. **Parameters:** config (`Qwen2_5OmniDiTConfig`) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. ## Qwen2_5OmniToken2WavBigVGANModel[[transformers.Qwen2_5OmniToken2WavBigVGANModel]] #### transformers.Qwen2_5OmniToken2WavBigVGANModel[[transformers.Qwen2_5OmniToken2WavBigVGANModel]] [Source](https://github.com/huggingface/transformers/blob/v5.3.0/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L3421) The full Qwen2.5Omni Token2WavBigVGAN model. Which take mel spectrogram as input and predict waveform. This model inherits from [PreTrainedModel](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. **Parameters:** config (`Qwen2_5OmniBigVGANConfig`) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from_pretrained()](/docs/transformers/v5.3.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.