Multimodal Models

Introduction to Multimodal Models

A multimodal model works with more than one kind of information, such as text, images, audio, or video. It must represent each modality and learn how information in one modality relates to information in another.

Consider a model that receives a photograph and answers a written question about it. The image begins as pixels and the question begins as text tokens, so the first problem is representation: turn each kind of input into features a model can process.

The model must connect the words “red bicycle” with the corresponding region of the photograph. This is alignment: identify which parts of one modality correspond to parts of another.

To answer the question, information from the image and text must meet. This is fusion. Some systems combine features early, some combine later decisions, and some connect the modalities at several layers.

If the task asks for a caption or speech transcript, the model produces an object in another modality. This is translation, such as image to text or speech to text.

Finally, paired modalities can help one another during training. Co-learning describes that transfer: for example, image–text pairs can improve a visual representation that is later used without text.