Multimodal AI
Multimodal AI is AI that can work with more than one kind of input or output: text, images, audio, video, or files like PDFs and spreadsheets. You can give a multimodal model a photo of a whiteboard and ask for the action items, or a screenshot of an error and ask what went wrong.
Why it matters
A lot of work information isn’t plain text. Scanned contracts, charts in slide decks, screenshots of bugs, recorded calls and diagrams all used to need a separate step, such as OCR, transcription or someone describing the picture, before AI could use them. A multimodal model reads them directly and keeps the layout, which matters for tables and forms.
It matters for agents too. An agent that can see a screen can check whether a web page rendered properly, read a chart on a dashboard or operate an app that has no API.
Where it goes wrong
Models are better at describing what’s in an image than at reading precise detail. Small numbers in a dense table, handwriting and exact positions are common failure points, and a model will often state a misread number with complete confidence. When the exact figure matters, check it against the source, or extract the data into a structured form and review that.
Images, audio and video are counted in tokens as well, so a long recording or a folder of screenshots can cost far more than the same information written as text.
Input is not the same as output
Many models take images and audio in but only write text out. Generating images, speech or video is often done by separate models, even when a single app hides that from you. When a product says it’s multimodal, it’s worth checking which direction it means.