{
  "id": 5157947,
  "title": "How Does AI Know What's Inside an Image? Meet Vision Language Models",
  "url": "https://urgent.news/2026/09/02/how-does-ai-know-whats-inside-an-image-meet-vision-language-models",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-02T19:29:03.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/rijultp/how-does-ai-know-whats-inside-an-image-meet-vision-language-models-2m8o"
  },
  "original_language": "en",
  "account": "Rijul is constructing LiveReview, an AI code review tool designed for critical business systems. If you have the chance to engage with AI models that process images, you might be curious about how these models function and the processes occurring behind the scenes. This type of model is known as a Vision-Language Model (VLM). A straightforward way to conceptualize a VLM is to imagine it as an LLM (Language Model) modified to process both visual and textual data. While VLMs can differ in their architecture, they generally consist of a vision encoder, a connector or projector, and a language model.\n\nWhen a VLM \"sees\" an image, it undergoes a process akin to this: Image ↓ Vision Encoder ↓ Visual Representations ↓ Vision-Language Connector ↓ Language Model ↓ Text Response. The first essential component is the Vision Encoder, which processes the image and transforms the visual data into numerical representations comprehensible to the rest of the model. Depending on the specific vision architecture, the image may be segmented into smaller patches. Each patch is then analyzed by the vision encoder, which generates visual representations capturing visual patterns and context, such as shapes, objects, colors, text, spatial relationships, and more. The manner in which this occurs varies based on the vision encoder architecture, resulting in a representation of the image that can be passed to the language model.\n\nAfter obtaining visual representations, we delve into two crucial capabilities: perception and grounding. Perception involves understanding the visual content, not merely identifying objects but also comprehending relationships between objects, actions, scenes, and the overall visual context. For instance, given an image of a person riding a bicycle, the model may understand that the person is riding the bicycle. Grounding, on the other hand, connects language to specific parts of the visual input. For example, if we ask, \"Where is the dog?\" the model could associate the term \"dog\" with a specific region of the image. Technically, this can be symbolized as: { object : dog, location : center of the image }. In more advanced systems, grounding may involve associating language with regions, objects, points, or other parts of an image, often represented through bounding boxes, but not exclusively so. This association becomes particularly valuable when we want an AI system to not only identify an object but also understand its context within the image.\n\nWhile the term \"multimodal LLM\" is sometimes used interchangeably with VLM, the exact terminology varies depending on the model and its creators' descriptions. However, VLMs can be considered a subset within the broader field of multimodal AI. In essence, a VLM processes images and their associated language, enabling AI systems to interact with visual information alongside textual data. This capability allows for a more comprehensive understanding of visual content, such as UI designs, diagrams, screenshots, charts, or photographs, without the need for exhaustive textual descriptions. By providing the model with an image, VLMs empower AI systems to comprehend visual information seamlessly alongside language, facilitating interactions with visual content that were previously limited to textual descriptions.",
  "summary": "Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. Do you interact with AI models where you put in an image and then it processes it? Do you wonder what these models are and what is happening behind the scenes?…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}