Why AI needs world models to move beyond the screen

Generative AI can create remarkably convincing content. But for AI to operate reliably in the physical world, it must also predict what happens next and how its own actions could change the environment.

Context

In recent years, generative AI has advanced at a remarkable pace.We see it everywhere: Computer Vision has been revolutionized by image generators like DALL-E and Midjourney; enterprise systems are being transformed by Retrieval-Augmented Generation (RAG) and autonomous agents; and Large Language Models (LLMs) like ChatGPT and Gemini have become industry standards.

Yet, the physical world presents a fundamentally different challenge. 

Imagine asking a state-of-the-art robot that can interpret vision and language to navigate a crowded, chaotic kitchen and fetch a wine glass without breaking it. Even advanced vision-language-action systems can struggle when conditions differ from their training data or when a task requires precise, multi-step physical reasoning. [1] Despite the groundbreaking innovations of late, current AI models still face important limitations when applied to complex, real-world physical scenarios.

Why?  Because generating a plausible response is not the same as predicting the consequences of an action.

Recognizing patterns Is not the same as modelling the world

Generative models are exceptional at discovering and replicating statistical correlations within data, but they struggle to generalize the underlying rules of reality. Part of this is because some elements of our world are purely stochastic, inherently random and unpredictable.

This does not mean these models have no useful representation of the world. It means that their training objectives are not always designed around the questions a physical system must answer:

• What is changing?
• What will probably happen next?
• What will happen if I take a particular action?
• How certain am I about that prediction?

Part of the difficulty is that the real world contains both predictable structure and stochastic or inherently uncertain events.

The Wolf-versus-Husky problem 

Imagine a dataset [2] where every picture of a Husky is taken in the snow, and every picture of a Labrador is taken in a green garden. A generative model will quickly converge on a shortcut: it will look at the background rather than the animal. If you place a Husky in a green garden, the model flags it as a Labrador.

This happens because the model lacks the abstraction required to truly understand context. Compounding this issue is the nature of the data itself. Generative AI usually treats data as discrete classes : LLMs predict the next word token from a fixed vocabulary. But the real world is inherently continuous, fluid, and dynamic.

To reach the next frontier of automation, true autonomy, advanced robotics, and reliable self-driving cars, we need a new paradigm. We need World Models.

Why world models are emerging (and why LLMs need a new paradigm) 

LLMs are highly capable pattern-recognition and prediction systems. They are trained in a self-supervised manner in vast amounts of text to answer a deceptively simple question:, "What is the most likely token to follow this sequence given the context?" While this works remarkably well for language, it is not, on its own, sufficient  when an AI needs to understand physical constraints like gravity, momentum, friction, or fluid dynamics. You cannot simply "text-prompt" a vehicle safely through a hydroplaning event.

Many world models address this challenge by encoding sensory input—such as images, video, or robotic sensor data, into a latent space rather than attempting to predict every detail of the raw input. 

Think of  latent space as a highly optimized, mathematical "mental sandbox." The level of abstraction inside this sandbox is shaped by the task at hand.

Let's look at a practical example: a self-driving car.

If a vehicle is travelling at 80 km/h and its control system considers applying the accelerator, a model trained to generate future camera frames might attempt to predict the next image frame. This is a highly complex task. A pedestrian might enter the scene, or a cloud might shift in the sky. Some of these events are stochastic, meaning they are difficult or impossible to predict precisely.

A frame-generation model could be penalized during training for failing to predict where the cloud would move, even though that detail may have little relevance to the driving decision.

A well-designed world model handles this differently.Rather than reconstructing every visible detail, it can predict the next latent representation of the state. Its representation may place less emphasis on the shifting cloud or a stray leaf blowing across the camera. Instead, its “mental sandbox” can focus on the vehicle’s dynamics, the trajectory of the road, and relevant obstacles.

By  reducing the influence of task-irrelevant details, the model can create a more useful representation for predicting outcomes and evaluating possible actions.

World models vs. generative AI: what’s the difference?

To see how these paradigms diverge, we can compare them across three core pillars:

How world models actually work 

Modern world models, frequently built on Joint Embedding Predictive Architecture (JEPA) [3] frameworks, rely on three distinct, interconnected components to process reality:
‍

1. The Context Encoder

The system ingests raw sensory data, such as video feeds, LiDAR point clouds, or robotic joint angles, and passes the data through the context encoder. This component compresses the high-dimensional input into a more compact latent representation.

The objective is to preserve information that is useful for the task while reducing unnecessary detail. However, the model does not automatically distinguish useful information from noise; what it learns to preserve depends on its architecture, training data, and learning objective.

2. The Predictor

The predictor takes the current latent representation and estimates a target or future representation.

In an action-conditioned world model, it may also receive a proposed action. This action could be represented as a scalar or vector—for example, a command to turn a steering wheel five degrees to the left.

It then estimates how the represented state may change after that action. Critically, it can output a predicted latent vector rather than a raw image.

For a vehicle, the predictor might estimate how acceleration would affect factors such as speed, position, and proximity to nearby obstacles, without reconstructing every visual detail of the environment.

In other JEPA-style systems, the predictor may instead estimate the representation of a masked region of an image or a future segment of a video. The exact prediction target depends on the architecture and training objective.

3. The Target Encoder

The target encoder is primarily used during training.

It processes the target observation—such as a masked region of an image or an actual future video frame—and encodes it into the same latent space as the context input.

The system then compares the predictor’s output with the target encoder’s representation. The difference between the two is used as a training signal to improve the model’s predictions.

In many JEPA implementations, the target encoder is not required during inference, although the precise configuration may vary between architectures.

4. The Downstream Component

Once the model has learned useful representations or predictive dynamics, it can be connected to a downstream, task-specific component. This is where prediction can inform a practical decision or action.

Whether the goal is classification, regression, or robotic motor control, this downstream component uses the learned representations to support task-specific outputs.

For a robot or autonomous vehicle, this may involve a planner that evaluates possible actions and a controller that executes the selected movement. The complete system may also require an objective function, uncertainty estimation, real-time sensor feedback, and explicit safety constraints.

Although a world model can learn useful approximations of environmental dynamics, it should not be described as a complete “physics engine,” nor does it guarantee highly accurate decisions in unfamiliar situations.

The road ahead: challenges and future 

While world models represent a promising step forward, the field still faces significant hurdles before these systems can be deployed reliably at scale.

The Threat of Representation Collapse

Training  representation-learning models can be difficult. One potential failure mode is representation collapse, in which the model finds a trivial way to minimize its training objective.

It may begin mapping entirely different real-world scenarios to the same constant or minimally informative latent vector. Although this may reduce the training loss, it prevents the model from learning meaningful distinctions between different inputs and can render its representations unusable.

Autoregressive Error Propagation

Some world models operate autoregressively, which means they feed their own previous predictions back into the model to simulate further into the future.

If a latent state contains even a small bias or error, that mistake can accumulate over multiple prediction steps, causing the model’s internal simulation to drift away from reality.

High Computational and Interaction Costs

Building a useful model of a complex environment can require vast amounts of data, repeated interaction with the environment, and significant computational infrastructure.

Physical interaction data can be especially difficult to collect because real-world experimentation may be slow, costly, or unsafe.

Conclusion

The future of artificial intelligence is not merely a collection of smarter chatbots, more sophisticated prompting techniques, or more creative image generators.

One of the field’s next major frontiers is the development of systems that can interact with the physical world safely and intelligently while adapting to changing and unexpected conditions.

By moving beyond the generation of convincing pixels and toward representations that capture relevant structure and dynamics, world models can help AI systems predict outcomes, evaluate possible actions, and operate more effectively in physical environments.

They are not a complete solution on their own, but they may become an important part of the broader engineering systems needed to move AI beyond the screen and into the real world.