TTT Models Might Be The Future of Generative AI

Following a period of transformer-based AI dominance, researchers are now exploring alternative architectures.

Transformer models are the foundation of OpenAI’s video-generation tool Sora and text-generating systems like Anthropic’s Claude, Google’s Gemini, and GPT-4o. However, these models encounter technical limitations, particularly in terms of computational requirements.

Transformers struggle to efficiently process and analyze large datasets when using standard hardware. This inefficiency is leading to significant and potentially unsustainable increases in power consumption as companies expand their infrastructure to meet transformers’ demands.

Researchers from Stanford, UC San Diego, UC Berkeley, and Meta introduced a new architecture called test-time training (TTT) this month. This team, which developed TTT over 18 months, asserts that TTT models can process substantially more data than transformers while consuming significantly less computational power.

The Hidden State in Transformers

A key element of transformer models is the “hidden state,” which can be described as an extensive list of data. As a transformer processes information, it adds entries to this hidden state to “remember” what it has just processed. For example, when a model is analyzing a book, the hidden state values might include representations of words or word fragments.

Yu Sun, a post-doctoral researcher at Stanford and co-contributor to the TTT research, explained the concept: “If you think of a transformer as an intelligent entity, then the lookup table — its hidden state — is the transformer’s brain. This specialized brain enables the well-known capabilities of transformers such as in-context learning.”

While the hidden state is a crucial factor in the power of transformers, it also imposes limitations. For a transformer to “say” even a single word about a book it has just processed, it would need to scan its entire lookup table – a task that requires as much computational power as rereading the entire book.

Sun and his team conceived the idea of substituting the hidden state with a machine learning model, essentially creating an AI within an AI structure.

While technically complex, the core concept is that the internal machine learning model of the TTT system, unlike a transformer’s lookup table, doesn’t continuously expand as it processes more data. Instead, it encodes processed data into representative variables called weights, which is the source of TTT models’ high performance. The size of a TTT model’s internal model remains constant regardless of the amount of data it processes.

Sun envisions future TTT models capable of efficiently processing billions of data points, including words, images, audio recordings, and videos. This far surpasses the capabilities of current models.

Sun explained, “Our system can say X words about a book without the computational complexity of rereading the book X times.” He added, “Large video models based on transformers, such as Sora, can only process 10 seconds of video, because they only have a lookup table ‘brain.’ Our eventual goal is to develop a system that can process a long video resembling the visual experience of a human life.”

Mistrust around the TTT models

The potential for TTT models to replace transformers remains uncertain at this early stage.

TTT models are not directly interchangeable with transformers. Moreover, the researchers have only developed two small models for study, making it challenging to compare TTT as a method with larger transformer implementations currently in use.

Mike Cook, a senior lecturer in the Department of Informatics at King’s College London, who was not involved in the TTT research, commented: “I think it’s a perfectly interesting innovation, and if the data backs up the claims that it provides efficiency gains then that’s great news, but I couldn’t tell you if it’s better than existing architectures or not.” He added, “An old professor of mine used to tell a joke when I was an undergrad: How do you solve any problem in computer science? Add another layer of abstraction. Adding a neural network inside a neural network definitely reminds me of that.”

Nevertheless, the increasing pace of research into alternatives to transformers indicates a growing acknowledgment of the need for a significant advancement in AI architecture.

This week, AI company Mistral introduced a model called Codestral Mamba, which is based on state space models (SSMs), another alternative to transformers. Like TTT models, SSMs appear to offer greater computational efficiency than transformers and can handle larger datasets.

AI21 Labs is also researching SSMs. Additionally, Cartesia, which developed some of the earliest SSMs and the original Mamba and Mamba-2 models (after which Codestral Mamba is named), is continuing work in this area.

If these initiatives prove successful, they could potentially make generative AI even more accessible and global than it is currently – a development that could have both positive and negative implications.