Emerging AI Architectures Beyond Transformers

𝗙𝗼𝗿𝗴𝗲𝘁 𝗚𝗲𝗺𝗶𝗻𝗶 𝟯 𝗳𝗼𝗿 𝗮 𝗺𝗼𝗺𝗲𝗻𝘁! Google quietly dropped a paper that might redefine the next decade of AI. While everyone was busy debating benchmarks, Nested Learning landed… and almost nobody noticed. Big mistake. This paper is probably one of the most groundbreaking theoretical advances from Google in years because it challenges a core assumption of deep learning: that stacking more layers and scaling larger models is the path to intelligence. Instead, the authors propose Nested Learning (NL), a new paradigm where neural networks are seen as systems of nested optimization problems, each with its own memory, update frequency, and context flow. And the implications are huge! 🔥 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 🔸It explains how in-context learning actually emerges in large models. 🔸Shows that optimizers like Adam or Momentum aren’t just math tricks. They are associative memory modules that literally compress gradients into internal knowledge. 🔸Provides a neuroscientifically-inspired view of how models could one day learn continuously, instead of freezing after pretraining. 🔸Introduces HOPE, a new architecture that outperforms Transformers and modern RNNs across multiple tasks, with dynamic self-modifying components and a continuum memory system. This paper suggests a world where models don’t just predict but they learn to learn, adapt, and modify themselves, even at test time. If you care about the future beyond scaling laws, this is a must-read. Link to the paper in the comments 👇 #AI #DeepLearning #LLM #Transformers #GenAI

The AI landscape has rapidly evolved beyond just large language models. Today’s systems rely on a wide range of foundational model types—each designed for specific modalities, tasks, and constraints. This visual covers 12 foundational AI models and their core workflows. This is intended for engineers, researchers, and builders who want a structured view of the ecosystem. Here’s a breakdown of what’s included: → LLM (Large Language Models) – GPT, LLaMA Trained using transformer architecture to generate coherent, human-like text. The workflow involves data collection, tokenization, pattern learning, fine-tuning, and deployment. → SLM (Small Language Models) – Phi, TinyLLaMA Lightweight and efficient for on-device or low-resource environments. Focuses on model compression, compact training, and benchmarking. → VLM (Vision-Language Models) – CLIP, Flamingo Learns joint understanding between images and text. Ideal for tasks like image captioning and visual QA. → MLLM (Multimodal Large Language Models) – Gemini Designed to process and align multiple modalities such as text, image, audio, and video. → LAM (Large Action Models) – RT-2, InstructDiffusion Generates sequences of executable actions using behavioral and reinforcement learning data. → LRM (Large Reasoning Models) – DeepSeek-R1 Structured for tool use, chain-of-thought reasoning, and test-time modularity in logic-heavy tasks. → MoE (Mixture of Experts) – Mixtral Activates a subset of specialized models per input to reduce computation cost and improve performance. → SSM (State Space Models) – Mamba, RetNet Efficient at long-context sequence modeling using dynamic systems and parallelism. → RNN (Recurrent Neural Networks) – LSTM, GRU Uses hidden states to process time-dependent data, maintaining memory across input sequences. → CNN (Convolutional Neural Networks) – EfficientNet Learns spatial patterns in image data via convolution layers, pooling, and hierarchical stacking. → SAM (Segment Anything Model) – Meta Segments objects from images based on prompts (text, points, or boxes), making it useful for dynamic image understanding. → LNN (Liquid Neural Networks) – LFMs Leverages differential equations to adapt in real-time, supporting applications in time-sensitive environments. This chart is designed to help you understand not just what these models are, but how they work under the hood. If you're working in AI,  this foundational understanding is crucial for making informed architectural decisions.

This could be a watershed moment for AI as the 'Deep Learning' era may be evolving into something new. For the last decade, the researchers and engineers have focused on enhancing AI by stacking more layers, which characterizes, the Deep Neural Networks. But a seminal new paper from Google Research for NeurIPS 2025 exposes a fundamental flaw in this approach, these models are static! Once trained, modern models are frozen in time, experiencing a form of 'anterograde amnesia' where they cannot learn from the present without forgetting the past. The paper titled 'Nested Learning: The Illusion of Deep Learning Architectures' by Ali Behrouz, Meisam Razaviyayn, Peiling Zhong, and Vahab Mirrokni proposes a paradigm shift:- Nested Learning (NL). Instead of merely stacking layers, NL reimagines models as a system of 'nested optimization problems', each operating at its own speed. Inspired by human brain waves, where high-frequency neurons manage the immediate present and low-frequency oscillations consolidate long-term memory, this approach unlocks the potential for true continual learning. Additionally, the authors introduced HOPE, a new architecture based on this paradigm. HOPE demonstrates superior performance, surpassing Transformers, RetNet, and Titans in language modeling and reasoning tasks. This could serve as the blueprint for the next generation of AI. Blog - https://lnkd.in/dQ_vermU Paper - https://lnkd.in/di8wnF7r #ArtificialIntelligence #MachineLearning #GoogleResearch #NestedLearning #ContinualLearning #AI

I spent a few hours this weekend deep in a fascinating paper that proposes a completely different architecture to transformers (which power all the chatbots and LLMs we use today). "The Dragon Hatchling" (yes, that's really the name - points for style) proposes something remarkable: a language model architecture that's both competitive with GPT-2 AND could plausibly run on biological neurons. 𝗧𝗵𝗲 𝗕𝗮𝗯𝘆 𝗗𝗿𝗮𝗴𝗼𝗻 𝗛𝗮𝘁𝗰𝗵𝗹𝗶𝗻𝗴 (𝗕𝗗𝗛) 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗶𝘀 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁: 1. 𝗟𝗼𝗰𝗮𝗹 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻𝘀 𝗰𝗿𝗲𝗮𝘁𝗲 𝗴𝗹𝗼𝗯𝗮𝗹 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 - Each of 1 million "neurons" follows simple Hebbian rules ("neurons that fire together, wire together"), yet collectively performs language reasoning at scale 2. 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝗺𝗲𝗺𝗼𝗿𝘆 𝗲𝗺𝗲𝗿𝗴𝗲𝘀 𝗻𝗮𝘁𝘂𝗿𝗮𝗹𝗹𝘆 - The model maintains context not in a cache, but in temporary strengthening of connections between concepts. Just like us. 3. 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗲𝗺𝗲𝗿𝗴𝗲𝘀, 𝗻𝗼𝘁 𝗶𝗺𝗽𝗼𝘀𝗲𝗱 - The network spontaneously organises into modules and clusters. No one designs this hierarchy - it discovers itself. 4. 𝗜𝗻𝘁𝗲𝗿𝗽𝗿𝗲𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗶𝘀𝗻'𝘁 𝘀𝗮𝗰𝗿𝗶𝗳𝗶𝗰𝗲𝗱 - You can literally watch which "synapses" strengthen when the model thinks about specific concepts. The thoughts are visible. At Brightbeam, we work across domains where we need to understand the nature of intelligence and when we can rely on digital intelligence (and which types) or whether we must have humans in the loop, and what we can expect. These domains need AI that's not just accurate but 𝘶𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥𝘢𝘣𝘭𝘦 𝘢𝘯𝘥 𝘵𝘳𝘶𝘴𝘵𝘸𝘰𝘳𝘵𝘩𝘺 - this architectural approach offers something different. The version in the paper performs as well as gpt-2, with similar number of parameters, but it points at how we could be building systems whose reasoning we can inspect, whose behaviour is grounded in principles we understand, and whose "thought processes" map to something interpretable. And it's more efficient. Understanding the 𝘯𝘢𝘵𝘶𝘳𝘦 of the intelligence we're building matters. Are we creating black boxes that happen to work? Or systems we genuinely understand? 𝗧𝗵𝗶𝘀 𝗶𝘀 𝗲𝗮𝗿𝗹𝘆 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵 - not battle-tested, not production-ready. But that's exactly when it's most interesting to engage with. The architectures we explore today shape the systems we build tomorrow. Once transformers were scaled up, magic things started to happen. Maybe something similar will happen with BDH. At Brightbeam, we're constantly exploring beyond the immediate utility of what current LLMs can deliver. Not because we dismiss that utility - we build with it daily - but because understanding the landscape of 𝘱𝘰𝘴𝘴𝘪𝘣𝘭𝘦 intelligences helps us choose the 𝘳𝘪𝘨𝘩𝘵 intelligence for each problem. It's definitely too early to claim success. But like world models, this is an area worth keeping an eye on.

What comes after Transformers? Neural Memory and Test-Time Training. Google Research presented 2 new papers during NeurIps with an architecture that actively learns and update their own parameters during inference, acting as a "long-term neural memory" rather than a static context window. Implementation: 1️⃣ Titans replaces the fixed-state memory of linear RNNs with a deep Multi-Layer Perceptron (MLP) memory module. 2️⃣ The model updates this memory at test-time by calculating a "surprise metric" based on the gradient of the input data. 3️⃣ MIRAS framework generalizes this by treating memory as an optimization problem with customizable loss functions and regularization. 4️⃣ Training is parallelized by chunking sequences, using linear operations within chunks and non-linear updates across chunks. 5️⃣ Models incorporate "Persistent Memory" (fixed learnable weights) alongside the dynamic "Contextual Memory" to store task-specific knowledge. Insights: - 💡 Attention mechanisms excellent at short-term memory but fails at efficient long-term storage due to quadratic costs. - 📈 Deep memory structures (MLPs) significantly outperform vector/matrix-based compression used in Mamba and other linear RNNs. - 🛠️ Memory updates are effective when driven by "surprise", high gradients indicate unexpected, memorable data. - 📚 Forgetting mechanisms in recurrent models are mathematically equivalent to retention regularization (weight decay). - 📉 Standard L2 (Mean Squared Error) objectives make memory sensitive to outliers; L1 or Huber loss provides better stability. - 🧠 Titans outperforms GPT-4 on "Needle in a Haystack" tasks with 2M+ token contexts despite having fewer parameters. - ⚡ Deep memory modules exhibit a trade-off where increased depth improves perplexity but slightly reduces training throughput. Titans and MIRAS show potential to replace, or at least augment, pure Transformer architectures. The hybrid approach (using attention for the immediate context and Neural Memory for the deep history) suggests the future might be a convergence of RNN efficiency and Transformer performance. Blog: https://lnkd.in/eXEEwd_t Titan Paper: https://lnkd.in/ejxAWBJD Miras Paper: https://lnkd.in/e27DWiyr

Transformers just got a major upgrade, and it doesn’t involve more compute. A new model architecture called FNet might be the key to making LLMs fast and efficient enough to run on devices like phones and wearables. The twist? It replaces self-attention entirely with something much simpler: the Fast Fourier Transform (FFT). Here’s the big idea: 🧠 Self-attention is powerful, but slow.   It scales quadratically with input length (O(N²)), which becomes a bottleneck for long sequences and edge deployment. ⚡ FNet replaces attention with a fixed, math-based mixer.   Instead of letting each token “vote” on what matters (as in attention), FNet mixes information across the entire sequence using a 2D FFT. It’s more like signal processing than decision-making. What makes this special:   • No learned parameters in the mixing step   • Global context is still preserved   • Faster compute: O(N log N) instead of O(N²)   • Smaller models, faster training 🚀 Why this matters:   FNet could power low-latency, low-cost models for edge devices, mobile apps, and real-time multi-agent systems. It’s a shift from scaling *up* to designing *smarter*. 🚸 One catch: FNet alone gets 92–97% of BERT’s performance.   But a hybrid model (FNet + a few attention layers) closes that gap, while keeping most of the speed benefits. 💡The big takeaway?   Architectural efficiency might be the next frontier in AI, not just bigger models, but better-designed ones

Transformers have powered the AI revolution, but they come with major flaws. They struggle with long contexts, burn excessive energy, and fail to adapt dynamically to new information. Enter Titans, a novel architecture from Google Research, designed to tackle these issues by introducing a hierarchical memory system: • 𝐒𝐡𝐨𝐫𝐭-𝐭𝐞𝐫𝐦 𝐦𝐞𝐦𝐨𝐫𝐲: Handles immediate context, similar to transformers. • 𝐋𝐨𝐧𝐠-𝐭𝐞𝐫𝐦 𝐦𝐞𝐦𝐨𝐫𝐲: Retains key information across extended inputs (10,000+ tokens). • 𝐏𝐞𝐫𝐬𝐢𝐬𝐭𝐞𝐧𝐭 𝐦𝐞𝐦𝐨𝐫𝐲: Stores reusable knowledge for future tasks. What sets Titans apart is their ability to 𝐫𝐞𝐟𝐢𝐧𝐞 𝐦𝐞𝐦𝐨𝐫𝐲 𝐝𝐲𝐧𝐚𝐦𝐢𝐜𝐚𝐥𝐥𝐲 𝐝𝐮𝐫𝐢𝐧𝐠 𝐫𝐮𝐧𝐭𝐢𝐦𝐞. Unlike transformers, which rely entirely on pretraining, Titans can adapt to new inputs on the fly, making them far more versatile for real-world tasks. Additionally, by compressing less relevant data, Titans reduce energy use for long-context tasks by 20-25%. This is crucial in addressing AI’s growing resource demands. But how does all these change anything for AI practitioners? Let’s discuss this. In my opinion, Titans could potentially reduce our reliance on specialized models by making general-purpose systems more adaptable. With their dynamic memory, Titans don’t necessarily need retraining for every domain-specific task. For example, rather than building entirely new models for healthcare or legal use cases, a Titan could learn and adapt to these domains in real time. Of course, whether this holds true in noisy, real-world datasets is an open question, but it’s an exciting possibility to explore. Another area where Titans might make a big difference is in retrieval-augmented generation (RAG) systems. Their ability to refine retrieved knowledge iteratively could make RAG outputs much more coherent and accurate, especially for complex, multi-step tasks. However, there’s still the challenge of how this added capability impacts performance in real-time settings—longer reasoning chains might mean slower responses, which could be a trade-off we need to evaluate. I also think Titans could simplify the design of AI agents. By integrating retrieval, reasoning, and adaptability into one system, they might eliminate the need for separate components or multi-agent setups. This could reduce infrastructure complexity and make agents more versatile overall. Titans represent an exciting step forward, but they also raise critical questions. Are we addressing the core limitations of transformers, or just trading one set of inefficiencies for another? How scalable and practical are these ideas for the kinds of applications we use AI for every day? I’d love to hear your thoughts 🔗: https://lnkd.in/ejZYS7ia

Is deep learning architecture actually an "illusion"? 🧠 A fascinating new paper from Google titled "Nested Learning: The Illusion of Deep Learning Architecture" challenges how we fundamentally build AI models. It argues that the distinction between "architecture" (like Transformers) and "optimizers" (like Adam) is artificial. Instead, they propose that a model should be viewed as a system of nested optimization loops, all running at different speeds. Here is the breakdown of this new paradigm: 📍 The Problem: "AI Amnesia". Current LLMs are static. Once pre-training ends, their weights freeze. They suffer from "anterograde amnesia" meaning they can process immediate context, but they cannot permanently learn from new experiences without expensive retraining. 📍 The Solution: Nested Learning (NL) The authors introduce a framework where every component is a "learner." ❥ Continuum Memory: Instead of just Short-Term Memory (Attention) and Long-Term Memory (Frozen Weights), the model uses a spectrum of memory blocks updating at different frequencies. ❥ "Hope" Architecture: A new model designed to replace the standard Transformer block. It is "self-referential," meaning it generates its own learning rules and update parameters on the fly based on the context. Key Results: 📍 Continual Learning: The model significantly outperforms standard baselines in learning new tasks without "catastrophic forgetting" (wiping out old knowledge). 📍 Massive Context: In "Needle-in-a-Haystack" tests, the architecture maintained accuracy up to 10M context length (with fine-tuning), where standard Transformers failed much earlier. 📍 Reasoning: The 1.3B parameter "Hope" model outperformed equivalent Transformers and RNNs (like RetNet and RWKV) on reasoning benchmarks like PIQA and HellaSwag. The Takeway: The paper suggests that to get to the next level of AI, we shouldn't just stack more layers. Instead, we should stack more "time scales," allowing models to learn, adapt, and remember simultaneously. Limitation: While the results are impressive, shifting away from the highly optimized Transformer infrastructure presents significant engineering and scaling challenges for immediate adoption. #MachineLearning #DeepLearning #AIResearch #NeuralNetworks #ArtificialIntelligence

🧠 Titans and Transformers Unite! What if AI models could learn and memorize information in real-time, just like humans do? New research (link in comments) from Google introduces "Titans" - a new architecture that's challenging how we think about AI memory. The key innovation? A neural long-term memory module that learns to identify and store surprising or important information during inference, similar to how human memory prioritizes unexpected events. Three fascinating findings: - Titans outperformed both Transformers and modern recurrent models across multiple tasks, while scaling to massive 2M+ context windows - far beyond traditional limits. - The architecture introduces a "surprise-based" memory system, measuring both immediate surprise and the flow of information over time. This helps it determine what's truly worth remembering. - In needle-in-haystack tasks, Titans achieved 98.6% accuracy on 16K sequences - significantly outperforming GPT-4 and other large language models, despite using far fewer parameters. Titans introduces a two-tier memory system: - Short-term: Uses attention for precise, immediate understanding - Long-term: A neural memory module that learns what's worth remembering, just like our brains prioritize surprising or important events The real breakthrough? Titans can learn during deployment: - Adapts its memory in real-time - Uses "surprise metrics" to decide what to remember - Maintains fast training AND inference speeds The implications? We might be seeing the emergence of AI systems that can learn and adapt during deployment, rather than remaining static after training. What do you think - could this approach to AI memory revolutionize how we build adaptive systems? #MachineLearning #AI #DeepLearning #NeuralNetworks #Innovation