Trends In AI Training Techniques For Limited Data

For years, advancing pathology AI meant one thing: hoarding massive, proprietary datasets to train ever-larger models. But a new paper suggests the field might be hitting diminishing returns on simply adding more data. Histopathology data is highly redundant. The vast majority of tissue patches look similar, creating a long-tailed distribution where rare, diagnostically critical features get drowned out by common morphologies. Consequently, brute-force scaling is highly inefficient. Saarthak Kapse et al. released 𝙂𝙚𝙣𝘽𝙞𝙤-𝙋𝙖𝙩𝙝𝙁𝙈, a 1.1 billion parameter open-weight foundation model that challenges the "more is better" dogma. Despite using only 10% to 20% of the training data required by current leading proprietary models (like Virchow2 or UNI2), it achieves state-of-the-art results across clinical, molecular, and robustness benchmarks. Here is how they achieved unprecedented data efficiency: • 𝘼𝙪𝙩𝙤𝙢𝙖𝙩𝙚𝙙 𝘿𝙖𝙩𝙖 𝘾𝙪𝙧𝙖𝙩𝙞𝙤𝙣: Instead of using every available WSI, the team built a fully unsupervised pipeline using hierarchical clustering and stratified sampling to select tiles based on morphological diversity. This approach filters out redundant patterns, allowing the model to train on just 177k public WSIs while prioritizing high-entropy content like rare histological variants. • 𝙏𝙝𝙚 𝙅𝙀𝘿𝙄 (𝙅𝙀𝙋𝘼 + 𝘿𝙄𝙉𝙊) 𝙋𝙧𝙚𝙩𝙧𝙖𝙞𝙣𝙞𝙣𝙜 𝙍𝙚𝙘𝙞𝙥𝙚: Standard DINO pretraining captures good global morphology, but the authors went a step further by introducing a novel dual-stage strategy. After an initial DINO phase, they froze the encoder and used it as a teacher for a JEPA-based student. By tasking the student with predicting visible regions and outpainting missing ones, the model learned highly fine-grained, spatially-aware representations without relying on raw pixel reconstruction. • 𝙎𝙩𝙖𝙩𝙚-𝙤𝙛-𝙩𝙝𝙚-𝘼𝙧𝙩 𝙍𝙤𝙗𝙪𝙨𝙩𝙣𝙚𝙨𝙨: When tested on the PathoROB benchmark (which measures resilience to varying scanners and stains across multi-center datasets), 𝙂𝙚𝙣𝘽𝙞𝙤-𝙋𝙖𝙩𝙝𝙁𝙈 established a new state-of-the-art average Robustness Index of 0.888, significantly outperforming much larger, data-heavy models like Virchow2 and UNI2. 𝙏𝙝𝙚 𝙩𝙖𝙠𝙚𝙖𝙬𝙖𝙮: The future of clinical AI is not just about who has the biggest private dataset. Intelligent data curation and optimized learning objectives can match or exceed the performance of unconstrained scaling, offering a path to more accessible, transparent, and robust pathology AI. https://lnkd.in/evdMNVak --- Keeping up with the literature is increasingly a team sport. This analysis was supported by NotebookLM and grounded in my own review and experience. If you found this useful, let me know in the comments. If it missed the mark, I want that feedback too. Weekly briefings on making vision AI work in the real world → https://lnkd.in/guekaSPf

How do we build AI for science? Augment with AI or replace with AI? Popular prescription is to augment AI into existing workflows rather than replace them, e.g., keep the approximate numerical solver for simulations, and use AI only to correct its errors in every time step. The other extreme is to completely discard the existing workflow and replace it fully with AI. We have seen this approach win in areas like weather forecasting. Such end-to-end AI is significantly better for speed: 1000-million x faster. In our latest paper, we show end-to-end learning also wins in data efficiency, which is counterintuitive. Where do these savings come from? The former approach that augments AI relies only on fully accurate training data that is expensive. But end-to-end learning can use both approximate and accurate training data, if the model can learn how to mix them correctly. In many physical systems, coarse-grid numerical solvers yield approximate data while fine-grid solvers fully resolve the scales and yield exact answers. It turns out that Neural Operators offer a perfect solution when such multi-fidelity and multi-resolution data is available, and can learn with high data efficiency requiring only a small amount of fully resolved data, since it can also utilize approximate training data. In contrast, the standard approach of augmenting AI to a coarse-grid numerical solver (closure model) can only train on fully-resolved simulations, making it very expensive and hard to train. Our results are applicable in multi-scale chaotic systems that have traditionally required running long simulations at high resolution such as climate change or plasma in nuclear fusion and astrophysics. Now you can replace expensive simulation fully with AI (Neural Operators), and also train it without requiring such simulations in large numbers for training in many scenarios.

I recently delved into some intriguing research about the often-overlooked potential of Small Language Models (SLMs). While LLMs usually grab the headlines with their impressive capabilities, studies on SLMs fascinate me because they challenge the “bigger is better” mindset. They highlight scenarios where smaller, specialized models not only hold their own but actually outperform their larger counterparts. Here are some key insights from the research: 𝟏. 𝐑𝐞𝐚𝐥-𝐓𝐢𝐦𝐞, 𝐏𝐫𝐢𝐯𝐚𝐜𝐲-𝐅𝐨𝐜𝐮𝐬𝐞𝐝 𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬: SLMs excel in situations where data privacy and low latency are critical. Imagine mobile apps that need to process personal data locally or customer support bots requiring instant, accurate responses. SLMs can deliver high-quality results without sending sensitive information to the cloud, thus enhancing data security and reducing response times. 𝟐. 𝐒𝐩𝐞𝐜𝐢𝐚𝐥𝐢𝐳𝐞𝐝, 𝐃𝐨𝐦𝐚𝐢𝐧-𝐒𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐓𝐚𝐬𝐤𝐬: In industries like healthcare, finance, and law, accuracy and relevance are paramount. SLMs can be fine-tuned on targeted datasets, often outperforming general LLMs for specific tasks while using a fraction of the computational resources. For example, an SLM trained on medical terminology can provide precise and actionable insights without the overhead of a massive model. 𝟑. 𝐀𝐝𝐯𝐚𝐧𝐜𝐞𝐝 𝐓𝐞𝐜𝐡𝐧𝐢𝐪𝐮𝐞𝐬 𝐟𝐨𝐫 𝐋𝐢𝐠𝐡𝐭𝐰𝐞𝐢𝐠𝐡𝐭 𝐀𝐈: SLMs leverage sophisticated methods to maintain high performance despite their smaller size: • Pruning: Eliminates redundant parameters to streamline the model. • Knowledge Distillation: Transfers essential knowledge from larger models to smaller ones, capturing the “best of both worlds.” • Quantization: Reduces memory usage by lowering the precision of non-critical parameters without sacrificing accuracy. These techniques enable SLMs to run efficiently on edge devices where memory and processing power are limited. Despite these advantages, the industry often defaults to LLMs due to a few prevalent mindsets: • “Bigger is Better” Mentality: There’s a common belief that larger models are inherently superior, even when an SLM could perform just as well or better for specific tasks. • Familiarity Bias: Teams accustomed to working with LLMs may overlook the advanced techniques that make SLMs so effective. • One-Size-Fits-All Approach: The allure of a universal solution often overshadows the benefits of a tailored model. Perhaps it’s time to rethink our approach and adopt a “right model for the right task” mindset. By making AI faster, more accessible, and more resource-efficient, SLMs open doors across industries that previously found LLMs too costly or impractical. What are your thoughts on the role of SLMs in the future of AI? Have you encountered situations where a smaller model outperformed a larger one? I’d love to hear your experiences and insights.

Fine-tuning is being outpaced. Very fast. A year ago, I would have recommended fine-tuning without hesitation. Domain-specific problem? Unique engineering challenge? The default answer was: train your own specialized model. That logic made perfect sense until reality caught up. Recently, Davy Demeyer challenged this idea under one of my posts about Industrial Foundation Models. His point hit hard: Why spend months fine-tuning a model when the next breakthrough arrives before your training run is even finished? And that’s exactly what we’re seeing now. New models are released in a rhythm that feels almost monthly. Each one more capable. Each one making your freshly fine-tuned version feel outdated the moment it goes live. So the real shift is happening elsewhere. The center of gravity is moving from Fine-Tuning to Context Management. Instead of training models, we structure and deliver the right domain knowledge at the right moment. No additional training costs. No risks of locking yourself into a snapshot of yesterday’s architecture. Just clean, standardized, reusable engineering data injected directly into the model’s context when needed. My key takeaways from the past months: • Fine-tuning might be still useful in rare cases but it’s becoming a niche, not the norm. • Context Management scales faster, adapts quicker and survives model turnover. • Companies that standardize their engineering data will dominate the next wave of LLM and Agentic AI adoption. • The real competitive advantage won’t be your model, it will be your data readiness. The trend is unmistakable: We’re moving toward context-driven systems and reusable engineering knowledge, not custom-trained, short-lived models. What’s your perspective on this shift? Vlad Larichev | Octavian Ciupitu | Atilla Akdere | Sebastian Angerer | Arne Breitsprecher #AIEngineering #LLMStrategy #ContextManagement #Industry40

In the last blog I talked about the importance of Classical ML/DL.  This post focuses on Finetuning image models. (Part 4 of #ArchitectingAI) Pre-trained image models are powerful. Fine-tuning them correctly is the real skill. Transfer learning lets you start with a backbone — ResNet, MobileNet, EfficientNet — already trained on millions of images, and adapt it to your problem. Less data, faster training, better results. I applied this to classify surface defects in industrial steel project. It works well and rewards a meticulous approach. Each wrong decision compounds! 5 key things that actually matter: 1. 𝐓𝐡𝐞 𝐟𝐫𝐨𝐳𝐞𝐧 𝐥𝐚𝐲𝐞𝐫𝐬 𝐚𝐫𝐞 𝐚𝐥𝐫𝐞𝐚𝐝𝐲 𝐝𝐨𝐢𝐧𝐠 𝐦𝐨𝐬𝐭 𝐨𝐟 𝐭𝐡𝐞 𝐰𝐨𝐫𝐤. 𝑭𝒊𝒏𝒆-𝒕𝒖𝒏𝒊𝒏𝒈 𝒊𝒔 𝒓𝒆𝒇𝒊𝒏𝒆𝒎𝒆𝒏𝒕, 𝒏𝒐𝒕 𝒓𝒆𝒍𝒆𝒂𝒓𝒏𝒊𝒏𝒈. Early layers in a pre-trained model capture universal patterns — edges, textures, shapes — that transfer across domains. Later layers are task-specific. Freezing the backbone and training only a new classification head gets you most of the way there. Unfreezing the whole network is rarely worth it. 2. 𝐃𝐨𝐦𝐚𝐢𝐧 𝐠𝐚𝐩 𝐝𝐞𝐭𝐞𝐫𝐦𝐢𝐧𝐞𝐬 𝐲𝐨𝐮𝐫 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐬𝐭𝐫𝐚𝐭𝐞𝐠𝐲. 𝑻𝒉𝒆 𝒇𝒖𝒓𝒕𝒉𝒆𝒓 𝒚𝒐𝒖𝒓 𝒅𝒂𝒕𝒂 𝒇𝒓𝒐𝒎 𝑰𝒎𝒂𝒈𝒆𝑵𝒆𝒕, 𝒕𝒉𝒆 𝒎𝒐𝒓𝒆 𝒚𝒐𝒖 𝒏𝒆𝒆𝒅 𝒕𝒐 𝒖𝒏𝒇𝒓𝒆𝒆𝒛𝒆. Natural images transfer easily. Industrial textures, medical scans, satellite imagery — larger domain gap, less directly applicable features. Know your gap before deciding how many layers to unfreeze. More gap = more fine-tuning needed. 3. 𝐃𝐚𝐭𝐚 𝐚𝐮𝐠𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐢𝐬 𝐧𝐨𝐭 𝐨𝐩𝐭𝐢𝐨𝐧𝐚𝐥 𝐰𝐢𝐭𝐡 𝐬𝐦𝐚𝐥𝐥 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬. 𝑻𝒉𝒆 𝒎𝒐𝒅𝒆𝒍 𝒘𝒊𝒍𝒍 𝒎𝒆𝒎𝒐𝒓𝒊𝒔𝒆 𝒚𝒐𝒖𝒓 𝒅𝒂𝒕𝒂 𝒊𝒇 𝒚𝒐𝒖 𝒈𝒊𝒗𝒆 𝒊𝒕 𝒕𝒉𝒆 𝒄𝒉𝒂𝒏𝒄𝒆. Augmentation creates diversity the model hasn't seen — rotations, skews, flips, contrast shifts, brightness changes, blur. It forces generalisation over memorisation, critical when domain-specific data is limited. Apply it before model-specific preprocessing — wrong order means augmenting corrupted inputs, silently. 4. 𝐅𝐫𝐞𝐞𝐳𝐞 𝐟𝐢𝐫𝐬𝐭. 𝐅𝐢𝐧𝐞-𝐭𝐮𝐧𝐞 𝐬𝐞𝐜𝐨𝐧𝐝. 𝑨𝒍𝒘𝒂𝒚𝒔 𝒊𝒏 𝒕𝒉𝒂𝒕 𝒐𝒓𝒅𝒆𝒓. A randomly initialised head will undo what you borrowed. New classifier weights start random. Early gradients are large and noisy — if the backbone is already unfrozen, they overwrite representations learned from millions of images. Train the head first, stabilise it, then unfreeze selectively. 5. 𝐓𝐫𝐞𝐚𝐭 𝐩𝐫𝐞-𝐭𝐫𝐚𝐢𝐧𝐞𝐝 𝐰𝐞𝐢𝐠𝐡𝐭𝐬 𝐚𝐬 𝐟𝐫𝐚𝐠𝐢𝐥𝐞 𝐝𝐮𝐫𝐢𝐧𝐠 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠. 𝑼𝒔𝒆 𝒂 𝒎𝒖𝒄𝒉 𝒔𝒎𝒂𝒍𝒍𝒆𝒓 𝒍𝒆𝒂𝒓𝒏𝒊𝒏𝒈 𝒓𝒂𝒕𝒆. A standard learning rate undoes the representations you were trying to preserve. Unfreeze only the last few layers — more layers at a higher learning rate tend to overfit or underfit fast. Transfer learning is powerful because it builds on what's already been learned. Know your domain gap. Know what to freeze. Know when to fine-tune. Do it with care.

42.1% error reduction with 85% less data. At Ento, we use a lot of traditional black-box Machine Learning to model building energy consumption, and they're great for many use cases. But they have their limits. When we're dealing with: - Plenty of indoor sensor data - Limited historical data - The need to actively control a building’s HVAC system ... plain black-box approaches often fall short. That’s why I’ve been following key trends around blending data-driven methods with physical modeling: 🔹 Transfer Learning: Use data from similar buildings to improve models. 🔹 Digital Twins: Blend data-driven methods and physical simulations. 🔹 Physics-Informed AI: Embed physical laws into the learning process to improve results. Just last month, three papers in these fields came out from leading researchers: - GenTL: A universal model, pretrained on 450 building archetypes, achieved a 42.1% average error reduction when fine-tuned with 85% less data. From Fabian Raisch et al. - An Open Digital Twin Platform: Han Li and Tianzhen Hong from LBNL built a modular platform that fuses live sensor data, weather feeds, and physics-based EnergyPlus models. - Physics-informed modeling: A new study proved that Kolmogorov–Arnold Networks (KANs) can rediscover fundamental heat transfer equations. From Xia Chen et al. Which of these 3 trends do you see having the biggest real-world impact in the next 2-3 years?

As AI becomes core to business transformation, one obstacle keeps surfacing - #Data Not just how much we have—but how usable, accessible, and compliant it is. That’s why #synthetic #data is gaining traction across industries. It offers a privacy-safe, cost-effective way to train AI models when: • Data is scarce • Real data can’t be shared due to privacy or regulation • Labeling is too expensive or slow • Biases in historical data need to be mitigated Using advanced ML techniques like GANs, diffusion models, or LLMs, synthetic data mimics real-world scenarios—enabling faster, safer AI development. Applications are already real and expanding: healthcare, finance, NLP, voice, vision, education, even location-based AI. Yes, we must address risks like privacy leakage or fairness gaps—but with differential privacy, anonymization, and thoughtful evaluation, synthetic data is proving to be a strategic enabler for modern AI pipelines. If you’re exploring AI in regulated or data-sensitive environments, this might be helpful . #SyntheticData #AI #DataPrivacy #DigitalTransformation #MachineLearning #AIGovernance #EnterpriseAI

🚀 5 Underused AI Ideas Powering the Next Generation of Startups Forget LLM wrappers. These AI approaches are building real-world traction in high-impact sectors. Symbolic Regression Advantage: Explainable, science-grade AI Best For: Health, Engineering, IoT Example: Nutonian’s Eureqa rediscovered Kepler's laws, enabling transparent, equation-based models in biotech and energy. Why It Matters: In science-heavy fields, Eureqa offered confidence through transparent models, not just black-box predictions. Causal Inference Advantage: Actionable, trustable decisions Best For: SaaS, EdTech, Health, Policy Example: Alectio used causal analysis to improve data selection, reducing costs and improving model performance. Why It Matters: Causal logic guided data collection, reducing the need for large datasets while enhancing model fairness. Hypernetworks Advantage: Real-time personalization, small models Best For: EdTech, Health, Consumer Apps Example: Sana Labs uses hypernetwork-inspired AI for personalized learning in healthcare and corporate training, improving skill retention. Why It Matters: Personalized, adaptive learning reduces dropout rates and increases training outcomes. Differentiable Programming Advantage: Hybrid symbolic-neural systems Best For: Robotics, Logistics, FinTech Example: Instadeep applied this to logistics optimization, leading to a $680M acquisition by BioNTech. Why It Matters: It solves complex, dynamic problems that pure neural nets or rule engines can't. Contrastive Learning Advantage: Data-efficient domain-specific pretraining Best For: LegalTech, MedTech, Manufacturing Example: PathAI uses contrastive learning for AI diagnostics in pathology with minimal data. Why It Matters: It scaled diagnostics in data-scarce settings, speeding AI-powered healthcare adoption. 🧠 Final Thought These AI ideas are already powering the next wave of strategic startups. They think smarter, adapt faster, and require less data—setting the stage for the future of AI.

The AI industry thinks it's running out of training data. It's not. It's running out of meaning per data point. I just published a research paper and video on what I'm calling Derived Data Abundance — a framework that multiplies effective training data by running existing content through multiple independent embedding models, each extracting a different dimension of meaning the others miss. Two systems I built prove the concept: From 16 minutes of video, my ClipCannon pipeline extracted 12,000+ individually labeled data points across 7 embedding modalities and 4,044 dimensions — enough to train identity-locked avatar generation. From a single piece of text, my Context Graph system (370K lines of Rust) produces 13 independent representations + 78 cross-correlations — a 91× multiplication in meaningful signal. No synthetic data. No model collapse risk. Every derived signal is grounded in real observation, measured by frozen models that don't participate in the training feedback loop. I'm calling this Meaning Compression. Where Google's TurboQuant compresses representations into fewer bits (meaning → fewer bits), this decompresses raw data into richer signal (raw data → more meaning). TurboQuant makes AI cheaper to run. Derived Data Abundance makes AI cheaper to train — and training data is the more expensive problem by orders of magnitude. The data wall isn't a wall. It's a door. Full Research Paper: https://lnkd.in/gPHmiGvt Video Presentation: https://lnkd.in/gh6FyVWC #AI #MachineLearning #TrainingData #DataCrisis #EmbeddingModels #MeaningCompression #DeepLearning #AIResearch