Training AI Models With Limited Data

We trained a humanoid with 22-DoF dexterous hands to assemble model cars, operate syringes, sort poker cards, fold/roll shirts, all learned primarily from 20,000+ hours of egocentric human video with no robot in the loop. Humans are the most scalable embodiment on the planet. We discovered a near-perfect log-linear scaling law (R² = 0.998) between human video volume and action prediction loss, and this loss directly predicts real-robot success rate. Humanoid robots will be the end game, because they are the practical form factor with minimal embodiment gap from humans. Call it the Bitter Lesson of robot hardware: the kinematic similarity lets us simply retarget human finger motion onto dexterous robot hand joints. No learned embeddings, no fancy transfer algorithms needed. Relative wrist motion + retargeted 22-DoF finger actions serve as a unified action space that carries through from pre-training to robot execution. Our recipe is called "EgoScale": - Pre-train GR00T N1.5 on 20K hours of human video, mid-train with only 4 hours (!) of robot play data with Sharpa hands. 54% gains over training from scratch across 5 highly dexterous tasks. - Most surprising result: a *single* teleop demo is sufficient to learn a never-before-seen task. Our recipe enables extreme data efficiency. - Although we pre-train in 22-DoF hand joint space, the policy transfers to a Unitree G1 with 7-DoF tri-finger hands. 30%+ gains over training on G1 data alone. The scalable path to robot dexterity was never more robots. It was always us. - Website: https://lnkd.in/gxzgeP-2 - Paper: https://lnkd.in/g7PJdz_8

If you are wondering how RLHF works, and how we can teach large language models to be helpful, harmless, and honest, read along 👇 The key isn’t just in scaling up model size, it’s in aligning models with human intent. The InstructGPT paper (2022) introduced a three-step process called Reinforcement Learning from Human Feedback (RLHF). And even today, it remains the foundation of how we build instruction-following models like ChatGPT. Let me walk you through the workflow in plain terms, based on the now-famous diagram below 👇 𝟭. 𝗦𝘂𝗽𝗲𝗿𝘃𝗶𝘀𝗲𝗱 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 (𝗦𝗙𝗧) → Start by showing the model examples of great answers to real prompts, written by humans. → These examples help the model learn how to respond: clear, direct, and grounded. → Think of this as training a junior writer by giving them a stack of perfect first drafts. → Even with a small dataset (13k samples), this creates a solid instruction-following base. 𝟮. 𝗥𝗲𝘄𝗮𝗿𝗱 𝗠𝗼𝗱𝗲𝗹 (𝗥𝗠) → Next, we collect several outputs for the same prompt and ask humans to rank them from best to worst. → We then train a separate model- the reward model, to predict those rankings. → Now, we’ve turned human preferences into a numerical score the model can optimize for. → This is the real magic: turning subjective feedback into something that can guide learning. 𝟯. 𝗥𝗲𝗶𝗻𝗳𝗼𝗿𝗰𝗲𝗺𝗲𝗻𝘁 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 (𝗣𝗣𝗢) → Now the model generates new answers, gets scored by the reward model, and adjusts its behavior to maximize reward. → We use Proximal Policy Optimization (PPO), an RL algorithm that gently nudges the model in the right direction without making it forget what it already knows. → A “KL penalty” keeps it from straying too far, like a seatbelt keeping it grounded. 𝗪𝗵𝘆 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀❓ ✅ A small 1.3B model trained with this pipeline outperformed GPT-3 (175B) in human evaluations. ✅ It generalized to unseen domains with little extra supervision. ✅ And it required orders of magnitude less data than pre-training. 𝗪𝗵𝗮𝘁 𝘁𝗵𝗶𝘀 𝗺𝗲𝗮𝗻𝘀 𝗳𝗼𝗿 𝗯𝘂𝗶𝗹𝗱𝗲𝗿𝘀❓ → Bigger isn’t always better. Better feedback leads to better behavior. → Pairwise comparisons are often more scalable than manual ratings. → RLHF lets us teach models values, not just vocabulary. If you're building AI systems, aligning them with human preferences isn’t just a safety concern- it’s a product strategy. --------- Share this with your network ♻️ Follow me (Aishwarya Srinivasan) for more AI insights.

Massachusetts Institute of Technology researchers just dropped something wild; a system that lets robots learn how to control themselves just by watching their own movements with a camera. No fancy sensors. No hand-coded models. Just vision. Think about that for a second. Right now, most robots rely on precise digital models to function - like a blueprint telling them exactly how their joints should bend, how much force to apply, etc. But what if the robot could just... figure it out by experimenting, like a baby flailing its arms until it learns to grab things? That’s what Neural Jacobian Fields (NJF) does. It lets a robot wiggle around randomly, observe itself through a camera, and build its own internal "sense" of how its body responds to commands. The implications? 1) Cheaper, more adaptable robots - No need for expensive embedded sensors or rigid designs. 2) Soft robotics gets real - Ever tried to model a squishy, deformable robot? It’s a nightmare. Now, they can just learn their own physics. 3) Robots that teach themselves - instead of painstakingly programming every movement, we could just show them what to do and let them work out the "how." The demo videos are mind-blowing; a pneumatic hand with zero sensors learning to pinch objects, a 3D-printed arm scribbling with a pencil, all controlled purely by vision. But here’s the kicker: What if this is how all robots learn in the future? No more pre-loaded models. Just point a camera, let them experiment, and they’ll develop their own "muscle memory." Sure, there are still limitations (like needing multiple cameras for training), but the direction is huge. This could finally make robotics flexible enough for messy, real-world tasks - agriculture, construction, even disaster response. #AI #MachineLearning #Innovation #ArtificialIntelligence #SoftRobotics #ComputerVision #Industry40 #DisruptiveTech #MIT #Engineering #MITCSAIL #RoboticsResearch #MachineLearning #DeepLearning

You cannot train AI on reality alone anymore. There is not enough of it. Jensen Huang explains why NVIDIA built Cosmos, an AI world model that generates synthetic training data grounded in physics. The problem is simple. Teaching physical AI like robotics requires vast amounts of diverse interaction data. Videos exist, but not nearly enough to capture the variety of situations robots will encounter. So NVIDIA transformed compute into data. Using synthetic data generation grounded by laws of physics, they can selectively generate training scenarios that would be impossible to capture otherwise. The example Huang shows is remarkable. A basic traffic simulator output gets fed into Cosmos. What emerges is physically plausible surround video that AI can learn from. This solves a fundamental limitation. You cannot train autonomous systems on every possible scenario by recording reality. There are not enough cameras or time. But you can simulate physics accurately enough that AI trained on synthetic data generalises to real environments. This applies beyond robotics. Any AI learning physical interactions, from manufacturing to logistics to infrastructure monitoring, faces the same data scarcity problem. Synthetic data generation grounded in physics laws is how you create training sets reality cannot provide. The organisations building AI for physical systems will either master synthetic data generation or get limited by whatever reality they can record. Watch the full presentation to hear Huang explain how Cosmos generates training data for physical AI. What physical AI application needs synthetic data because reality cannot provide enough examples? #AI #SyntheticData #Robotics #NVIDIA #MachineLearning

Reinforcement learning from human feedback (RLHF)—the technique for getting large language models to follow instructions—usually operates in two separate phases: collect a large set of human judgments comparing pairs of model responses, then train a reward model (a network that scores responses) and use those scores to push the language model toward higher-rated outputs. Google DeepMind researchers instead interleave the two, updating both models as each new comparison arrives, and pick which response pairs to send to humans based on which choice would be most informative. Informativeness rests on a small ensemble of networks attached to the reward model, which produces a spread of reward predictions instead of a single number; pairs where ensemble members disagree most about the winner are the ones routed to humans. They report needing roughly 10x fewer human labels than standard offline RLHF at the 20K-comparison scale, and project a 1000x reduction at one million comparisons. Read with an AI tutor on ChapterPal: https://lnkd.in/eJK9aFER PDF: https://lnkd.in/ecQFNxcp

If you are building Physical AI, your workflow just got a lot shorter, smarter. NVIDIA dropped a big collection of open-source agent skills for Physical AI last week, and I want to break down why this matters if you're the one actually doing the work. Here's the problem every Physical AI developer knows too well: the pipeline is brutal. Generate synthetic data, set up simulation, configure sensors, train, evaluate, deploy — and most of it is boilerplate, glue code, and tribal knowledge buried in someone's head. The hard part was never the idea. It was the 200 lines of setup before you could test the idea. Skills change that. Think of them as repeatable, agent-executable instructions. Instead of hand-writing the setup, you tell your coding agent — Claude Code, Cursor, OpenAI Codex, whatever you use — what you want, and the skill tells the agent which NVIDIA tools to call, what outputs to produce, and how to validate the result. The knowledge that used to live in a senior engineer's head is now portable. A few that stand out: → ovrtx skills (renderer creation, USD loading, stepping & rendering) — spin up physically accurate camera/LiDAR/radar simulation programmatically, straight into CUDA memory at thousands of frames/sec for RL loops. Check out this skill in action. Its super easy. https://lnkd.in/gF6d6cYh → Realtime Viewer — a browser-based OpenUSD viewer that streams RTX-rendered output straight to the web. No heavy desktop client, no install friction. You — or your customer, or a teammate three time zones away — can see the live simulation state in a browser tab. This is the "see, share and validate fast" half. Here is a live walkthrough https://lnkd.in/gSYb9kjh → Neural Reconstruction & Video Augmentation — turn real-world fleet captures into simulation environments. → Defect Image Generation — synthetic defect data for inspection models. And this isn't theoretical. Pegatron cut model training and deployment time by 67%. Inventec dropped defect data collection effort by 30%. Delta improved detection rate by 17%. Those are real numbers from teams shipping today. What this really means for a developer: less time fighting setup, more time on the actual problem. The barrier to standing up a digital twin or a sim loop went from "spin up a heavy desktop project" to "ask your agent." That's a different speed of iteration. 110 skills, 24 products, all open source — on GitHub (NVIDIA/skills) and skills.sh, usable with any coding agent. Github: https://lnkd.in/gikb6-KA Some are runnable instantly on NVIDIA Brev as preconfigured Launchables, so you can try before you commit a single line. If you're building robots, AVs, vision AI, or industrial twins, this is worth an afternoon of your time. What would you automate first? Curious where other builders see the biggest time savings. #PhysicalAI #Omniverse #OpenUSD #Robotics #DigitalTwins #NVIDIA Jessica Ji Edmar MendizabalMara Mahoney Kristen Rumley Meaghan Fitzpatrick

ChatGPT owes much of its "intelligence" to us humans. Despite the widespread belief that these models are purely machine-trained, the human touch has truly guided them to their current heights of sophistication. 🔥🧠 Now, this sparks an intriguing conversation about Reinforcement Learning from Human Feedback (RLHF) - a crucial technology behind these models. In the world of LLMs, RLHF is not an alien concept, but the degree to which it shapes models like GPT is underappreciated. RLHF isn't an elite club limited to the playground of tech giants. Even smaller models and tight-budget teams can harness its powers to remarkable effects.🚀🌍 Our latest blog post delves into the synergy between RLHF and Large Language Models (LLMs) and how you can use it to your advantage, regardless of your project's scale. Here's a sneak peek into what you'll discover: 1️⃣ The blend of RL and NLP to create RLHF for enhanced language understanding. 2️⃣ The essential stages of RLHF and how it's akin to teaching a parrot to communicate. 3️⃣ Real-world applications and challenges of RLHF, including the issue of hallucinations in LLMs. 4️⃣ A practical guide on implementing RLHF, even with limited resources.

Using LLMs for market research and consumer understanding is a smart idea. Large FMCG companies have a long history of effectively using MR to build their brands. However these can be very expensive and time consuming. New age companies (Specially in Auto, Tech, Finance, even D2C and others) have shied away from this rigour precisely for the same reason. They need to turn around products and innovation much faster and fear the costs associated. Therefore they simply wing it. This is one of the reasons for high failure rate of new companies. Not knowing your customers well before building your business. Role play with LLMs can boost customer understanding and behaviour significantly, even if it is not 100%. A recent paper co-authored by PyMC Labs and Colgate-Palmolive teams demonstrates how LLMs can be used to replace sophisticated Market Research with 90% accuracy. Not just qualitative but quantitate. The paper "LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings" summarises "Consumer research costs companies billions annually yet suffers from panel biases and limited scale. Large language models (LLMs) offer an alternative by simulating synthetic consumers, but produce unrealistic response distributions when asked directly for numerical ratings. We present semantic similarity rating (SSR), a method that elicits textual responses from LLMs and maps these to Likert distributions using embedding similarity to reference statements. Testing on an extensive dataset comprising 57 personal care product surveys conducted by a leading corporation in that market (9,300 human responses), SSR achieves 90% of human test–retest reliability while maintaining realistic response distributions (KS similarity > 0.85). Additionally, these synthetic respondents provide rich qualitative feedback explaining their ratings. This framework enables scalable consumer research simulations while preserving traditional survey metrics and interpretability." I have been using some of these techniques. Saves a lot of time and some results are quite surprising (never thought of). Before taking your company, product, brand or campaign to market, use LLMs to garner better consumer understanding and test. It can save a lot of heartburn.

Exciting Research Alert: Improving RAG with Self-Generated Demonstrations I just came across a fascinating paper that addresses a critical challenge in Retrieval-Augmented Generation (RAG) systems. The research team from USC and Meta has developed a novel approach called Self-Demo Retrieval-Augmented Instruction Tuning (SD-RA-IT) that significantly improves how Large Language Models (LLMs) handle retrieved information. >> The Problem They're Solving When fine-tuning LLMs for RAG, we typically use human-written responses that weren't created with the retrievals in mind. This creates two major issues: - Misalignment between retrievals and responses - Training on out-of-distribution (OOD) text that the model wouldn't naturally generate These issues lead to hallucinations and poor performance in RAG systems. >> Their Innovative Solution The researchers propose a brilliantly simple yet effective approach: 1. Generate multiple response candidates from the LLM itself using the instruction and retrievals 2. Filter these self-generated responses for correctness against gold answers 3. Train the model on these in-distribution self-demos instead of human-written responses 4. When no good response is found, train the model to generate a refusal This method ensures the training data matches the model's own distribution while still providing accurate supervision. >> Technical Implementation Details Their implementation uses Llama-3-8B-Instruct and Llama-3-70B-Instruct models with fairseq2 for training and vLLM for inference. They employ automatic prompt optimization to generate diverse response candidates and use a tournament-style filtering process to select the best responses. The team evaluated two training objectives: - Supervised fine-tuning (SFT) with cross-entropy loss - Direct preference optimization (DPO) using rejected responses as negative examples >> Impressive Results The results speak for themselves: - Higher precision (accuracy on attempted questions) - Higher recall (successful attempts on answerable questions) - Lower counterfactual accuracy (better at refusing questions it would get wrong) - Minimal degradation in non-RAG settings - Superior performance across different numbers of retrievals Most importantly, SD-RA-IT models are significantly better at avoiding hallucinations by refusing to answer questions they're likely to get wrong, while still correctly extracting answers from relevant retrievals. This research provides valuable insights for anyone working with RAG systems. By training on self-generated demonstrations, we can create more reliable and accurate RAG-enabled language models.