🤖 Apple + Nvidia = Smarter Robots

Hi Futurists,

Today we’re talking about Nvidia’s use of the Apple Vision Pro to record humans controlling robots for more realistic training data 👀

Remember, this week we are going to change up the domain from which you receive this email. Thank you for your continued support!

Best,
Lex

Accelerate AI Innovation with Gretel & Lambda Lab

Learn how Gretel and Lambda Lab unlock faster experimentation so teams can easily vet approaches, fail fast, and be much more agile in delivering a LLM solution that works. In this webinar, you will learn how to use Lambda Labs to fine-tune an SLM on several versions of the synthetic dataset from Gretel to reinforce how easy it is to experiment with task-specific LLMs.

Highlight of the Day

Transforming Robot Training with Apple's Vision Pro: Nvidia's Project Gr00t

Nvidia is pushing the boundaries of humanoid robot development with its latest initiative, Project Gr00t, aiming to bridge the "simulation gap" by utilizing Apple's Vision Pro headset for more realistic training data. This approach uses the headset to gather high-quality data by allowing humans to control robots from a first-person perspective, performing tasks such as making toast or retrieving a glass. As Jim Fan, Nvidia’s Senior Research Manager for Embodied AI, notes, “Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real-time.”

The magic happens when this human-generated data is fed into Nvidia’s RoboCasa simulation framework. The MimicGen system generates new actions and filters out unsuccessful attempts. This method, as Fan describes, trades "compute for expensive human data by GPU-accelerated simulation," effectively breaking the barrier of limited teleoperation data collection.

This technique could significantly narrow the sim-to-real gap — the challenge of translating robot training from simulation to the complexities of the real world. At the recent Siggraph conference, Nvidia CEO Jensen Huang highlighted the "three-computer problem" in robotics, underscoring the need for separate systems to create, simulate, and deploy AI. This comprehensive approach ensures robust development and optimization before real-world application, promising a new era in humanoid robotics.

Summary

• Data Collection: Nvidia uses Apple's Vision Pro for gathering first-person perspective data.

• Data Multiplication: RoboCasa framework exponentially expands collected data.

• Action Generation: MimicGen system creates new actions and filters out failures.

• Goal: Bridge the sim-to-real gap in robotic training.

• Comprehensive Approach: Nvidia CEO highlights the "three-computer problem" for thorough AI development.

What we think

Nvidia's integration of Apple’s Vision Pro into Project Gr00t is a game-changer for a way to train humanoid robots. The exponential data multiplication via RoboCasa and MimicGen could lead to significant advancements in how robots learn and interact with the real world, potentially accelerating the deployment of AI in everyday tasks.

It is weird though to use tech like Vision Pro to put ourselves in the seat of robots, only to train and make robots in our image.

👉 Read More

The peer group you’ve been waiting for.

The most successful people surround themselves with incredible talent. But today, it’s hard to find those people and facilitate the meetings. That’s why we built Sidebar - small groups of peers, facilitated by experts, that give you the advice you need now. No more sifting through blogs or scrubbing podcasts. Get the insights you need from incredible people like you, so you can get back to crushing your goals.

93% of users say Sidebar has been a game-changer.

Get Started

Cool AI Tools

1. Jobright AI

   Your AI job search copilot

   

1. Motiff

   AI powered professional UI design tool

   

1. EduWiz.AI

   Write magical paperwork in seconds with AI

   

Best of the Rest

• Google pulls Gemini AI ad from Olympics after backlash

• Microsoft gains major AI client as TikTok spends $20 million monthly

• Can a Large Language Model Recognize Itself?

• The hidden climate cost of AI: How tech giants are struggling to go green

• Big Tech Fails to Convince Wall Street That AI Is Paying Off

And now your moment of AI art Zen

That’s all for today folks!

• If you’re enjoying the newsletter, share with a friend by sending them this link: 👉 https://www.theedge.so/subscribe 

• Looking for past newsletters? You can find them all here.

• Working on a cool A.I. project that you would like us to write about? Reply to this email with details, we’d love to hear from you!