Physical AI Lab

Move your finger.
A robot arm follows.

Three hours. Webcam → perception → state vector → simulation → RL → your hand controlling a robot.

21→4 landmarks→angles
50k→1 steps to policy
1 line loop closed
Slides & Handbook →
Webcam raw pixels Perception MediaPipe · M1 State Vector numpy float32 common format Sim Agent PPO / Hand · M3–4 Action torques · M4 World / Sim MuJoCo · Reacher · M2 feedback — loop is closed Foundation Model reasoning Gemini · Module 5 M1 M1 M1 → M4 M3 / M4 M4
The dashed red path is why this isn't just inference — each action changes the world, which changes what the system perceives next.

Six modules. One complete pipeline.

M0 Kickoff 15 min

We draw the pipeline together. Every module after this builds one piece of it.

M1 Perception 30 min

Your webcam sees 921,600 numbers per frame. We compress that down to four.

M2 Simulation 20 min

We never write a physics simulator. We just use one.

M3 Reinforcement Learning 30 min

The agent starts random. Watch it figure out balance on its own — no one told it what balance means.

M4 Perception → Action 40 min

One line of code maps your hand to a robot torque. The loop closes here.

This is the moment everything connects.
M5 Foundation Models 25 min

We swap the hand for Gemini. It looks at the camera and decides what to do.

M6 Wrap-up 15 min

Everything you ran today is what researchers actually use. MuJoCo, SB3, MediaPipe. Not toys.

Where this fits.

The pattern you build in M5 — camera frame → model → action — is the same architecture behind the models pushing the frontier right now. The lab gives you the working foundation. These are where it goes next.

VLA — Vision-Language-Action

RT-2 (Google DeepMind) · π0 / π0-FAST (Physical Intelligence) · OpenVLA (Stanford/Berkeley) · GR00T N1 (NVIDIA) — open foundation model for humanoid robots

World Models

NVIDIA Cosmos 3 — combines physical reasoning, world generation and action prediction in one model · Cosmos Policy — post-trained on Cosmos for robot manipulation

Imitation Learning

ACT (Action Chunking with Transformers) · Diffusion Policy · LeRobot (Hugging Face) — the place to go when you want real hardware

Also a handbook.

The slides work in both directions — as a live presentation and as a self-contained reference you read at your own pace. Every concept has a plain-language explanation, an analogy, and code you can run. The complete Physical AI stack, in one scrollable document.

Read as handbook →

Prerequisites

Windows 11. Chrome or Edge. That's it for hardware. The rest downloads itself.

Setup Guide GitHub Repository

Jim Seelan

I built this so people could actually touch the full stack in one afternoon — not hear about it, not watch a video. The real thing. Run it, break it, change a number, see what happens.

— Jim Seelan · LinkedIn