Physical AI connects perception and action. This lab explores feedback through a person controlling a simulated robot arm, alongside separate perception, simulation and model demos.

Move your hand.
Watch a robot arm respond.

In one session, you’ll turn webcam hand tracking into control signals for a simulated robot arm. You’ll also explore physics simulation, reinforcement learning and a separate Gemini demo that returns structured JSON from a camera image. A dedicated GPU or previous robotics experience isn’t required; basic Python familiarity will help.

Open the slides & handbook
Webcam raw pixels Perception MediaPipe · M1 State Vector numpy float32 common format Sim Agent PPO / Hand · M3–4 Action torques · M4 World / Sim MuJoCo · Reacher · M2 person watches arm and adjusts hand Foundation Model reasoning Gemini · Module 5 Webcam raw pixels Perception MediaPipe · M1 State Vector numpy float32 common format Sim Agent PPO / Hand · M3–4 Action torques · M4 World / Sim MuJoCo · Reacher · M2 person adjusts hand Foundation model reasoning Gemini · M5
The diagram is a conceptual overview, not a literal wiring diagram. In Module 4, the person watches the simulated arm and adjusts their hand; the webcam tracks the hand, not the arm. The separate Gemini demo returns a proposed JSON response.

A kickoff, five technical modules, and a wrap-up.

  1. M0

    Kickoff

    15 min

    We draw the pipeline together. Every module after this builds one piece of it.

  2. M1

    Perception

    30 min

    Use MediaPipe to extract hand landmarks, then calculate a small set of joint angles for the controller.

  3. M2

    Simulation

    20 min

    Use Gymnasium and MuJoCo to explore a physics simulation without building the simulator yourself.

  4. M3

    Reinforcement Learning

    30 min

    The agent starts with an untrained policy. Watch how its CartPole performance changes as it learns from rewards.

  5. M4

    Perception → Action

    40 min

    Map hand landmarks to control signals and use them to move a simulated robot arm.

    Here, your hand movements become actions in the simulation.
  6. M5

    Foundation Models

    25 min

    Inspect a separate Gemini demo that turns a camera image into a structured JSON response with a proposed assistive-robot action. It does not control the simulated arm.

  7. M6

    Wrap-up

    15 min

    You’ll have used established tools found in robotics and reinforcement-learning workflows, including MuJoCo, Stable-Baselines3 and MediaPipe.

What you’ll build and understand.

By the end of the lab, you’ll have worked through the main stages of a simple Physical AI system: reading hand landmarks from a webcam, converting measurements into a state vector, exploring a simulation, training a reinforcement-learning agent and connecting perception to action.

Basic Python familiarity assumed. No ML, robotics, or prior hardware experience needed.

Where this fits.

The M5 demo illustrates a broader idea: using visual input and a model to produce a structured, proposed action. It is separate from the simulated-arm teleoperation exercise and is not a complete trained VLA system. Research systems extend these ideas in different ways, from mapping images and instructions to robot actions to learning how an environment changes over time.

Vision-language-action models

Use visual input and language instructions to produce actions for a robot. Architectures, training methods and levels of human supervision vary by system.

  • RT-2 Google DeepMind
  • π0 / π0-FAST Physical Intelligence
  • OpenVLA Stanford / Berkeley
  • GR00T N1 NVIDIA, open foundation model for humanoid robots

World models

Learn to simulate how the environment responds to actions — enabling planning and training without running the real world every step.

  • NVIDIA Cosmos 3 physical reasoning, world generation and action prediction in one model
  • Cosmos Policy post-trained on Cosmos for robot manipulation

Imitation learning

Imitation learning trains a policy from demonstrations. Module 4 illustrates human-in-the-loop teleoperation; this lab does not record a complete demonstration dataset.

  • ACT Action Chunking with Transformers
  • Diffusion Policy
  • LeRobot Hugging Face LeRobot provides tools and resources for robot datasets, imitation learning and supported hardware

Also a handbook.

Can't attend a session? The slides are a self-contained handbook you can read at your own pace. It includes 11 concept explanations, module instructions, code examples, references, project ideas and a quick reference. Use it before a workshop, follow along during a session, or work through the examples on your own.

Read the handbook

Prerequisites

Use a Windows 11 laptop with Chrome or Edge, and complete setup before the session. Basic Python familiarity is recommended.

You’ll read and modify existing scripts; you won’t need to build a model from scratch. No prior ML or robotics experience is required.

Setup guide GitHub repository

Why this exists.

I built this workshop so people could try the pipeline themselves, rather than just hear about it or watch a video. Run the code, change a value, break something, and see what happens.

— Jim Seelan