Hands-on Robot Learning

Johns Hopkins University · Fall 2026

Hands-on Robot Learning

This course provides a hands-on introduction to the design and implementation of modern learning-based robotic systems. Students will work in small teams to assemble and operate an SO-101 arm, collect robot data, and develop the software needed to train and deploy learned policies.

Course: EN.601.498 / EN.601.698
Instructor: Krishna Murthy Jatavallabhula
Course assistants: Xinyi Yin, Yinzhe (Steven) Zhou, and Aleks Santari
Lecture: Ames 234, Thursdays 3 pm–5:30 pm
Instructor office hours: Thursdays, 12:00–1:00 PM (Hackerman 226E), please email ahead of time to confirm
CA office hours: Xinyi Yin: Wednesdays, 2–3 pm
Shriver 001
Yinzhe (Steven) Zhou: Fridays, 9:30–10:30 am
Wyman Park building, 265W
Aleks Santari: Tuesdays, 6–7 pm
Malone 122

About

Robot learning has made significant strides in just the last 2 years. For the first time, there exist ‘generalist’ manipulation systems that one can download from the web, and deploy on their robots and environments without substantial finetuning, and achieve non-trivial task success rates. At the same time, making robot learning models (a.k.a. ‘policies’) work, and generalize across a fairly broad set of tasks and environments, is an open research problem. This has caused an undeniable shift in the skillsets that robotics practitioners (engineers, scientists) must now have, to get into (and succeed at) frontier robotics labs.

This course aims to address that gap. In this class, we will take a ‘full-stack’ perspective of robot learning – we will understand (and build) robot hardware (i.e., a small tabletop robot arm) and discuss the entirety of the modern robot learning lifecycle. This includes task design and selection, curating and processing data, training neural network models on the processed data, deploying these trained models, and evaluating how the trained approaches fare.

This is not an introductory class. We expect students to already have requisite knowledge in deep learning (training and debugging transformer models, diffusion models) or be able to pick them up independently. We will do an extremely short recap of the essentials for the class (one 2.5-hour lecture). We then will look at various data curation and processing strategies, single-task imitation learning, multi-task policy learning, and will discuss some of the frontiers (world models, for example).

Instructor

Krishna Murthy Jatavallabhula

Assistant Professor of Computer Science
Johns Hopkins University

Course goals

By the end of the course, students should be able to:

Describe the modern landscape of data-driven robot learning and the common system designs used in robot learning.
Assemble, calibrate, and safely operate a robotic manipulator built from readily available electromechanical components.
Implement, train, and evaluate modern imitation learning approaches for simple manipulation tasks.
Deploy a vision-language-action (VLA) model on a physical robot and characterize its success and failure modes.
Compare the tradeoffs between policy architectures, low-level design choices, and data collection methods.
Define evaluation metrics and failure taxonomies, and conduct reproducible robot experiments.

Course materials

Syllabus and course policies (PDF)

The syllabus PDF contains the original tentative calendar. See the updated schedule below for current class dates and assignment demos.

Schedule

The schedule is tentative and may change as the semester develops.

Date	Title	Topics	Slides	Resources
Module 1 · Introduction to robot learning
Sep 3	Introduction to modern robot learning	The data-driven robotics lifecycle; the state of the art in robot learning.
Sep 10	Essentials (whirlwind recap)	Robot design and payload computations; smart servos and how they work; electromechanical components of the SO-101; anatomy of a robot manipulator; interpreting motor datasheets; joint-space and Cartesian-space representations; forward and inverse kinematics; Jacobians and singularities; position-based feedback control; PID control; and a quick recap of modern machine learning.	Slides	SO-101 servo testing H-bridge (why and how) FeeTech STS3215 datasheet Robot Modeling and Control (Spong, Hutchinson, and Vidyasagar) MIT 6.S191: Introduction to Deep Learning Inverse Kinematics applet Degrees of Freedom applet
Module 2 · Data for robot learning
Sep 17	Data for robot learning: teleoperation and other sources	Data sources for robot learning; What does/should a demonstration contain; commanded-vs-realized poses; leader–follower teleoperation; GELLO and gravity compensation; Task-space teleoperation; Wearable interfaces and UMI; Dexterous and whole-body human retargeting; and Open problems in scalable robot data collection.	Lecture 03 — Data for Robot Learning	GELLO NVIDIA SO-101 calibration guide Universal Manipulation Interface (UMI) DROID DexUMI
Module 3 · Single-task imitation learning
Sep 24	Imitation learning and action chunking	Policies, states, observations, and Markov decision processes; Behavioral cloning; Covariate shift and compounding errors; Recovery data, DART, and DAgger; Policy evaluation and causal confusion; Action chunking and observation, prediction, and execution horizons; ACT architecture, training, and inference; Practical policy training and controller consistency; Temporal ensembling; and Tradeoffs in chunk length and policy context.	Lecture 04 — Imitation Learning, Action Chunking
Oct 1	SO-101 assembly and calibration (in-class hands-on session).	In-class build session: arm assembly, motor configuration, calibration, camera checks, and basic safety.
Oct 8	Generative policies	Multimodality in robot learning; diffusion and flow matching; diffusion-model training and sampling; Diffusion Policy and conditioning; preprocessing, normalization, and closed-loop evaluation; relevant research on diffusion policies; action-chunk inference, prediction and execution horizons, real-time chunking, continuity, and reactivity.	Lecture 06 — Generative Policies	Week 06 interactive demos Denoising Diffusion Probabilistic Models (DDPM) Diffusion Policy 3D Diffusion Policy Equivariant Diffusion Policy CUPID: Curating Data your Robot Loves Diffusion Steering with Latent Space RL (DSRL) You’ve Got a Golden Ticket Real-Time Action Chunking with Large Models Smooth-As-Butter Robot Policies
Module 4 · Generalist robot learning
Oct 15	Multitask policy learning and vision-language-action (VLA) models	Generalization and multitask policy learning; task-conditioned diffusion policies and Large Behavior Models; language-conditioned policies and early VLAs; representing heterogeneous robot data; action tokenization (FAST, VQ-BeT, OAT) and continuous action representations; Octo and π₀; learning from multiple data sources with GR00T; latent actions, simulation and video-based data augmentation; catastrophic forgetting and knowledge insulation; π_0.5 and open-world generalization.	Lecture 07a — Multitask Policy Learning and VLAs	Large Behavior Models for Multitask Dexterous Manipulation RT-2 Action tokenization: FAST, VQ-BeT, OAT Octo π₀ GR00T: N1, N1.5 Knowledge insulation π_0.5: Open-world generalization
Oct 22	No class	Fall Break.
Oct 29	VLA models, continued	Continued discussion of VLAs and multitask policies.
Nov 5	Hands-on policy learning with LeRobot; Assignment 1 in-class demos	Course-staff tutorial on policy-learning best practices: dataset preparation, training, checkpointing, deployment, debugging, and closed-loop evaluation. Assignment 1 demos of handcrafted manipulation skills.		Assignment 1
Nov 12	Assignment 2 in-class demos — single-task policies	Demonstrate learned single-task policies; discuss task success, failure cases, and lessons from data collection, training, and deployment.		Assignment 2
Nov 19	Coding agents as policies	Code as Policies and variants such as CaP-X; ENPIRE and related systems.
Nov 26	No class	Fall recess.
Dec 3	World models and policy learning	Learned dynamics and world models in robotics; action-centric versus goal-centric frameworks; policy learning, planning, prediction, and action extraction; reward and value models; and evaluation.
Dec 10	Assignment 3 / final demos	Multi-task policy evaluations, demos, and short presentations.		Assignment 3

Assignments

The course has four assignments. There are no written exams. Assignment 1–3 handouts are available below as webpages and PDFs.

Assignment	Scope	Materials	Weight
A0 · Build and calibrate	Assemble the SO-101, configure the motors, calibrate the arm, and document a working setup.	—	5%
A1 · Handcrafted manipulation skill	Implement and evaluate a manipulation skill using an explicit perception-and-control pipeline.	HTML · PDF	10%
A2 · Single-task imitation learning	Collect demonstrations, train a task-specific policy, deploy it, and analyze its behavior.	HTML · PDF	30%
A3 · Multi-task robot manipulation solutions	Build a unified system that performs multiple tasks; demonstrate, explain, and analyze how it works.	HTML · PDF	35%

Grading

A0 · Build and calibrate	5%
A1 · Handcrafted manipulation skill	10%
A2 · Single-task imitation learning	30%
A3 · Multi-task robot manipulation solutions	35%
Participation, team contribution, and individual assessment	20%
Total	100%

Assignments will be carried out in teams of three. The final grade will reflect course participation, your contribution to the team, and an individual assessment.

Grading will consider the quality of the robot setup you produce, evaluation metrics for robot tasks, and an analysis of what worked well versus what needs improvement.