Hands-on Robot Learning
Johns Hopkins University · Fall 2026
Hands-on Robot Learning
This course provides a hands-on introduction to the design and implementation of modern learning-based robotic systems. Students will work in small teams to assemble and operate an SO-101 arm, collect robot data, and develop the software needed to train and deploy learned policies.
- Course
- EN.601.498 / EN.601.698
- Instructor
- Krishna Murthy Jatavallabhula
- Course assistants
- Xinyi Yin, Yinzhe Zhou, and Aleks Santari
- Lecture
- Ames 234, Thursdays 3 pm–5:30 pm
About
Robot learning has made significant strides in just the last 2 years. For the first time, there exist ‘generalist’ manipulation systems that one can download from the web, and deploy on their robots and environments without substantial finetuning, and achieve non-trivial task success rates. At the same time, making robot learning models (a.k.a. ‘policies’) work, and generalize across a fairly broad set of tasks and environments, is an open research problem. This has caused an undeniable shift in the skillsets that robotics practitioners (engineers, scientists) must now have, to get into (and succeed at) frontier robotics labs.
This course aims to address that gap. In this class, we will take a ‘full-stack’ perspective of robot learning – we will understand (and build) robot hardware (i.e., a small tabletop robot arm) and discuss the entirety of the modern robot learning lifecycle. This includes task design and selection, curating and processing data, training neural network models on the processed data, deploying these trained models, and evaluating how the trained approaches fare.
This is not an introductory class. We expect students to already have requisite knowledge in deep learning (training and debugging transformer models, diffusion models) or be able to pick them up independently. We will do an extremely short recap of the essentials for the class (one 2.5-hour lecture). We then will look at various data curation and processing strategies, single-task imitation learning, multi-task policy learning, and will discuss some of the frontiers (world models, for example).
Instructor
Krishna Murthy Jatavallabhula
Assistant Professor of Computer Science
Johns Hopkins University
Course goals
By the end of the course, students should be able to:
- Describe the modern landscape of data-driven robot learning and the common system designs used in robot learning.
- Assemble, calibrate, and safely operate a robotic manipulator built from readily available electromechanical components.
- Implement, train, and evaluate modern imitation learning approaches for simple manipulation tasks.
- Deploy a vision-language-action (VLA) model on a physical robot and characterize its success and failure modes.
- Compare the tradeoffs between policy architectures, low-level design choices, and data collection methods.
- Define evaluation metrics and failure taxonomies, and conduct reproducible robot experiments.
Course materials
Schedule
The schedule is tentative and may change as the semester develops.
| Date | Title | Topics | Slides | Resources |
|---|---|---|---|---|
| Module 1 · Introduction to robot learning | ||||
| Sep 3 | Introduction to modern robot learning | The data-driven robotics lifecycle; the state of the art in robot learning. | ||
| Sep 10 | Essentials (whirlwind recap) | Robot design and payload computations; smart servos and how they work; electromechanical components of the SO-101; anatomy of a robot manipulator; interpreting motor datasheets; joint-space and Cartesian-space representations; forward and inverse kinematics; Jacobians and singularities; position-based feedback control; PID control; and a quick recap of modern machine learning. | Slides | |
| Module 2 · Data for robot learning | ||||
| Sep 17 | Data for robot learning: teleop and other sources | Teleoperation interfaces; leader–follower systems; GELLO; AR/VR teleoperation; UMI and variants; and other sources of robot data. | ||
| Sep 24 | SO-101 assembly and calibration | In-class build session: arm assembly, motor configuration, calibration, camera checks, and basic safety. | ||
| Module 3 · Single-task imitation learning | ||||
| Oct 1 | Imitation learning and action chunking | Behavior cloning; ALOHA; Action Chunking Transformers. | ||
| Oct 8 | Generative policies | Multimodality; diffusion policies; flow matching. | ||
| Oct 15 | Policy inference stack | Asynchronous observation and action; latency; real-time action chunking; temporal ensembling; and more. | ||
| Oct 22 | No class | Fall Break. | ||
| Module 4 · Generalist robot learning | ||||
| Oct 29 | Introduction to vision-language-action (VLA) models | Early VLAs: the RT series, OpenVLA, and Octo; π0 and variants; bilevel (hierarchical) policies; Large Behavior Models; and more. | ||
| Nov 5 | VLA models continued | Action representations; tokenization; fine-tuning; inference. | ||
| Nov 12 | Coding agents as policies | Code as Policies and variants such as CaP-X; ENPIRE and related systems. | ||
| Nov 19 | World models and policy learning | World models in robotics; action-centric versus goal-centric world-modeling frameworks; planning and action extraction; exemplar frameworks; reward and value models. | ||
| Nov 26 | No class | Fall recess. | ||
| Module 5 · Evaluating robot models | ||||
| Dec 3 | Evaluating generalist robot models | Statistical significance testing; metrics for evaluating policies and world models. | ||
| Dec 10 | Final demos | Final policy evaluations, demos, and short presentations. | ||
Assignments
The course has four assignments. There are no written exams. Assignment details will be announced in class and posted both on Canvas and on this website.
| Assignment | Scope | Weight |
|---|---|---|
| A0 · Build and calibrate | Assemble the SO-101, configure the motors, calibrate the arm, and document a working setup. | 5% |
| A1 · Handcrafted manipulation skill | Implement and evaluate a manipulation skill using an explicit perception-and-control pipeline. | 10% |
| A2 · Single-task imitation learning | Collect demonstrations, train a task-specific policy, deploy it, and analyze its behavior. | 30% |
| A3 · Multi-task policy learning | Train or adapt a policy across multiple tasks and evaluate where transfer helps or fails. | 35% |
Grading
| A0 · Build and calibrate | 5% |
|---|---|
| A1 · Handcrafted manipulation skill | 10% |
| A2 · Single-task imitation learning | 30% |
| A3 · Multi-task policy learning | 35% |
| Participation, team contribution, and individual assessment | 20% |
| Total | 100% |
Assignments will be carried out in teams of three. The final grade will reflect course participation, your contribution to the team, and an individual assessment.
Grading will consider the quality of the robot setup you produce, evaluation metrics for robot tasks, and an analysis of what worked well versus what needs improvement.