Schedule (tentative)

Here is a tentative syllabus for the course. Readings will be filled in over time.

Recommended Textbooks

  1. RL: Reinforcement Learning: An Introduction. Richard S. Sutton and Andrew G. Barto. Second Edition, MIT Press, Cambridge, MA, 2018. Available online.

  2. CV: Computer Vision: Algorithms and Applications. Richard Szeliski, Microsoft Research. Available online.

  3. RKH: Robotic Systems. Kris Hauser. Draft available online.

  4. MR: Modern Robotics: Mechanics, Planning, and Control. Frank C. Park, Kevin M. Lynch. Cambridge University Press. Available Online.

Date Topic Slides Readings Assignments
Aug 25 Introduction pdf  
Part I Review
Aug 27 Computer Vision Review pdf, pptx CV Chapters 4, 7, 14  
Sep 1 Computer Vision Review pdf, pptx  
Assignment 1
Sep 3 Robotics Review pdf RKH Chapters 5, 6, 10, 17  
Sep 8 MDP Review [Dynamic Programming] pdf RL Chapters 3, 4  
Sep 10 MDP Review [Monte Carlo, TD] David Silver's Slides RL Chapters 5, 6  
Sep 15 MDP Review [Model Free Control] David Silver's Slides RL Chapters 5, 6.
Assignment 1 Due
Sep 15, 2020 at 11:59:59 PM
Sep 17 Q-Learning pdf, keyDDPG
Sep 22 Policy Gradients David Silver's Slides ACKTR
Assignment 2
Part II Alternatives to Solving Unknown MDPs
Sep 24 Model Building pdf, key PILCO
Sep 29 Model Building pdf, key Deep Visual Foresight
Benchmarking MBRL
Oct 1 Imitation Learning pdf, key DAgger
E2E Visuomotor Policies
Oct 6 Inverse RL pdf, key Inverse RL
Apprenticeship Learning
Assignment 2 Due
Oct 6, 2020 at 11:59:59 PM
Oct 8 Inverse RL pdf, key MaxEnt IRL Project Proposals Due
Oct 8, 2020 at 11:59:59 PM
Informal Early Feedback
Oct 13 Self-supervision pdf, key Self-supervised Grasping
Self-supervised Pushing and Grasping
Assignment 3
Oct 15 Exploration notes Curiosity
Planning to Explore
Oct 20 Sim to Real pdf, key ANYmal  
Oct 22 Hierarchies pdf, key Feudal RL  
Oct 27 Social Learning pdf, key Grasping in the wild
Navigation Subroutines
Assignment 3 Due
Oct 28, 2020 at 11:59:59 PM
Oct 29 Social Learning pdf, key
pdf, key
Value Learning from Videos
Perceptual Rewards
Nov 3 Election Day (No class)  
Nov 5 Differentiable Planners pdf, key Differentiable MPC Project Progress Report Due
Nov 5, 2020 at 11:59:59 PM
Part III Case Studies
Nov 10 Navigation pdf, key Agile Autonomous Driving  
Nov 12 Navigation pdf, key CMP
Neural Topological SLAM
Nov 17 Manipulation pdf, key Dex-net 2.0  
Nov 19 Manipulation pdf, key Re-grasping using Touch
Hierarchical Object-Centric Controllers
Nov 24 Fall Break (No class)  
Nov 26 Fall Break (No class)  
Part IV Perspectives
Dec 1 Lessons from Cognitive Science and Psychology pdf, pptx 6 Lessons
Lake et al. 2017
Dec 3 Data vs Algorithms  
The Bitter Lesson+A Better Lesson ICES
Project Final Report Due
Dec 6, 2020 at 11:59:59 PM
Dec 8 Project Presentations  

PILCO: A model-based and data-efficient approach to policy search
Marc Deisenroth and Carl Rasmussen
ICML 2011

Deep visual foresight for planning robot motion
Chelsea Finn and Sergey Levine
ICRA 2017

Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Kurtland Chua, Roberto Calandra, Rowan McAllister, Sergey Levine
NeurIPS 2018

Benchmarking model-based reinforcement learning
Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba
arXiv preprint arXiv:1907.02057 2019

A reduction of imitation learning and structured prediction to no-regret online learning
Stephane Ross, Geoffrey Gordon, Drew Bagnell

End-to-end training of deep visuomotor policies
Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel
JMLR 2016

Maximum entropy inverse reinforcement learning.
Brian Ziebart, Andrew Maas, J Bagnell, Anind Dey
AAAI 2008

Apprenticeship learning via inverse reinforcement learning
Pieter Abbeel and Andrew Ng
ICML 2004

The development of embodied cognition: Six lessons from babies
Linda Smith and Michael Gasser
Artificial life 2005

Building machines that learn and think like people
Brenden Lake, Tomer Ullman, Joshua Tenenbaum, Samuel Gershman
Behavioral and brain sciences 2017

Playing atari with deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller
arXiv preprint arXiv:1312.5602 2013


Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation
Yuhuai Wu, Elman Mansimov, Roger Grosse, Shun Liao, Jimmy Ba
NeurIPS 2017

Continuous control with deep reinforcement learning
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra
ICLR 2016

Off-policy deep reinforcement learning without exploration
Scott Fujimoto, David Meger, Doina Precup
ICML 2019

Learning agile and dynamic motor skills for legged robots
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, Marco Hutter
Science Robotics 2019

Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours
Lerrel Pinto and Abhinav Gupta
ICRA 2016

Learning synergies between pushing and grasping with self-supervised deep reinforcement learning
Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, Thomas Funkhouser
IROS 2018

Planning to Explore via Self-Supervised World Models
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, Deepak Pathak
ICML 2020

Curiosity-driven exploration by self-supervised prediction
Deepak Pathak, Pulkit Agrawal, Alexei Efros, Trevor Darrell
ICML 2017

Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations
Shuran Song, Andy Zeng, Johnny Lee, Thomas Funkhouser
RAL 2020

Learning Navigation Subroutines from Egocentric Videos
Ashish Kumar, Saurabh Gupta, Jitendra Malik
CoRL 2019

Differentiable MPC for end-to-end planning and control
Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, J Kolter
NeurIPS 2018

Differntiable Spatial Planning using Transformers
Anonymous Anonymous
Open Review 2020

Neural Topological SLAM for Visual Navigation
Devendra Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, Saurabh Gupta
CVPR 2020

More than a feeling: Learning to grasp and regrasp using vision and touch
Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward Adelson, Sergey Levine
RAL 2018

Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics
Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Ojea, Ken Goldberg
RSS 2017

Semantic Visual Navigation by Watching YouTube Videos
Matthew Chang, Arjun Gupta, Saurabh Gupta
NeurIPS 2020

Algorithms for inverse reinforcement learning.
Andrew Ng and Stuart Russell
ICML 2000

Feudal reinforcement learning
Peter Dayan and Geoffrey Hinton
NeurIPS 1993

Unsupervised perceptual rewards for imitation learning
Pierre Sermanet, Kelvin Xu, Sergey Levine
Robotics: Science and Systems 2017

Agile autonomous driving using end-to-end deep imitation learning
Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, Byron Boots
Robotics: Science and Systems 2018

Cognitive mapping and planning for visual navigation
Saurabh Gupta, Varun Tolani, James Davidson, Sergey Levine, Rahul Sukthankar, Jitendra Malik
International Journal of Computer Vision 2019

Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation
Mohit Sharma, Jacky Liang, Jialiang Zhao, Alex LaGrassa, Oliver Kroemer
CoRL 2020

The Bitter Lesson
Rich Sutton
Blogpost 2019

A Better Lesson
Rodney Brooks
Blogpost 2019