CS 330: Deep Multi-Task and Meta Learning

Fall 2021, Class: Mon, Wed 11:30am-1:00pm, NVIDIA Auditorium

Description:

While deep learning has achieved remarkable success in supervised and reinforcement learning problems, such as image classification, speech recognition, and game playing, these models are, to a large degree, specialized for the single task they are trained for. This course will cover the setting where there are multiple tasks to be solved, and study how the structure arising from multiple tasks can be leveraged to learn more efficiently or effectively. This includes:

goal-conditioned reinforcement learning techniques that leverage the structure of the provided goal space to learn many tasks significantly faster
meta-learning methods that aim to learn efficient learning algorithms that can learn new tasks quickly
curriculum and lifelong learning, where the problem requires learning a sequence of tasks, leveraging their shared structure to enable knowledge transfer

This is a graduate-level course. By the end of the course, students will be able to understand and implement the state-of-the-art multi-task learning and meta-learning algorithms and be ready to conduct research on these topics.

Format:

The course will include in-person lectures (also livestreamed and recorded over zoom), three graded homework assignments, two optional homework assignments, and a course project. The lectures will discuss the fundamentals of topics required for understanding and designing multi-task and meta-learning algorithms in both supervised learning and reinforcement learning domains. The assignments will focus on coding problems that emphasize these fundamentals. Finally, students will present their projects at the poster session at the end of the quarter.

Prerequisites:

CS 229 or an equivalent introductory machine learning course is required. CS 221 or an equivalent introductory artificial intelligence course is recommended but not required.

Lecture Videos:

If you are looking for publicly-available lecture videos from the Fall 2019 offering, they are here. Materials from the Fall 2019 offering are here and materials from the Fall 2020 offering are here. Lecture videos from this Fall 2021 offering will be processed and made publicly available after the course. For students enrolled in the course, recorded lecture videos will be posted to canvas after each lecture.

Staff

Prof. Chelsea Finn

Instructor

OH: Wed 1-2:30pm

Webpage

Dr. Karol Hausman

Instructor

Webpage

Amelie Byun

Course Coordinator

Rafael Rafailov

Rafael Rafailov

Head Teaching Assistant

OH: Thur 5:00-7:00pm

Dilip Arumugam

Teaching Assistant

OH: Fri 12:30-2:30pm

Archit Sharma

Teaching Assistant

OH: Thur 1:00-3:00pm

Evan Liu

Teaching Assistant

OH: Tue 3:00-5:00pm

Kevin Yu

Teaching Assistant

OH: Sun 3:00-5:00pm

Kyle Hsu

Teaching Assistant

OH: Tue 11:00am-1:00pm

Siyi Tang

Teaching Assistant

OH: Mon 1:00-3:00pm

Kamil Ali

Teaching Assistant

OH: Sun 1:00-3:00pm

Edwin Pan

Teaching Assistant

OH: Sat 5:00-7:00pm

Timeline

Date	Lecture	Deadlines	Optional reading
Week 1 Mon, Sep 20	Lecture Course introduction & start of multi-task learning (Chelsea Finn & Karol Hausman)	Optional Homework 0 out [PDF][Zip]
Week 1 Wed, Sep 22	Lecture Supervised multi-task learning, transfer learning (Chelsea Finn)		P1: Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. Kendall et al. (2018) P2: Universal Language Model Fine-tuning for Text Classification. Howard et al. (2018)
Week 1 Thur, Sep 23	TA Session PyTorch tutorial		Tutorial is at 6pm over zoom and will be recorded.
Week 2 Mon, Sep 27	Lecture Meta-learning problem statement, black-box meta-learning (Chelsea Finn)	Due Optional Homework 0 Homework 1 out [PDF][Zip]	P1:One-shot Learning with Memory-Augmented Neural Networks. Santoro et al. (2016)
Week 2 Wed, Sep 29	Lecture Optimization-based meta-learning (Chelsea Finn)		P1: Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Finn et al. (2017) P2: Meta-Learning with Differentiable Convex Optimization. Lee et al. (2019)
Week 3 Mon, Oct 4	Lecture Few-shot learning via metric learning (Chelsea Finn)		P1: Matching Networks for One Shot Learning. Vinyals et al. (2017) P2: Prototypical Networks for Few-shot Learning. Snell et al. (2017)
Week 3 Wed, Oct 6	Lecture Advanced meta-learning topics (Chelsea Finn)	Due Homework 1 Homework 2 out[PDF][Zip]	P1: Meta-Learning without Memorization. Yin et al. (2020)
Week 3 Thur, Oct 7	TA Session Variational inference tutorial		Tutorial is at 4pm over zoom and will be recorded.
Week 4 Mon, Oct 11	Lecture Bayesian meta-learning (Chelsea Finn)		P1: Conditional Neural Processes. Garnelo et al. (2018) P2: Meta-Learning Probabilistic Inference For Prediction. Gordon et al. (2019)
Week 4 Tue, Oct 12	TA Session Reinforcement learning tutorial		Tutorial is at 5pm over zoom and will be recorded.
Week 4 Wed, Oct 13	Leacture RL review (Karol Hausman)	Due Project proposal
Week 5 Mon, Oct 18	Lecture Multi-task RL and goal-conditioned RL (Karol Hausman)	Due Homework 2 Homework 3 out[PDF][Zip]	P1: Hindsight Experience Replay. Andrychowicz et al. (2018) P2: MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale. Kalashnikov et al. (2021)
Week 5 Wed, Oct 20	Lecture Model-based RL for multi-task learning (Chelsea Finn)		P1: Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. Ebert et al. (2018) P2: Deep Dynamics Models for Learning Dexterous Manipulation. Nagabandi et al. (2019)
Week 6 Mon, Oct 25	Lecture Meta-RL 1: Adaptable models and policies (Chelsea Finn)		P1: RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. Duan et al. (2016)
Week 6 Wed, Oct 27	Lecture Meta-RL 2: Learning to explore (Chelsea Finn)	Due Homework 3 Optional Homework 4 out[PDF][Zip]	P1: Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables. Rakelly et al. (2019) P2: Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices. Liu et al. (2021)
Week 7 Mon, Nov 1	Lecture Offline RL and offline MTRL (Karol Hausman)		P1: Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Levine et al. (2020) P2: Conservative Q-Learning for Offline Reinforcement Learning. Kumar et al. (2020) P3: Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills. Chebotar et al. (2021)
Week 7 Wed, Nov 3	Lecture Hierarchical RL and skill discovery (Karol Hausman)		P1: Data-Efficient Hierarchical Reinforcement Learning. Nachum et al. (2018) P2: Diversity is All You Need: Learning Skills without a Reward Function. Eysenbach et al. (2018) P3: Dynamics-Aware Unsupervised Discovery of Skills. Sharma et al. (2019)
Week 8 Mon, Nov 8	Guest Lecture What do language models learn from language modeling? (Colin Raffel (UNC, Hugging Face))	Due Optional Homework 4	Lecture is from 12-1pm (virtual).
Week 8 Wed, Nov 10	Guest Lecture Learned optimizers: why they're the future, why they're hard, and what they can do now (Jascha Sohl-Dickstein (Google))	Due Project milestone
Week 9 Mon, Nov 15	Lecture Lifelong learning: problem statements, forward & backward transfer (Karol Hausman)		P1: Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning. Julian et al. (2020) P2: Online Meta-Learning. Finn et al. (2019) Lecture will be over zoom and recorded.
Week 9 Wed, Nov 17	Lecture Frontiers and open problems (Karol Hausman)		Guest speaker slides: Learning to edit pre-trained models. (Eric Mitchell) BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning. (Eric Jang) Language Conditioned Imitation Learning Over Unstructured Data. (Corey Lynch) Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation. (Suraj Nair) Autonomous Reinforcement Learng. (Archit Sharma) Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos. (Annie Chen) Reinforcement Learning as Sequence Modeling. (Igor Mordatch) Lifelong Robotic Reinforcement Learning by Retaining Experiences. (Annie Xie) Lecture will be over zoom and recorded.
Week 10 Mon, Nov 22	Thanksgiving Break
Week 10 Wed, Nov 24	Thanksgiving Break
Week 11 Mon, Nov 29	No Class
Week 11 Tue, Nov 30	Presentations Project poster session		The poster session will be held in McCaw Hall at the Alumni Center from 11:30am-1:30pm.
Week 12 Wed, Dec 8	No Class	Due Final project report

Course Calendar

Course calendar with details of lectures, TA sessions, and office hours is available here.

Grading and Course Policies

Homeworks (15% each): There are three graded homework assignments (Homeworks 1-3), each worth 15% of the grade. Assignments will require training neural networks in PyTorch. Homework 0 is optional and is worth 2% extra grade. Homework 4 is optional and will either replace one prior homework grade or part of the project grade (whichever is better for grade). All assignments are due on Gradescope at 11:59 pm Pacific Time on the respective due date.

Project (55%): There's a research-level project of your choice. You may form groups of 1-3 students to complete the project, and you are encouraged to start early! Detailed guidelines on the project can be found here.

Late Days: You have 6 total late days across homeworks and project proposal submissions. You may use a maximum of 2 late days for any single assignment. Late days are not allowed for final project report.

Lecture Attendance: While we do not require lecture attendance, students are encouraged to join the live lecture. To accommodate various circumstances, we will be live-streaming the in-person lecture via a zoom link on canvas. Students joining the zoom live stream will be able to ask questions. For those who cannot join the live lectures, lecture recordings will also be available on Canvas shortly following the lecture.

Honor Code: Students are free to form study groups and may discuss homework in groups. However, each student must write down the solutions and code from scratch independently, and without referring to any written notes from the joint session. When debugging code together, you are only allowed to look at the input-output behavior of each other's programs and not the code itself. In other words, each student must understand the solution well enough in order to reconstruct it by him/herself. It is an honor code violation to copy, refer to, or look at written or code solutions from a previous year, including but not limited to: official solutions from a previous year, solutions posted online, and solutions you or someone else may have written up in a previous year. Furthermore, it is an honor code violation to post your assignment solutions online, such as on a public git repo. For more details about honor code, see The Stanford Honor Code and The Stanford Honor Code Pertaining to CS Courses.

Health and Safety Expectation (COVID-19 Protocols)

Following Stanford’s policies, everyone is required to wear a mask indoors, regardless of vaccination status. This includes any in-person lectures or office hour sessions. Some community members may have preferences that go beyond the requirements; it is important that we treat each others' preferences with respect and care. You can find the most current policies on campus masking requirements on the COVID-19 Health Alerts site.

Academic Accommodations

If you need an academic accommodation based on a disability, please register with the Office of Accessible Education (OAE). Professional staff will evaluate your needs, support appropriate and reasonable accommodations, and prepare an Academic Accommodation Letter for faculty. To get started, or to re-initiate services, please visit oae.stanford.edu. If you already have an Academic Accommodation Letter, please send your letter to cs330-aut2122-staff@lists.stanford.edu. OAE Letters should be sent to us at the earliest possible opportunity so that the course staff can partner with you and OAE to make the appropriate accommodations.

Note on Financial Aid

All students should retain receipts for books and other course-related expenses, as these may be qualified educational expenses for tax purposes. If you are an undergraduate receiving financial aid, you may be eligible for additional financial aid for required books and course materials if these expenses exceed the aid amount in your award letter. For more information, review your award letter or visit the Student Budget website.

© Chelsea Finn 2021