CS 330: Deep Multi-Task and Meta Learning, Fall 2023

Lectures: Mon, Wed 3:00pm – 4:20pm, Gates B1

Description

While deep learning has achieved remarkable success in many problems such as image classification, natural language processing, and speech recognition, these models are, to a large degree, specialized for the single task they are trained for. This course will cover the setting where there are multiple tasks to be solved, and study how the structure arising from multiple tasks can be leveraged to learn more efficiently or effectively. This includes:

self-supervised pre-training for downstream few-shot learning and transfer learning
meta-learning methods that aim to learn efficient learning algorithms that can learn new tasks quickly
curriculum and lifelong learning, where the problem requires learning a sequence of tasks, leveraging their shared structure to enable knowledge transfer

This is a graduate-level course. By the end of the course, students will be able to understand and implement the state-of-the-art multi-task learning and meta-learning algorithms and be ready to conduct research on these topics.

Format

The course will include in-person lectures (also livestreamed and recorded over Panopto on Canvas), four graded homework assignments, one optional homework assignment, and a course project. The lectures will discuss the fundamentals of topics required for understanding and designing multi-task and meta-learning algorithms in various domains. The assignments will focus on coding problems that emphasize these fundamentals. Finally, students will present their projects at the poster session at the end of the quarter.

Prerequisites

Machine learning: CS229 or equivalent is a prerequisite. We will be assuming knowledge of concepts including, but not limited to (stochastic) gradient descent and cross-validation, and pre-requisites such as probability theory, multivariable calculus, and linear algebra.

Some familiarity with deep learning: The course will build on deep learning concepts such as backpropagation, convolutional networks, and recurrent neural networks. The assignments will involve programming in PyTorch. The first week will include a short PyTorch review tutorial.

Lecture Videos

For the current offering, recorded lecture videos are posted to Canvas after each lecture. Publicly available recordings of previous offerings are available below.

Previous Offerings

Staff

Prof. Chelsea Finn

Instructor

OH: Mon 4:30 – 6pm

Location: Gates 358

Amelie Byun

Course Manager

Ansh Khurana

Head Teaching Assistant

OH: Fri 2 – 4pm

Location: Thornton 209

Alex Sun

Teaching Assistant

OH: Sun 2 – 4pm

Location: Zoom (link on Canvas)

Yoonho Lee

Teaching Assistant

OH: Tue 3 – 5pm

Location: Bldg. 200, Room 013

Max Sobol Mark

Teaching Assistant

OH: Wed 6 – 8pm

Location: Zoom (link on Canvas)

Course Schedule and Materials

Date	Lecture	Deliverables	Notes and Optional Reading
Week 1 Wed, Sep 27	Lecture Course introduction & start of multi-task learning	Homework 0 out [PDF, code, template]
Week 1 Thur, Sep 28	TA session PyTorch tutorial		Tutorial is 3:00 – 4:20pm at Gates B3. See Canvas for Zoom link. [Colab notebook]
Week 2 Mon, Oct 2	Lecture Multi-task learning		P1: Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. Kendall et al. (2018) P2: Universal Language Model Fine-tuning for Text Classification. Howard et al. (2018)
Week 2 Wed, Oct 4	Lecture Transfer learning & fine-tuning	Due Homework 0 Homework 1 out [PDF, code, template]	Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (Kumar et al., 2022)
Week 3 Mon, Oct 9	Lecture Black-box meta-learning & in-context learning		One-shot Learning with Memory-Augmented Neural Networks (Santoro et al., 2016)
Week 3 Wed, Oct 11	Lecture Optimization-based meta-learning	Due Project survey Homework 2 out [PDF, code, template ]	Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (Finn et al., 2017)
Week 3 Thur, Oct 12	TA session MAML		Tutorial is 4:30 – 5:20pm at Gates B1. See Canvas for Zoom link.
Week 4 Mon, Oct 16	Lecture Few-shot learning via metric learning	Due Homework 1	Matching Networks for One Shot Learning (Vinyals et al., 2017) Prototypical Networks for Few-shot Learning (Snell et al., 2017)
Week 4 Wed, Oct 18	Lecture Unsupervised pre-training for few-shot learning (contrastive)		Representation Learning with Contrastive Predictive Coding (van den Oord et al., 2019) A Simple Framework for Contrastive Learning of Visual Representations (Chen et al., 2020)
Week 5 Mon, Oct 23	Lecture Unsupervised pre-training for few-shot learning (generative)	Due Project proposal	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2019)
Week 5 Wed, Oct 25	Lecture Advanced meta-learning topics (task construction)	Due Homework 2 Homework 3 out [PDF, code , template]	Meta-Learning without Memorization (Yin et al., 2020)
Week 6 Mon, Oct 30	Lecture Variational inference		Tutorial on Variational Autoencoders (Doersch, 2021)
Week 6 Wed, Nov 1	Lecture Bayesian meta-learning
Week 6 Thur, Nov 2	TA session Bayesian Meta Learning		Tutorial is 4:30 – 5:20pm at Gates B1. See Canvas for Zoom link.
Week 7 Mon, Nov 6	Lecture Advanced meta-learning topics (large-scale meta-optimization)	Due Homework 3 Homework 4 out [PDF, template/code] (optional)	Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies (Vicol et al., 2021)
Week 7 Wed, Nov 8	Guest Lecture James Harrison	Learned Optimizers
Week 8 Mon, Nov 13	Lecture Lifelong learning
Week 8 Wed, Nov 15	Lecture No Lecture	Due Project milestone Due Homework 4 (optional)
Week 9 Mon, Nov 20	Thanksgiving break
Week 9 Wed, Nov 22	Thanksgiving break
Week 10 Mon, Nov 27	Guest lecture Jason Wei
Week 10 Wed, Nov 29	Lecture Domain Adaptation and Domain Generalization		Domain-Adversarial Training of Neural Networks (Ganin et al., 2016)
Week 11 Mon, Dec 4	Lecture Frontiers & Open Challenges
Week 11 Wed, Dec 6	Presentations Project poster session		The poster session will be held at Packard Atrium, from 3:00 – 6:00 pm.
Week 12 Mon, Dec 11	No class	Due Final project report

Course Calendar

A course calendar with details of lectures, TA sessions, office hours, and miscellaneous course events is available in a variety of formats:

Grading and Course Policies

Homeworks (50%): There are four graded homework assignments: Homework 0 is a warmup worth 5%, and Homeworks 1 – 3 are each worth 15%. Assignments will require training neural networks in PyTorch. Homework 4 is optional and will either replace one of Homeworks 1 – 3 or part of the project grade (whichever is better for your grade). All assignments are due on Gradescope at 11:59 pm Pacific Time on the respective due date.

Project (50%): This a research-level project of your choice. You may form groups of 1 – 3 students to complete the project, and you are encouraged to start early! Detailed guidelines on the project can be found here. You can use this template for your project proposal. Some past year projects are listed here for reference.

Extra Credit (up to 2%): Throughout the course, there are several participation opportunities that allow students to earn extra credit. The total extra credit that can be earned for the course is capped at 2%, and opportunities for this extra credit include answering questions on Ed and participating in live lecture polls. SCPD students will be able to submit the poll after the lecture. Here is the breakdown for extra credit opportunities:

Live lecture polls - 0.15% for each attempted poll in the class. SCPD students would be able to submit the poll by 11:59 PM the next day.
Guest lecture polls - 0.25% for each attempted poll in the class. SCPD students would be able to submit the poll by 11:59 PM the next day.
Ed participation - students with > 2 endorsed answers would be able to earn extra credit as per the following scheme:
- The top answerer will earn 2% extra credit
- The top 10th percentile of answerers will earn 1.5% extra credit
- The top 15th percentile of answerers will earn 1% extra credit
- The top 20th percentile of answerers will earn 0.5% extra credit

Late Days: You have 6 total late days across homework and project deliverables (anything worth a grade), except for the project poster. You may use a maximum of 2 late days for any single deliverable. Late days used for group projects apply to all members of the group.

Lecture Attendance: While we do not require lecture attendance, students are encouraged to join the live lecture. Students can also earn extra credit by attending lectures. To accommodate various circumstances, we will be live-streaming the in-person lecture via Panopto on Canvas. For those who cannot join the live lectures, lecture recordings will also be available on Canvas shortly following the lecture.

Honor Code: Students are free to form study groups and may discuss homework in groups. However, each student must write down the solutions and code from scratch independently and without referring to any written notes from group sessions. When debugging code together, students may only look at the input-output behavior of each other's programs and not their code itself. In other words, each student must understand the solution well enough in order to reconstruct it by themself. It is an honor code violation to copy, refer to, or look at written or code solutions from a previous year, including but not limited to: official solutions from a previous year, solutions posted online, and solutions you or someone else may have written up in a previous year. Furthermore, it is an honor code violation to post your assignment solutions online, such as on a public git repo, or use AI-driven code completion tools such as GitHub Copilot. You are, however, allowed to use such tools for the project. For more details about honor code, see The Stanford Honor Code and The Stanford Honor Code Pertaining to CS Courses.

Academic Accommodations

If you need an academic accommodation based on a disability, please register with the Office of Accessible Education (OAE). Professional staff will evaluate your needs, support appropriate and reasonable accommodations, and prepare an Academic Accommodation Letter for faculty. To get started, or to re-initiate services, please visit oae.stanford.edu. If you already have an Academic Accommodation Letter, please send your letter to cs330-staff-aut2324@cs.stanford.edu. OAE Letters should be sent to us at the earliest possible opportunity so that the course staff can partner with you and OAE to make the appropriate accommodations.

Note on Financial Aid

All students should retain receipts for books and other course-related expenses as these may be qualified educational expenses for tax purposes. If you are an undergraduate receiving financial aid, you may be eligible for additional financial aid for required books and course materials if these expenses exceed the aid amount in your award letter. For more information, review your award letter or visit the Student Budget website.

Notice on Lecture Recording

Video cameras located in the back of the room will capture the instructor presentations in this course. For your convenience, you can access these recordings by logging into the course Canvas site. These recordings might be reused in other Stanford courses, viewed by other Stanford students, faculty, or staff, or used for other education and research purposes. Note that while the cameras are positioned with the intention of recording only the instructor, occasionally a part of your image or voice might be incidentally captured. If you have questions, please contact a member of the teaching team.