While deep learning has achieved remarkable success in many problems such as image classification, natural language processing, and speech recognition, these models are, to a large degree, specialized for the single task they are trained for. This course will cover the setting where there are multiple tasks to be solved, and study how the structure arising from multiple tasks can be leveraged to learn more efficiently or effectively. This includes:
This is a graduate-level course. By the end of the course, students will be able to understand and implement the state-of-the-art multi-task learning and meta-learning algorithms and be ready to conduct research on these topics.
The course will include in-person lectures (also livestreamed and recorded over Panopto on Canvas), four
graded homework assignments, one optional homework assignment, and a course project. The lectures will
discuss the fundamentals of topics required for understanding and
designing multi-task and meta-learning algorithms in various domains. The assignments will focus on coding problems that emphasize
these fundamentals. Finally, students will present their projects at the poster session at
the end of the quarter.
Machine learning: CS229 or equivalent is a prerequisite. We will be assuming knowledge of concepts including, but not limited to (stochastic) gradient descent and cross-validation, and pre-requisites such as probability theory, multivariable calculus, and linear algebra.
Some familiarity with deep learning: The course will build on deep learning concepts such as backpropagation, convolutional networks, and recurrent neural networks. The assignments will involve programming in PyTorch. The first week will include a short PyTorch review tutorial.
For the current offering, recorded lecture videos are posted to Canvas after each lecture. Publicly available recordings of previous offerings are available below.
Date | Lecture | Deliverables | Notes and Optional Reading |
---|---|---|---|
Week 1 Wed, Sep 27 |
Lecture Course introduction & start of multi-task learning |
Homework 0 out
[PDF, code, template] |
|
Week 1 Thur, Sep 28 |
TA session PyTorch tutorial |
|
|
Week 2 Mon, Oct 2 |
Lecture Multi-task learning |
|
|
Week 2 Wed, Oct 4 |
Lecture Transfer learning & fine-tuning |
Due Homework 0 Homework 1 out [PDF, code, template] |
|
Week 3 Mon, Oct 9 |
Lecture Black-box meta-learning & in-context learning |
|
|
Week 3 Wed, Oct 11 |
Lecture Optimization-based meta-learning |
Due Project survey Homework 2 out [PDF, code, template ] |
|
Week 3 Thur, Oct 12 |
TA session MAML |
|
|
Week 4 Mon, Oct 16 |
Lecture Few-shot learning via metric learning |
Due Homework 1 |
|
Week 4 Wed, Oct 18 |
Lecture Unsupervised pre-training for few-shot learning (contrastive) |
|
|
Week 5 Mon, Oct 23 |
Lecture Unsupervised pre-training for few-shot learning (generative) |
Due Project proposal |
|
Week 5 Wed, Oct 25 |
Lecture Advanced meta-learning topics (task construction) |
Due Homework 2 Homework 3 out [PDF, code , template] |
|
Week 6 Mon, Oct 30 |
Lecture Variational inference |
|
|
Week 6 Wed, Nov 1 |
Lecture Bayesian meta-learning |
|
|
Week 6 Thur, Nov 2 |
TA session Bayesian Meta Learning |
|
|
Week 7 Mon, Nov 6 |
Lecture Advanced meta-learning topics (large-scale meta-optimization) |
Due Homework 3 Homework 4 out [PDF, template/code] (optional) |
|
Week 7 Wed, Nov 8 |
Guest Lecture James Harrison | Learned Optimizers |
|
Week 8 Mon, Nov 13 |
Lecture Lifelong learning | ||
Week 8 Wed, Nov 15 |
Lecture No Lecture |
Due Project milestone Due Homework 4 (optional) |
|
Week 9 Mon, Nov 20 |
Thanksgiving break | ||
Week 9 Wed, Nov 22 |
Thanksgiving break | ||
Week 10 Mon, Nov 27 |
Guest lecture Jason Wei | ||
Week 10 Wed, Nov 29 |
Lecture Domain Adaptation and Domain Generalization |
|
|
Week 11 Mon, Dec 4 |
Lecture Frontiers & Open Challenges | ||
Week 11 Wed, Dec 6 |
Presentations Project poster session |
|
|
Week 12 Mon, Dec 11 |
No class |
Due Final project report |
A course calendar with details of lectures, TA sessions, office hours, and miscellaneous course events is available in a variety of formats:
Homeworks (50%): There are four graded homework assignments: Homework 0 is a warmup worth 5%, and Homeworks 1 – 3 are each worth 15%. Assignments will require training neural networks in PyTorch. Homework 4 is optional and will either replace one of Homeworks 1 – 3 or part of the project grade (whichever is better for your grade). All assignments are due on Gradescope at 11:59 pm Pacific Time on the respective due date.
Project (50%): This a research-level project of your choice. You may form groups of 1 – 3 students to complete the project, and you are encouraged to start early! Detailed guidelines on the project can be found here. You can use this template for your project proposal. Some past year projects are listed here for reference.
Extra Credit (up to 2%): Throughout the course, there are several participation opportunities that allow students to earn extra credit. The total extra credit that can be earned for the course is capped at 2%, and opportunities for this extra credit include answering questions on Ed and participating in live lecture polls. SCPD students will be able to submit the poll after the lecture. Here is the breakdown for extra credit opportunities:
Late Days: You have 6 total late days across homework and project deliverables (anything worth a grade), except for the project poster. You may use a maximum of 2 late days for any single deliverable. Late days used for group projects apply to all members of the group.
Lecture Attendance: While we do not require lecture attendance, students are encouraged to join the live lecture. Students can also earn extra credit by attending lectures. To accommodate various circumstances, we will be live-streaming the in-person lecture via Panopto on Canvas. For those who cannot join the live lectures, lecture recordings will also be available on Canvas shortly following the lecture.
Honor Code: Students are free to form study groups and may discuss homework in groups. However, each student must write down the solutions and code from scratch independently and without referring to any written notes from group sessions. When debugging code together, students may only look at the input-output behavior of each other's programs and not their code itself. In other words, each student must understand the solution well enough in order to reconstruct it by themself. It is an honor code violation to copy, refer to, or look at written or code solutions from a previous year, including but not limited to: official solutions from a previous year, solutions posted online, and solutions you or someone else may have written up in a previous year. Furthermore, it is an honor code violation to post your assignment solutions online, such as on a public git repo, or use AI-driven code completion tools such as GitHub Copilot. You are, however, allowed to use such tools for the project. For more details about honor code, see The Stanford Honor Code and The Stanford Honor Code Pertaining to CS Courses.
If you need an academic accommodation based on a disability, please register with the Office of Accessible Education (OAE). Professional staff will evaluate your needs, support appropriate and reasonable accommodations, and prepare an Academic Accommodation Letter for faculty. To get started, or to re-initiate services, please visit oae.stanford.edu. If you already have an Academic Accommodation Letter, please send your letter to cs330-staff-aut2324@cs.stanford.edu. OAE Letters should be sent to us at the earliest possible opportunity so that the course staff can partner with you and OAE to make the appropriate accommodations.
All students should retain receipts for books and other course-related expenses as these may be qualified educational expenses for tax purposes. If you are an undergraduate receiving financial aid, you may be eligible for additional financial aid for required books and course materials if these expenses exceed the aid amount in your award letter. For more information, review your award letter or visit the Student Budget website.
Video cameras located in the back of the room will capture the instructor presentations in this course. For your convenience, you can access these recordings by logging into the course Canvas site. These recordings might be reused in other Stanford courses, viewed by other Stanford students, faculty, or staff, or used for other education and research purposes. Note that while the cameras are positioned with the intention of recording only the instructor, occasionally a part of your image or voice might be incidentally captured. If you have questions, please contact a member of the teaching team.
© Chelsea Finn 2023