CS 330: Deep Multi-Task and Meta Learning, Fall 2023

Lectures: Mon, Wed 3:00pm – 4:20pm, Gates B1



Description

While deep learning has achieved remarkable success in many problems such as image classification, natural language processing, and speech recognition, these models are, to a large degree, specialized for the single task they are trained for. This course will cover the setting where there are multiple tasks to be solved, and study how the structure arising from multiple tasks can be leveraged to learn more efficiently or effectively. This includes:

  • self-supervised pre-training for downstream few-shot learning and transfer learning
  • meta-learning methods that aim to learn efficient learning algorithms that can learn new tasks quickly
  • curriculum and lifelong learning, where the problem requires learning a sequence of tasks, leveraging their shared structure to enable knowledge transfer

This is a graduate-level course. By the end of the course, students will be able to understand and implement the state-of-the-art multi-task learning and meta-learning algorithms and be ready to conduct research on these topics.

Format

The course will include in-person lectures (also livestreamed and recorded over Panopto on Canvas), four graded homework assignments, one optional homework assignment, and a course project. The lectures will discuss the fundamentals of topics required for understanding and designing multi-task and meta-learning algorithms in various domains. The assignments will focus on coding problems that emphasize these fundamentals. Finally, students will present their projects at the poster session at the end of the quarter.

Prerequisites

Machine learning: CS229 or equivalent is a prerequisite. We will be assuming knowledge of concepts including, but not limited to (stochastic) gradient descent and cross-validation, and pre-requisites such as probability theory, multivariable calculus, and linear algebra.

Some familiarity with deep learning: The course will build on deep learning concepts such as backpropagation, convolutional networks, and recurrent neural networks. The assignments will involve programming in PyTorch. The first week will include a short PyTorch review tutorial.

Lecture Videos

For the current offering, recorded lecture videos are posted to Canvas after each lecture. Publicly available recordings of previous offerings are available below.

Previous Offerings


Staff

Chelsea Finn

Prof. Chelsea Finn

Instructor
OH: Mon 4:30 – 6pm
Location: Gates 358
Amelie Byun

Amelie Byun

Course Manager
Ansh Khurana

Ansh Khurana

Head Teaching Assistant
OH: Fri 2 – 4pm
Location: Thornton 209
Alex Sun

Alex Sun

Teaching Assistant
OH: Sun 2 – 4pm
Location: Zoom (link on Canvas)
Yoonho Lee

Yoonho Lee

Teaching Assistant
OH: Tue 3 – 5pm
Location: Bldg. 200, Room 013
Max Sobol Mark

Max Sobol Mark

Teaching Assistant
OH: Wed 6 – 8pm
Location: Zoom (link on Canvas)

Course Schedule and Materials

Date Lecture Deliverables  Notes and Optional Reading
Week 1
Wed, Sep 27
Lecture Course introduction & start of multi-task learning Homework 0 out
[PDF, code, template]
Week 1
Thur, Sep 28
TA session PyTorch tutorial
  • Tutorial is 3:00 – 4:20pm at Gates B3. See Canvas for Zoom link. [Colab notebook]
Week 2
Mon, Oct 2
Lecture Multi-task learning
Week 2
Wed, Oct 4
Lecture Transfer learning & fine-tuning Due Homework 0
Homework 1 out
[PDF, code, template]
Week 3
Mon, Oct 9
Lecture Black-box meta-learning & in-context learning
Week 3
Wed, Oct 11
Lecture Optimization-based meta-learning Due Project survey
Homework 2 out [PDF, code, template ]
Week 3
Thur, Oct 12
TA session MAML
  • Tutorial is 4:30 – 5:20pm at Gates B1. See Canvas for Zoom link.
Week 4
Mon, Oct 16
Lecture Few-shot learning via metric learning Due Homework 1
Week 4
Wed, Oct 18
Lecture Unsupervised pre-training for few-shot learning (contrastive)
Week 5
Mon, Oct 23
Lecture Unsupervised pre-training for few-shot learning (generative) Due Project proposal
Week 5
Wed, Oct 25
Lecture Advanced meta-learning topics (task construction) Due Homework 2
Homework 3 out [PDF, code , template]
Week 6
Mon, Oct 30
Lecture Variational inference
Week 6
Wed, Nov 1
Lecture Bayesian meta-learning
Week 6
Thur, Nov 2
TA session Bayesian Meta Learning
  • Tutorial is 4:30 – 5:20pm at Gates B1. See Canvas for Zoom link.
Week 7
Mon, Nov 6
Lecture Advanced meta-learning topics (large-scale meta-optimization) Due Homework 3
Homework 4 out [PDF, template/code] (optional)
Week 7
Wed, Nov 8
Guest Lecture James Harrison Learned Optimizers
Week 8
Mon, Nov 13
Lecture Lifelong learning
Week 8
Wed, Nov 15
Lecture No Lecture Due Project milestone
Due Homework 4 (optional)
Week 9
Mon, Nov 20
Thanksgiving break
Week 9
Wed, Nov 22
Thanksgiving break
Week 10
Mon, Nov 27
Guest lecture Jason Wei
Week 10
Wed, Nov 29
Lecture Domain Adaptation and Domain Generalization
Week 11
Mon, Dec 4
Lecture Frontiers & Open Challenges
Week 11
Wed, Dec 6
Presentations Project poster session
  • The poster session will be held at Packard Atrium, from 3:00 – 6:00 pm.
Week 12
Mon, Dec 11
No class Due Final project report

Course Calendar

A course calendar with details of lectures, TA sessions, office hours, and miscellaneous course events is available in a variety of formats:

Grading and Course Policies

Homeworks (50%): There are four graded homework assignments: Homework 0 is a warmup worth 5%, and Homeworks 1 – 3 are each worth 15%. Assignments will require training neural networks in PyTorch. Homework 4 is optional and will either replace one of Homeworks 1 – 3 or part of the project grade (whichever is better for your grade). All assignments are due on Gradescope at 11:59 pm Pacific Time on the respective due date.

Project (50%): This a research-level project of your choice. You may form groups of 1 – 3 students to complete the project, and you are encouraged to start early! Detailed guidelines on the project can be found here. You can use this template for your project proposal. Some past year projects are listed here for reference.

Extra Credit (up to 2%): Throughout the course, there are several participation opportunities that allow students to earn extra credit. The total extra credit that can be earned for the course is capped at 2%, and opportunities for this extra credit include answering questions on Ed and participating in live lecture polls. SCPD students will be able to submit the poll after the lecture. Here is the breakdown for extra credit opportunities:

  • Live lecture polls - 0.15% for each attempted poll in the class. SCPD students would be able to submit the poll by 11:59 PM the next day.
  • Guest lecture polls - 0.25% for each attempted poll in the class. SCPD students would be able to submit the poll by 11:59 PM the next day.
  • Ed participation - students with > 2 endorsed answers would be able to earn extra credit as per the following scheme:
    • The top answerer will earn 2% extra credit
    • The top 10th percentile of answerers will earn 1.5% extra credit
    • The top 15th percentile of answerers will earn 1% extra credit
    • The top 20th percentile of answerers will earn 0.5% extra credit

Late Days: You have 6 total late days across homework and project deliverables (anything worth a grade), except for the project poster. You may use a maximum of 2 late days for any single deliverable. Late days used for group projects apply to all members of the group.

Lecture Attendance: While we do not require lecture attendance, students are encouraged to join the live lecture. Students can also earn extra credit by attending lectures. To accommodate various circumstances, we will be live-streaming the in-person lecture via Panopto on Canvas. For those who cannot join the live lectures, lecture recordings will also be available on Canvas shortly following the lecture.

Honor Code: Students are free to form study groups and may discuss homework in groups. However, each student must write down the solutions and code from scratch independently and without referring to any written notes from group sessions. When debugging code together, students may only look at the input-output behavior of each other's programs and not their code itself. In other words, each student must understand the solution well enough in order to reconstruct it by themself. It is an honor code violation to copy, refer to, or look at written or code solutions from a previous year, including but not limited to: official solutions from a previous year, solutions posted online, and solutions you or someone else may have written up in a previous year. Furthermore, it is an honor code violation to post your assignment solutions online, such as on a public git repo, or use AI-driven code completion tools such as GitHub Copilot. You are, however, allowed to use such tools for the project. For more details about honor code, see The Stanford Honor Code and The Stanford Honor Code Pertaining to CS Courses.

Academic Accommodations

If you need an academic accommodation based on a disability, please register with the Office of Accessible Education (OAE). Professional staff will evaluate your needs, support appropriate and reasonable accommodations, and prepare an Academic Accommodation Letter for faculty. To get started, or to re-initiate services, please visit oae.stanford.edu. If you already have an Academic Accommodation Letter, please send your letter to cs330-staff-aut2324@cs.stanford.edu. OAE Letters should be sent to us at the earliest possible opportunity so that the course staff can partner with you and OAE to make the appropriate accommodations.

Note on Financial Aid

All students should retain receipts for books and other course-related expenses as these may be qualified educational expenses for tax purposes. If you are an undergraduate receiving financial aid, you may be eligible for additional financial aid for required books and course materials if these expenses exceed the aid amount in your award letter. For more information, review your award letter or visit the Student Budget website.

Notice on Lecture Recording

Video cameras located in the back of the room will capture the instructor presentations in this course. For your convenience, you can access these recordings by logging into the course Canvas site. These recordings might be reused in other Stanford courses, viewed by other Stanford students, faculty, or staff, or used for other education and research purposes. Note that while the cameras are positioned with the intention of recording only the instructor, occasionally a part of your image or voice might be incidentally captured. If you have questions, please contact a member of the teaching team.



    © Chelsea Finn 2023