CS 182/282A | Deep Neural Networks

Spring 2023

Lectures: Mon/Fri 9:00–10:30 am, Soda 306

Neural Networks

Description

Deep Networks have revolutionized computer vision, language technology, robotics and control. They have a growing impact in many other areas of science and engineering, and increasingly, on commerce and society. They do not however, follow any currently known compact set of theoretical principles. In Yann Lecun's words they require "an interplay between intuitive insights, theoretical modeling, practical implementations, empirical studies, and scientific analyses." This is a fancy way of saying “we don’t understand this stuff nearly well enough, but we have no choice but to muddle through anyway.” This course attempts to cover that ground and show you how to muddle through even as we aspire to do more.

Lecture reference materials

Lectures are webcast by the department and recordings will be posted to this youtube playlist. You must be logged into your @berkeley.edu account to access the videos. Lectures will have a substantial amount (if not all) of the content covered on the whiteboard (not on presentation slides)

Because Deep Learning is rapidly evolving field, the material covered in this course can change substantially from semester to semester. If you are interested in materials from previous iterations of this course, please see here: [Sp21] [Sp22] [Fa22]

Syllabus

W Date Lecture Topic Resources Section Homework
0 Jan 20 Introduction HW0 - Basics
Written Coding
Solution (Written) Self-Grade
1 Jan 23 Basic Principles Fa22 Notes Note 2 Scribe Notes Worksheet Code Solution HW1 - Math Review & Fully-Connected Network
Written Coding 1 Coding 2 Solution (Written) Self-Grade
Jan 27 Basic Principles, Optimizers Fa22 Notes Note 3 Scribe Notes
2 Jan 30 Initialization and Normalization Fa22 Notes Scribe Notes Worksheet Solution HW2 - Optimization and Initialization
Written Coding (Q2) Coding (Q5) Coding (Q6) Solution (Written) Self-Grade
Feb 3 Survey of Problems and Architectures Fa22 Notes Note 5 Scribe Notes
3 Feb 6 Survey of Problems and Architectures Fa22 Notes Note 6 Scribe Notes Worksheet Code Solution HW3 - Batch Normalization, Dropout, and Convolutions
Written Coding (Q4) Coding (Q5-1) Coding (Q5-2) Coding (Q5-3) Solution (Written) Self-Grade
Feb 10 ConvNets and Computer Vision Fa22 Notes Scribe Notes
4 Feb 13 ConvNets and Computer Vision Fa22 Notes Scribe Notes Worksheet Code Solution HW4 - Dropout and Convolutions (II)
Written Coding (Q1) Coding (Q4) Coding (Q5) Solution (Written) Self-Grade
Feb 17 Computer Vision in Practice Fa22 Notes Scribe Notes
5 Feb 20 Holiday Fa22 Notes Worksheet Solution HW5 - Graph Neural Networks
Written Coding (Q3) Coding (Q5) Solution (Written) Self-Grade
Feb 24 Dropout, Graph Neural Networks Fa22 Notes Scribe Notes
6 Feb 27 Graph Neural Networks Fa22 Notes Scribe Notes Worksheet Solution HW6 - Recurrent Neural Networks
Written Coding (Q3) Coding (Q4) Solution (Written) Self-Grade
Mar 3 Recurrent Neural Networks Fa22 Notes Scribe Notes
7 Mar 6 RNNs and LSTMs Fa22 Notes Scribe Notes Worksheet Solution HW7 - Autoencoders and Attention
Written Coding Solution (Written) Self-Grade
Mar 10 Seq-to-Seq, Attention Fa22 Notes Scribe Notes
8 Mar 13 Self-Supervision, Autoencoders Fa22 Notes Scribe Notes Worksheet Solution Study for Midterm
Mar 17 Transformers Fa22 Notes Scribe Notes
9 Mar 20 No Lecture: Study for Midterm Fa22 Notes Scribe Notes Worksheet (Basics) Worksheet (CNN&GNN) Worksheet (AE&RNN) Solution (Basics) Solution (CNN&GNN) Solution (AE&RNN) Midterm Midterm Solution
HW8 - Redo Midterm and Attention
Written Coding Solution (Written) Self-Grade
Mar 24 Transformers Fa22 Notes Scribe Notes
Mar 27 Spring Break
Mar 31 Spring Break
10 Apr 3 Meta-learning, fine-tuning, transfer Fa22 Notes Scribe Notes Worksheet Solution HW9 - Transformers and Masked Autoencoders
Written Coding (Q2) Coding (Q3) Coding (Q4) Solution (Written) Self-Grade
Apr 7 Meta-learning, fine-tuning, transfer Fa22 Notes Scribe Notes
11 Apr 10 Meta-learning, fine-tuning, transfer Fa22 Notes Scribe Notes Worksheet Solution HW10 - Meta-Learning, Prompting, and Compression
Written Coding (Q1) Coding (Q4) Coding (Q6a) Coding (Q6b) Solution (Written) Self-Grade
Apr 14 Generative Models Fa22 Notes Scribe Notes
12 Apr 17 Generative Models Fa22 Notes Scribe Notes Worksheet Paper Solution HW11 - Generative Models
Written Coding (Q1) Coding (Q3) Coding (Q4,5) Coding (Q6) Solution (Written) Self-Grade
Apr 21 Generative Models Fa22 Notes
13 Apr 24 Advanced Topics Fa22 Notes Worksheet Solution HW12 - Diffusion, Early Exit, and RLHF
Written Coding (Q3) Coding (Q4) Solution (Written) Self-Grade
Apr 28 Conclusion and Review Fa22 Notes
14 May 1 RRR Week Review Sessions Pre-MT Topics Pre-MT Sol Transformers Transformers Sol Fine-tuning Fine-tuning Sol Generative Models Generative Models Sol
May 5 RRR Week
15 May 8 Final Exam: Mon, May 8, 7pm - 10pm Final Final Solution