CS 182/282A
| Deep Neural Networks
Spring 2023
Lectures: Mon/Fri 9:00–10:30 am, Soda 306
Description
Deep Networks have revolutionized computer vision, language technology, robotics and control.
They have a growing impact in many other areas of science and engineering, and increasingly,
on commerce and society. They do not however, follow any currently known compact set of
theoretical principles. In Yann Lecun's words they require "an interplay between intuitive
insights, theoretical modeling, practical implementations, empirical studies,
and scientific analyses." This is a fancy way of saying “we don’t understand
this stuff nearly well enough, but we have no choice but to muddle through anyway.”
This course attempts to cover that ground and show you how to muddle through even
as we aspire to do more.
Lecture reference materials
Lectures are webcast by the department and recordings will be posted to this
youtube playlist.
You must be logged into your @berkeley.edu account to access the videos.
Lectures will have a substantial amount (if not all) of the content covered on the whiteboard (not on
presentation slides)
Because Deep Learning is rapidly evolving field, the material covered in this course can change
substantially from semester to semester.
If you are interested in materials from previous iterations of this course, please see here:
[Sp21] [Sp22]
[Fa22]
Syllabus
W
|
Date
|
Lecture Topic
|
Resources
|
Section
|
Homework
|
0 |
Jan 20 |
Introduction |
|
|
HW0 - Basics
Written
Coding
Solution (Written)
Self-Grade
|
1 |
Jan 23 |
Basic Principles
|
Fa22 Notes
Note 2
Scribe Notes
|
Worksheet
Code
Solution
|
HW1 - Math Review & Fully-Connected Network
Written
Coding 1
Coding 2
Solution (Written)
Self-Grade
|
Jan 27 |
Basic Principles, Optimizers
|
Fa22 Notes
Note 3
Scribe Notes
|
2 |
Jan 30 |
Initialization and Normalization
|
Fa22 Notes
Scribe Notes
|
Worksheet
Solution
|
HW2 - Optimization and Initialization
Written
Coding (Q2)
Coding (Q5)
Coding (Q6)
Solution (Written)
Self-Grade
|
Feb 3 |
Survey of Problems and Architectures
|
Fa22 Notes
Note 5
Scribe Notes
|
3 |
Feb 6 |
Survey of Problems and Architectures
|
Fa22 Notes
Note 6
Scribe Notes
|
Worksheet
Code
Solution
|
HW3 - Batch Normalization, Dropout, and Convolutions
Written
Coding (Q4)
Coding (Q5-1)
Coding (Q5-2)
Coding (Q5-3)
Solution (Written)
Self-Grade
|
Feb 10 |
ConvNets and Computer Vision
|
Fa22 Notes
Scribe Notes
|
4 |
Feb 13 |
ConvNets and Computer Vision
|
Fa22 Notes
Scribe Notes
|
Worksheet
Code
Solution
|
HW4 - Dropout and Convolutions (II)
Written
Coding (Q1)
Coding (Q4)
Coding (Q5)
Solution (Written)
Self-Grade
|
Feb 17 |
Computer Vision in Practice
|
Fa22 Notes
Scribe Notes
|
5 |
Feb 20 |
Holiday
|
Fa22 Notes
|
Worksheet
Solution
|
HW5 - Graph Neural Networks
Written
Coding (Q3)
Coding (Q5)
Solution (Written)
Self-Grade
|
Feb 24 |
Dropout, Graph Neural Networks
|
Fa22 Notes
Scribe Notes
|
6 |
Feb 27 |
Graph Neural Networks
|
Fa22 Notes
Scribe Notes
|
Worksheet
Solution
|
HW6 - Recurrent Neural Networks
Written
Coding (Q3)
Coding (Q4)
Solution (Written)
Self-Grade
|
Mar 3 |
Recurrent Neural Networks
|
Fa22 Notes
Scribe Notes
|
7 |
Mar 6 |
RNNs and LSTMs
|
Fa22 Notes
Scribe Notes
|
Worksheet
Solution
|
HW7 - Autoencoders and Attention
Written
Coding
Solution (Written)
Self-Grade
|
Mar 10 |
Seq-to-Seq, Attention
|
Fa22 Notes
Scribe Notes
|
8 |
Mar 13 |
Self-Supervision, Autoencoders
|
Fa22 Notes
Scribe Notes
|
Worksheet
Solution
|
Study for Midterm
|
Mar 17 |
Transformers
|
Fa22 Notes
Scribe Notes
|
9 |
Mar 20 |
No Lecture: Study for Midterm
|
Fa22 Notes
Scribe Notes
|
Worksheet (Basics)
Worksheet (CNN&GNN)
Worksheet (AE&RNN)
Solution (Basics)
Solution (CNN&GNN)
Solution (AE&RNN)
|
Midterm
Midterm Solution
HW8 - Redo Midterm and Attention
Written
Coding
Solution (Written)
Self-Grade
|
Mar 24 |
Transformers
|
Fa22 Notes
Scribe Notes
|
|
Mar 27 |
Spring Break
|
|
|
|
Mar 31 |
Spring Break
|
|
10 |
Apr 3 |
Meta-learning, fine-tuning, transfer
|
Fa22 Notes
Scribe Notes
|
Worksheet
Solution
|
HW9 - Transformers and Masked Autoencoders
Written
Coding (Q2)
Coding (Q3)
Coding (Q4)
Solution (Written)
Self-Grade
|
Apr 7 |
Meta-learning, fine-tuning, transfer
|
Fa22 Notes
Scribe Notes
|
11 |
Apr 10 |
Meta-learning, fine-tuning, transfer
|
Fa22 Notes
Scribe Notes
|
Worksheet
Solution
|
HW10 - Meta-Learning, Prompting, and Compression
Written
Coding (Q1)
Coding (Q4)
Coding (Q6a)
Coding (Q6b)
Solution (Written)
Self-Grade
|
Apr 14 |
Generative Models
|
Fa22 Notes
Scribe Notes
|
12 |
Apr 17 |
Generative Models
|
Fa22 Notes
Scribe Notes
|
Worksheet
Paper
Solution
|
HW11 - Generative Models
Written
Coding (Q1)
Coding (Q3)
Coding (Q4,5)
Coding (Q6)
Solution (Written)
Self-Grade
|
Apr 21 |
Generative Models
|
Fa22 Notes
|
13 |
Apr 24 |
Advanced Topics
|
Fa22 Notes
|
Worksheet
Solution
|
HW12 - Diffusion, Early Exit, and RLHF
Written
Coding (Q3)
Coding (Q4)
Solution (Written)
Self-Grade
|
Apr 28 |
Conclusion and Review
|
Fa22 Notes
|
14 |
May 1 |
RRR Week
|
|
Review Sessions
Pre-MT Topics
Pre-MT Sol
Transformers
Transformers Sol
Fine-tuning
Fine-tuning Sol
Generative Models
Generative Models Sol
|
|
May 5 |
RRR Week
|
|
15 |
May 8 |
Final Exam: Mon, May 8, 7pm - 10pm
|
|
|
Final
Final Solution
|
|
|
|