SysML——AI-Sys Spring 2019

AI-Sys

• Syllabus
• Projects
• Grading

AI-Sys Spring 2019

• When: Mondays and Wednesdays from 9:30 to 11:00
• Where: Soda 405
• Instructors: Ion Stoica and Joseph E. Gonzalez
• Announcements: Piazza
• Sign-up to Present: Google Spreadsheet
• Project Ideas: Google Spreadsheet
• If you have reading suggestions please send a pull request to this course website on Github by modifying the index.md file.

Course Description

The recent success of AI has been in large part due in part to advances in hardware and software systems. These systems have enabled training increasingly complex models on ever larger datasets. In the process, these systems have also simplified model development, enabling the rapid growth in the machine learning community. These new hardware and software systems include a new generation of GPUs and hardware accelerators (e.g., TPU and Nervana), open source frameworks such as Theano, TensorFlow, PyTorch, MXNet, Apache Spark, Clipper, Horovod, and Ray, and a myriad of systems deployed internally at companies just to name a few. At the same time, we are witnessing a flurry of ML/RL applications to improve hardware and system designs, job scheduling, program synthesis, and circuit layouts.

In this course, we will describe the latest trends in systems designs to better support the next generation of AI applications, and applications of AI to optimize the architecture and the performance of systems. The format of this course will be a mix of lectures, seminar-style discussions, and student presentations. Students will be responsible for paper readings, and completing a hands-on project. Readings will be selected from recent conference proceedings and journals. For projects, we will strongly encourage teams that contains both AI and systems students.

Course Syllabus

This is a tentative schedule. Specific readings are subject to change as new material is published.

Jump to Today

Week	Date (Lec.)	Topic
1	1/23/19  ( 1 )	Introduction and Course Overview This lecture will be an overview of the class, requirements, and an introduction to what makes great AI-Systems research. Slide Links Course Overview [pdf, pptx]
2	1/28/19  ( 2 )	Convolutional Neural Network Architectures Minor Update: We have moved the reading on auto-encoders to Wednesday. Reading notes for the two required readings below must be submitted using this google form by Monday the 28th at 9:30AM. We have asked that for each reading you answer the following questions: What is the problem that is being solved? What are the metrics of success? What are the key innovations over prior work? What are the key results? What are some of the limitations and how might this work be improved? How might this work have long term impact? If you find some of the reading confusing and want a more gentle introduction, the optional reading contains some useful explanatory blog posts that may help. Links Reading Quiz due before class. Intro Lecture + AlexNet [pdf, pptx] Classic Neural Architectures and Inception-v4 [pdf, pptx] The AlexNet paper that both help launch deep learning and also advocate for systems and ML. Take a look at how system constraints affected the model. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. In retrospect, the paper Rethinking the Inception Architecture for Computer Vision provides a better overview of the ideas and motivations behind the latest inception models. Additional Optional Reading
	1/30/19  ( 3 )	More Neural Network Architectures Links Reading Quiz due before class. Intro [pdf, pptx] Autoencoders [pdf, pptx] Graph Neural Networks [pdf, pptx]   We had originally assigned, Autoencoders, Unsupervised Learning, and Deep Architectures. However this paper is a bit theoretical for the goals of this class. Instead, you may alternatively read this overview paper and use it when filling in the reading form. Graph Neural Networks: A Review of Methods and Applications Additional Optional Reading
3	2/4/19  ( 4 )	Deep Learning Frameworks Links Reading Quiz due before class. Intro Lecture [pdf, pptx] TensorFlow Presentation [pdf, pptx]   TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems and/or TensorFlow OSDI Paper MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems Additional Optional Reading
	2/6/19  ( 5 )	RL Systems & Algorithms Links Reading Quiz due before class. RLlib [pdf] A3C [pdf]   Asynchronous Methods for Deep Reinforcement Learning RLlib: Abstractions for Distributed Reinforcement Learning Additional Optional Reading
4	2/11/19  ( 6 )	Application: Data Structure and Algorithms Links Reading Quiz due before class. Learned Indexes [pdf, pptx] Learning to Optimize Join Queries [pdf]   The Case for Learned Index Structures Learning to Optimize Join Queries With Deep Reinforcement Learning Additional Optional Reading
	2/13/19  ( 7 )	Distributed Systems for ML Links Reading Quiz due before class. Learned Cardinalities [pdf]   The Case for Learned Index Structures cont’d Learned Cardinalities: Estimating Correlated Joins with Deep Learning
5	2/18/19  ( 8 )	Administrative Holiday (Feb 18th)
	2/20/19  ( 9 )	Hyperparameter search Links Reading Quiz due before class. There was a mix-up in updating the reading and the wrong paper was swapped. You may either read the Hyperband paper (preferred) or the Vizer paper (see optional reading) for the second reading. A Generalized Framework for Population Based Training [pdf]   A Generalized Framework for Population Based Training Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization Additional Optional Reading
6	2/25/19  ( 10 )	Auto ML & Neural Architecture Search (1/2) Links Reading Quiz due before class. AutoML Overview [pdf, pptx] Designing Neural Networks with RL [pdf, pptx]   Efficient and Robust Automated Machine Learning Designing Neural Network Architectures using Reinforcement Learning `
	2/27/19  ( 11 )	Auto ML & Neural Architecture Search (2/2) Links Reading Quiz due before class. Semantic Segmentation AutoML slides [pdf]   Efficient Neural Architecture Search via Parameter Sharing Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
7	3/4/19  ( 12 )	Autonomous Vehicles Links Reading Quiz due before class. Autonomous Vehicles Overview [pdf, pptx] Presentation: The Architectural Implications of Autonomous Driving[pdf]   The Architectural Implications of Autonomous Driving ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst Additional Optional Reading
	3/6/19  ( 13 )	Deep Learning Compilers Links Reading Quiz due before class. DL Compiler Overview [pdf, pptx] Presentation PDF   TVM: An Automated End-to-End Optimizing Compiler for Deep Learning TensorComprehensions Additional Optional Reading
8	3/11/19  ( 14 )	Project Presentation Checkpoints
	3/13/19  ( 15 )	Application: Program synthesis Links Reading Quiz due before class.   Learning to Represent Programs with Graphs [pdf], [key] DeepCoder: Learning to write programs [pdf]
9	3/18/19  ( 16 )	Distributed Deep Learning (Part 1) Links Reading Quiz due before class. Overview [pdf, pptx]   Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour Large Scale Distributed Deep Networks Additional Optional Reading
	3/20/19  ( 17 )	Distributed Deep Learning (Part 2) Links Reading Quiz due before class.   Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent[pdf] Scaling Distributed Machine Learning with the Parameter Server
10	3/25/19  ( 18 )	Spring Break (March 25th)
	3/27/19  ( 19 )	Spring Break (March 27th)
11	4/1/19  ( 20 )	Application: Networking Links Reading Quiz due before class. Introduction [pdf, pptx] AI Applications in Network Congestion Control [pdf, pptx]   Neural Adaptive Video Streaming with Pensieve Internet Congestion Control via Deep Reinforcement Learning Additional Optional Reading
	4/3/19  ( 21 )	Dynamic Neural Networks Links Reading Quiz due before class. Introduction [pdf, pptx]   Outrageously large neural networks: The sparsely-gated mixture-of-experts layer SkipNet: Learning Dynamic Routing in Convolutional Networks
12	4/8/19  ( 22 )	Model Compression Links Reading Quiz due before class. Introduction [pdf, pptx]   Rethinking the Value of Network Pruning ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices Additional Optional Reading
	4/10/19  ( 23 )	Applications: Security Links Reading Quiz due before class. Helen [pdf, pptx]   Helen Federated Learning: Strategies for Improving Communication Efficiency Additional Optional Reading
13	4/15/19  ( 24 )	Application: Prediction Serving Links Reading Quiz due before class.   Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications Clipper: A Low-Latency Online Prediction Serving System Additional Optional Reading
	4/17/19  ( 25 )	Natural Language Processing Systems Links Reading Quiz due before class. Introduction [pdf, pptx]   Deep Speech 2: End-to-End Speech Recognition in English and Mandarin Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
14	4/22/19  ( 26 )	Explanability & Interpretability Links Reading Quiz due before class. Introduction [pdf, pptx]   “Why Should I Trust You?”: Explaining the Predictions of Any Classifier The Mythos of Model Interpretability Additional Optional Reading
	4/24/19  ( 27 )	Scheduling for DL Workloads Links Reading Quiz due before class. DL Scheduling slides [pdf] Dominant Resource Fairness (DRF) slides [pdf]   Optimus: an efficient dynamic resource scheduler for deep learning clusters [pdf] Gandiva: Introspective Cluster Scheduling for Deep Learning Additional Optional Reading
15	4/29/19  ( 28 )	Cortical Learning and Stoica Course Summary Links Reading Quiz due before class.   Cortical Learning via Prediction Additional Optional Reading
	5/1/19  ( 29 )	Neural Modular Networks and Gonzalez Course Summary Links Reading Quiz due before class. Neural Modular Networks Slides [pdf, pptx] Gonzalez Course Summary (Reflections on the Field of AI-Systems) [pdf, pptx]   Neural Modular Networks
16	5/6/19  ( 30 )	RRR Week (May 6th)
	5/8/19  ( 31 )	Poster Session from 9:00 to 11:00
17	5/13/19  ( 32 )	Final Reports Due Due at 11:59 PM Format: 8 pages (Google Doc) Email link to jegonzal@berkeley.edu and istoica@berkeley.edu

Projects

Detailed candidate project descriptions will be posted shortly. However, students are encourage to find projects that relate to their ongoing research.

Grading

Grades will be largely based on class participation and projects. In addition, we will require weekly paper summaries submitted before class.

• Projects: 60%
• Weekly Summaries: 20%
• Class Participation: 20%

Back to top

© 2017-2018 UC Berkeley · Privacy · Terms