Preamble

This repository contains the lecture slides and course description for the Deep Natural Language Processing course offered in Hilary Term 2017 at the University of Oxford.

This is an advanced course on natural language processing. Automatically processing natural language inputs and producing language outputs is a key component of Artificial General Intelligence. The ambiguities and noise inherent in human communication render traditional symbolic AI techniques ineffective for representing and analysing language data. Recently statistical techniques based on neural networks have achieved a number of remarkable successes in natural language processing leading to a great deal of commercial and academic interest in the field

This is an applied course focussing on recent advances in analysing and generating speech and text using recurrent neural networks. We introduce the mathematical definitions of the relevant machine learning models and derive their associated optimisation algorithms. The course covers a range of applications of neural networks in NLP including analysing latent dimensions in text, transcribing speech to text, translating between languages, and answering questions. These topics are organised into three high level themes forming a progression from understanding the use of neural networks for sequential language modelling, to understanding their use as conditional language models for transduction tasks, and finally to approaches employing these techniques in combination with other mechanisms for advanced applications. Throughout the course the practical implementation of such models on CPU and GPU hardware is also discussed.

This course is organised by Phil Blunsom and delivered in partnership with the DeepMind Natural Language Research Group.

Lecturers

  • Phil Blunsom (Oxford University and DeepMind)
  • Chris Dyer (Carnegie Mellon University and DeepMind)
  • Edward Grefenstette (DeepMind)
  • Karl Moritz Hermann (DeepMind)
  • Andrew Senior (DeepMind)
  • Wang Ling (DeepMind)
  • Jeremy Appleyard (NVIDIA)

TAs

  • Yannis Assael
  • Yishu Miao
  • Brendan Shillingford
  • Jan Buys

Timetable

Practicals

  • Group 1 - Monday, 9:00-11:00 (Weeks 2-8), 60.05 Thom Building
  • Group 2 - Friday, 16:00-18:00 (Weeks 2-8), Room 379
  1. Practical 1: word2vec
  2. Practical 2: text classification
  3. Practical 3: recurrent neural networks for text classification and language modelling
  4. Practical 4: open practical

Lectures

Public Lectures are held in Lecture Theatre 1 of the Maths Institute, on Tuesdays and Thursdays, 16:00-18:00 (Hilary Term Weeks 1,3-8).

Lecture Materials

1. Lecture 1a - Introduction [Phil Blunsom]

This lecture introduces the course and motivates why it is interesting to study language processing using Deep Learning techniques.

[slides]
[video]

2. Lecture 1b - Deep Neural Networks Are Our Friends [Wang Ling]

This lecture revises basic machine learning concepts that students should know before embarking on this course.

[slides]
[video]

3. Lecture 2a- Word Level Semantics [Ed Grefenstette]

Words are the core meaning bearing units in language. Representing and learning the meanings of words is a fundamental task in NLP and in this lecture the concept of a word embedding is introduced as a practical and scalable solution.

[slides]
[video]

Reading

Embeddings Basics

Datasets and Visualisation

Blog posts

Further Reading

4. Lecture 2b - Overview of the Practicals [Chris Dyer]

This lecture motivates the practical segment of the course.

[slides]
[video]

5. Lecture 3 - Language Modelling and RNNs Part 1 [Phil Blunsom]

Language modelling is important task of great practical use in many NLP applications. This lecture introduces language modelling, including traditional n-gram based approaches and more contemporary neural approaches. In particular the popular Recurrent Neural Network (RNN) language model is introduced and its basic training and evaluation algorithms described.

[slides]
[video]

Reading

Textbook

Blogs

6. Lecture 4 - Language Modelling and RNNs Part 2 [Phil Blunsom]

This lecture continues on from the previous one and considers some of the issues involved in producing an effective implementation of an RNN language model. The vanishing and exploding gradient problem is described and architectural solutions, such as Long Short Term Memory (LSTM), are introduced.

[slides]
[video]

Reading

Textbook

Vanishing gradients, LSTMs etc.

Dealing with large vocabularies

Regularisation and dropout

Other stuff

7. Lecture 5 - Text Classification [Karl Moritz Hermann]

This lecture discusses text classification, beginning with basic classifiers, such as Naive Bayes, and progressing through to RNNs and Convolution Networks.

[slides]
[video]

Reading

8. Lecture 6 - Deep NLP on Nvidia GPUs [Jeremy Appleyard]

This lecture introduces Graphical Processing Units (GPUs) as an alternative to CPUs for executing Deep Learning algorithms. The strengths and weaknesses of GPUs are discussed as well as the importance of understanding how memory bandwidth and computation impact throughput for RNNs.

[slides]
[video]

Reading

9. Lecture 7 - Conditional Language Models [Chris Dyer]

In this lecture we extend the concept of language modelling to incorporate prior information. By conditioning an RNN language model on an input representation we can generate contextually relevant language. This very general idea can be applied to transduce sequences into new sequences for tasks such as translation and summarisation, or images into captions describing their content.

[slides]
[video]

Reading

10. Lecture 8 - Generating Language with Attention [Chris Dyer]

This lecture introduces one of the most important and influencial mechanisms employed in Deep Neural Networks: Attention. Attention augments recurrent networks with the ability to condition on specific parts of the input and is key to achieving high performance in tasks such as Machine Translation and Image Captioning.

[slides]
[video]

Reading

11. Lecture 9 - Speech Recognition (ASR) [Andrew Senior]

Automatic Speech Recognition (ASR) is the task of transducing raw audio signals of spoken language into text transcriptions. This talk covers the history of ASR models, from Gaussian Mixtures to attention augmented RNNs, the basic linguistics of speech, and the various input and output representations frequently employed.

[slides]
[video]

12. Lecture 10 - Text to Speech (TTS) [Andrew Senior]

This lecture introduces algorithms for converting written language into spoken language (Text to Speech). TTS is the inverse process to ASR, but there are some important differences in the models applied. Here we review traditional TTS models, and then cover more recent neural approaches such as DeepMind's WaveNet model.

[slides]
[video]

13. Lecture 11 - (Coming Soon) Question Answering [Karl Moritz Hermann]

[slides]
[video]

14. Lecture 12 - (Coming Soon) Memory [Ed Grefenstette]

[slides]
[video]

Piazza

We will be using Piazza to facilitate class discussion during the course. Rather than emailing questions directly, I encourage you to post your questions on Piazza to be answered by your fellow students, instructors, and lecturers. However do please do note that all the lecturers for this course are volunteering their time and may not always be available to give a response.

Find our class page at: https://piazza.com/ox.ac.uk/winter2017/dnlpht2017/home

Assessment

The primary assessment for this course will be a take-home assignment issued at the end of the term. This assignment will ask questions drawing on the concepts and models discussed in the course, as well as from selected research publications. The nature of the questions will include analysing mathematical descriptions of models and proposing extensions, improvements, or evaluations to such models. The assignment may also ask students to read specific research publications and discuss their proposed algorithms in the context of the course. In answering questions students will be expected to both present coherent written arguments and use appropriate mathematical formulae, and possibly pseudo-code, to illustrate answers.

The practical component of the course will be assessed in the usual way.

Acknowledgements

This course would not have been possible without the support of DeepMind, The University of Oxford Department of Computer Science, Nvidia, and the generous donation of GPU resources from Microsoft Azure.

【MT】牛津的MT教程的更多相关文章

  1. 快播王欣发布匿名IM社交软件“马桶MT”

    2019年1月14日,快播王欣推出了一款匿名IM社交软件——马桶MT,它的灵感像是来自于美国的匿名分享应用Secret(已关闭). 原快播创始人王欣近日在微博预告了其新公司云歌人工智能推出一款全新社交 ...

  2. 一周学会Mootools 1.4中文教程:序论

    刚才发了几篇Mootools(以后直接简称Moo或Mt,看到这两个名字的时候不要感到奇怪),有一位热心的朋友"追杀"告诉我说现在已经出到1.4了,就不要再纠结于1.2了,想象一下有 ...

  3. QT5静态编译教程,主要针对vs2012(渡世白玉)

    QT5,VS2012静态编译,所有的库准备充分的话qwebkit也可以静态编译通过,但是我编译的版本使用中如果用了QWEBVIEW控件在连接时会出错. 注:我自己编译的环境是:win server 2 ...

  4. windbg调试系列教程:sos扩展的介绍和使用

    SOS是什么? 直观来说,sos就是一个程序集文件.这个程序集的作用就是让我们在使用windbg分析.net进程时,更加方便快捷.通过sos,我们可以清晰的查看CLR运行时的各类信息,辅助我们去理解托 ...

  5. 2、如何解决xamarin没有相关教程的的指导贴

    本篇文章主要在于解决xamarin相关文档偏少的问题. 最终的代码并不重要.重要的还是那种处理的方式 授人以渔 群里有群友讨论说需要读取安卓的 充电电流.这样的问题实际上在原生java有一堆.但是到了 ...

  6. QT5.8 VS2017 编译教程(可以使用VS2017 XP兼容包)

    1.下载QT5.8源码 这个我不做过多解释. 2.安装使用的环境 visual studio 2017  Python Perl  Ruby 安装好,并配置好环境PATH变量. 3.修改错误代码 错误 ...

  7. 超详细!Vuex手把手教程

    目录 1,前言 2,Vuex 是什么 3,5大属性说明 4,state 4.1 直接访问 4.1 使用mapState映射 5,getters 5.1 先在vuex中定义getters 5.2 直接获 ...

  8. Sass学习笔记之入门篇

    Sass又名SCSS,是CSS预处理器之一,,它能用来清晰地.结构化地描述文件样式,有着比普通 CSS 更加强大的功能. Sass 能够提供更简洁.更优雅的语法,同时提供多种功能来创建可维护和管理的样 ...

  9. 15 条实用 Linux/Unix 磁带管理命令

    导读 磁带设备应只用于定期的文件归档或将数据从一台服务器传送至另一台.通常磁带设备与 Unix 机器连接,用 mt 或 mtx 控制.强烈建议您将所有的数据同时备份到磁盘(也许是云中)和磁带设备中. ...

随机推荐

  1. 字符型液晶屏模拟控件(En)

    A replica CLCD module control. Initiated on May 5, 2012 Updated on Feb 21, 2017 Copyright 2012-2017 ...

  2. mybatis 参数格式异常-- Error querying database. Cause: java.lang.NumberFormatException: For input string

    mybatis中 <if></if>标签中进行判断时,如果传入的时字符格式和数字进行判断需要将数字进行转译,否则默认是数字和数字进行比较,这是就会出现参数格式异常如<if ...

  3. Java学习点滴——Class和反射

    基于<Java编程思想>第四版 前言 我们要操作一个类实例对象时,一般都要先知道这个类有哪些方法或者成员变量.反射就是在我们不知道这个类有哪些方法或成员变量时,使用特定方式得到类的这些信息 ...

  4. 授权管理-LDAP-介绍与环境搭建

    LDAP介绍 转自:https://blog.csdn.net/tanshizhen119/article/details/79942315 还是先来百度百科介绍. LDAP是轻量目录访问协议,英文全 ...

  5. unity transform 常用操作

    1.寻找物体 1.1 寻找满足条件的子物体 ` public static Transform FindObj(Transform transform, Func<Transform, bool ...

  6. http协议、模块、express框架以及路由器、中间件和mysql模块

    一.http协议 是浏览器和web服务器之间的通信协议 1.通用头信息 request url:请求的url,向服务器请求的数据 request method:请求的方式   get.post sta ...

  7. JS添加或删除HTML dom元素的方法实例分析

    本文实例讲述了JS代码添加或删除HTML dom元素的方法.分享给大家供大家参考,具体如下: 创建新的 HTML 元素 如需向 HTML DOM 添加新元素,您必须首先创建该元素(元素节点),然后向一 ...

  8. androidkiller连接模拟器并修改源码调试

    首先需要连接模拟器,首先在模拟器的bin目录下运行命令:nox_adb.exe connect 127.0.0.1:62001(可以disconnect关闭): 之后在androidkiller的bi ...

  9. i春秋misc部分writeup

    i春秋misc部分writeup 一.敲击 方方格格,然后看到下面的格式,猜测出是键盘上的布局,然后看这些字母形成的形状想那些字母,就是flag了 2.滴滴滴 放到ctfcack里解密,发现时栅栏密码 ...

  10. C# 中如何判断字符串的相似度

    基于 F23.StringSimilarity.dll  组件.Github 上可以搜索到该组件. 核心方法: var l = new Levenshtein(); double tempValue ...