【softmax分类器的加速器】

https://www.tensorflow.org/api_docs/python/tf/nn/sampled_softmax_loss

This is a faster way to train a softmax classifier over a huge number of classes.

【分类的结果集过大,选取子集】

https://www.tensorflow.org/api_guides/python/nn#Candidate_Sampling

Do you want to train a multiclass or multilabel model with thousands or millions of output classes (for example, a language model with a large vocabulary)? Training with a full Softmax is slow in this case, since all of the classes are evaluated for every training example. Candidate Sampling training algorithms can speed up your step times by only considering a small randomly-chosen subset of contrastive classes (called candidates) for each batch of training examples.

https://www.tensorflow.org/extras/candidate_sampling.pdf

【 compute F(x, y) for every class y ∈ L for every training example----耗时点,这是要解决的问题】

What is Candidate Sampling Say we have a multiclass or multi­label problem where each training example (x , ) consists of i Ti a context xi a small (multi)set of target classes Ti out of a large universe L of possible classes. For example, the problem might be to predicting the next word (or the set of future words) in a sentence given the previous words.

We wish to learn a compatibility function F(x, y) which says something about the compatibility of a class y with a context x . For example ­ the probability of the class given the context.

“Exhaustive” training methods such as softmax and logistic regression require us to compute F(x, y) for every class y ∈ L for every training example. When |L| is very large, this can be prohibitively expensive.

【the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary:子集】

https://arxiv.org/pdf/1412.2007.pdf

Neural machine translation, a recently proposed approach to machine translation based purely on neural networks, has shown promising results compared to the existing approaches such as phrase-based statistical machine translation. Despite its recent success, neural machine translation has its limitation in handling a larger vocabulary, as training complexity as well as decoding complexity increase proportionally to the number of target words. In this paper, we propose a method that allows us to use a very large target vocabulary without increasing training complexity, based on importance sampling. We show that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary. The models trained by the proposed approach are empirically found to outperform the baseline models with a small vocabulary as well as the LSTM-based neural machine translation models. Furthermore, when we use the ensemble of a few models with very large target vocabularies, we achieve the state-of-the-art translation performance (measured by BLEU) on the English->German translation and almost as high performance as state-of-the-art English->French translation system.

On Using Very Large Target Vocabulary for Neural Machine Translation Candidate Sampling Sampled Softmax的更多相关文章

  1. 课程五(Sequence Models),第三周(Sequence models & Attention mechanism) —— 1.Programming assignments:Neural Machine Translation with Attention

    Neural Machine Translation Welcome to your first programming assignment for this week! You will buil ...

  2. Sequence Models Week 3 Neural Machine Translation

    Neural Machine Translation Welcome to your first programming assignment for this week! You will buil ...

  3. 神经机器翻译 - NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE

    论文:NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE 综述 背景及问题 背景: 翻译: 翻译模型学习条件分布 ...

  4. 对Neural Machine Translation by Jointly Learning to Align and Translate论文的详解

    读论文 Neural Machine Translation by Jointly Learning to Align and Translate 这个论文是在NLP中第一个使用attention机制 ...

  5. Effective Approaches to Attention-based Neural Machine Translation(Global和Local attention)

    这篇论文主要是提出了Global attention 和 Local attention 这个论文有一个译文,不过我没细看 Effective Approaches to Attention-base ...

  6. 【转载 | 翻译】Visualizing A Neural Machine Translation Model(神经机器翻译模型NMT的可视化)

    转载并翻译Jay Alammar的一篇博文:Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models Wi ...

  7. [笔记] encoder-decoder NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE

    原文地址 :[1409.0473] Neural Machine Translation by Jointly Learning to Align and Translate (arxiv.org) ...

  8. Introduction to Neural Machine Translation - part 1

    The Noise Channel Model \(p(e)\): the language Model \(p(f|e)\): the translation model where, \(e\): ...

  9. 论文阅读 | Robust Neural Machine Translation with Doubly Adversarial Inputs

    (1)用对抗性的源实例攻击翻译模型; (2)使用对抗性目标输入来保护翻译模型,提高其对对抗性源输入的鲁棒性. 生成对抗输入:基于梯度 (平均损失)  ->  AdvGen 我们的工作处理由白盒N ...

随机推荐

  1. springBoot 整合 mybatis+Oracle

    现在比较流行的操作数据库操作层框架Mybatis,下面我们就来看看Springboot如何整合mybatis, 之前一直在用xml形式写sql,这次依然用xml的方式感觉这种还是比较灵活方便. 添加m ...

  2. Linux下Shell文件内容替换(sed)(转)

    sed -i 's/被替换的内容/要替换成的内容/g' file #-i为直接修改并保存 参考: http://blog.sina.com.cn/s/blog_7211cb9201019hgd.htm ...

  3. Python操作sqlite数据库小节

    学习了Python操作sqlite数据库,做一个小结,以备后用. import sqlite3import os# 进行数据库操作时,主要是参数如何传输try:# 链接数据库conn=sqlite3. ...

  4. 获取Android系统默认给每个app分配的内存上限

    ActivityManager activityManager = (ActivityManager) getSystemService(Context.ACTIVITY_SERVICE); int ...

  5. php正则表达式取子字符串及替换

    最近在学习如何用php编写cms,想把文章中的第一个图片提取出来当做缩略图显示到前面,想到的方法就是把文章内容作为一个大字符串,然后用正则表达式找出匹配出第一次出现<img src=" ...

  6. js ioc 实现

    var ar=[]; var o1={ id:'o1', o2:null } ar.push(o1); var o2={ id:'o2', o1:null } ar.push(o2) var ioc= ...

  7. 几种Tab的实现方法

    转载请注明出处,谢谢! 学了这久Android,今天来总结一下几种Tab的实现方法 实现方法一: ViewPage来实现 首先创建一个top.xml布局和一个bottom.xml布局,然后在主界面中通 ...

  8. hdu2955 Robberies (01背包)

    转载请注明出处:http://blog.csdn.net/u012860063 题目链接:pid=2955">http://acm.hdu.edu.cn/showproblem.php ...

  9. 【SharePoint】SharePoint 2013 使用PreSaveAction自定义客户端验证

    使用PreSaveAction函数实现客户端自定义验证. 例:[项目编号]为空时,必须填写[责任者]项.(其中[项目编号]为单行文本框,[责任者]为用户/组选择框.) function PreSave ...

  10. vue相关知识

    1.看https://www.bilibili.com/video/av27969216/?p=54,看他的就够了 https://juejin.im/post/5a5bc8c36fb9a01ca26 ...