神经⽹络注意⼒机制--AttentioninNeuralNetworks
这篇博⽂主要介绍神经⽹络中的注意⼒机制，代码实现了两个 soft visual attention
What is Attention? ⾸先来看看 注意⼒机制是什么？
Informally, a neural attention mechanism equips a neural network with the ability to focus on a subt of its inputs (or features): it lects specific inputs.
注意⼒机制可以让⼀个神经⽹络能够只关注其输⼊的⼀部分信息，它能够选择特定的输⼊。
attention is implemented as
f 是⼀个 attention network，其⽣成⼀个 attention vector a, 再讲 a 与输⼊ x 的 特征向量 z 相乘，这个 a 取值范围是 【0,1】，当我们说 soft attention 时，其取值是 0 到 1，当我们说 hard attention 其取值就只有 0 或 1。
菱角肉
为什么 attention 是重要的了？我们还有从 neural network 的本质说起， neural network 本质上就是⼀个 函数拟合器 function approximator，它的结构决定其可以拟合什么类型的函数，通常情况下输⼊向量彼此的作⽤⽅式只能是相加 A typical neural net is implemented as a chain of matrix multiplications and element-wi non-linearities, where elements of the input or feature vectors interact with each other only by addition
自立自强作文但是 注意⼒机制可以让输⼊向量之间的作⽤⽅式是相乘
Attention mechanisms compute a mask which is ud to multiply features
neural networks are universal function approximators and can approximate an arbitrary function to arbitrary precision, but only in the limit of an infinite number of hidden units. In any practical tting, that is not the ca: we are limited by the number of hidden units we can u.
共事神经⽹络可以拟合任意函数，但是受限于神经单元的数量，我们拟合的函数是有限制的。
The above definition of attention as multiplicative interactions allow us to consider a broader class of models if we relax the constrains on the values of the attention mask
注意⼒引⼊的 multiplicative interactions 可以让我们拟合更复杂的函数模型
解药歌词
Visual Attention
Attention 可以被应⽤到任意类型的输⼊，不管其形状如何，对于矩阵形式的输⼊ 如 图像，我们可以探讨 visual attention，
Hard Attention
对于图像的 Hard Attention 就是 image cropping ： g = I[y:y+h, x:x+w]， 这个 hard attention 的问题是 non-differentiable，可以通过 score-function estimator 来解决
水库调度
Soft Attention绿豆海带
⽂献 Show, Attend and Tell 中使⽤这种类型的 attention，
显陵
The model learns to attend to specific parts of the image while generating the word describing that part
Closing Thoughts
Attention mechanisms expand capabilities of neural networks: they allow approximating more complicated functions, or in more intuitive terms, they enable focusing on specific parts of the input.
Attention mechanisms 应该可以发挥更⼤的作⽤！11
>打臭嗝
