LSTM入门

结论

Conclusion

LSTM 是 RNN

RNN 中的每一步都从信息更加富集的地方提取信息。例如，你想使用 RNN

Grid LSTM 看起来极具潜力。Gregor等人、Chung 等人，或者 Bayer 与
Osendorfer 在生成模型中使用 RNN

Earlier, I mentioned the remarkable results people are achieving with
RNNs. Essentially all of these are achieved using LSTMs. They really
work a lot better for most tasks!

Written down as a set of equations, LSTMs look pretty intimidating.
Hopefully, walking through them step by step in this essay has made
them a bit more approachable.

LSTMs were a big step in what we can accomplish with RNNs. It’s
natural to wonder: is there another big step? A common opinion among
researchers is: “Yes! There is a next step and it’s attention!” The
idea is to let every step of an RNN pick information to look at from
some larger collection of information. For example, if you are using
an RNN to create a caption describing an image, it might pick a part
of the image to look at for every word it outputs. In fact, Xu, et
al.
(2015)
do exactly this –
it might be a fun starting point if you want to explore attention!
There’s been a number of really exciting results using attention, and
it seems like a lot more are around the corner…

Attention isn’t the only exciting thread in RNN research. For example,
Grid LSTMs by Kalchbrenner, et al.
(2015)
seem extremely
promising. Work using RNNs in generative models – such as Gregor, et
al.
(2015)
, Chung, et al.
(2015)
, or Bayer & Osendorfer
(2015)
– also seems very
interesting. The last few years have been an exciting time for
recurrent neural networks, and the coming ones promise to only be more
so!

RNN

RNN

RNN网络实质上是多个普通神经网络的连接，每个神经元向下一个传递信息，如下图所示:

RNN链式结构

“LSTMs”,a very special kind of recurrent neural network which
works,for many tasks,much much better tahn the standard version.

递归神经网络

Recurrent Neural Networks

Humans don’t start their thinking from scratch every second. As you
of previous words. You don’t throw everything away and start thinking
from scratch again. Your thoughts have persistence.

Traditional neural networks can’t do this, and it seems like a major
shortcoming. For example, imagine you want to classify what kind of
event is happening at every point in a movie. It’s unclear how a
events in the film to inform later ones.

Recurrent neural networks address this issue. They are networks with
loops in them, allowing information to persist.

$$x_t$$，和输出值 $$h_t$$
。一个循环保证信息一步一步在网络中传递。

In the above diagram, a chunk of neural network, class=”math inline”>$$A$$, looks at some input class=”math inline”>$$x_t$$ and outputs a value class=”math inline”>$$h_t$$. A loop allows information to be
passed from one step of the network to the next.

These loops make recurrent neural networks seem kind of mysterious.
However, if you think a bit more, it turns out that they aren’t all
that different than a normal neural network. A recurrent neural
network can be thought of as multiple copies of the same network, each
passing a message to a successor. Consider what happens if we unroll
the loop:

RNN 所取得成果的讨论留在 Andrej Karpathy 的博客里。RNN 真的很神奇！

——一种特殊的递归神经网络，在许多问题上比标准版本的 RNN

This chain-like nature reveals that recurrent neural networks are
intimately related to sequences and lists. They’re the natural
architecture of neural network to use for such data.

And they certainly are used! In the last few years, there have been
incredible success applying RNNs to a variety of problems: speech
recognition, language modeling, translation, image captioning… The
list goes on. I’ll leave discussion of the amazing feats one can
achieve with RNNs to Andrej Karpathy’s excellent blog post, The
Unreasonable Effectiveness of Recurrent Neural
Networks
.
But they really are pretty amazing.

Essential to these successes is the use of “LSTMs,” a very special
kind of recurrent neural network which works, for many tasks, much
much better than the standard version. Almost all exciting results
based on recurrent neural networks are achieved with them. It’s these
LSTMs that this essay will explore.

LSTM是什么？

LSTM(Long Short-Term Memory)

现状

Schmidhuber的实验室在 2006 年发表的方法。百度也使用了 CTC；苹果的 iPhone

Alexa 通过双向 LSTM 在家中与你交流，而谷歌使用 LSTM

Allo 中，也显著地提高了谷歌翻译的质量（从 2016

LSTM 的核心想法

The Core Idea Behind LSTMs

LSTM 的关键是细胞状态，即图中上方的水平线。

The key to LSTMs is the cell state, the horizontal line running
through the top of the diagram.

The cell state is kind of like a conveyor belt. It runs straight down
the entire chain, with only some minor linear interactions. It’s very
easy for information to just flow along it unchanged.

LSTM 可以通过所谓“门”的精细结构向细胞状态添加或移除信息。

The LSTM does have the ability to remove or add information to the
cell state, carefully regulated by structures called gates.

Gates are a way to optionally let information through. They are
composed out of a sigmoid neural net layer and a pointwise
multiplication operation.

S 形网络的输出值介于 0 和 1 之间，表示有多大比例的信息通过。0

The sigmoid layer outputs numbers between zero and one, describing how
much of each component should be let through. A value of zero means
“let nothing through,” while a value of one means “let everything
through!”

An LSTM has three of these gates, to protect and control the cell
state.

Variants on Long Short Term Memory

One popular LSTM variant, introduced by Gers & Schmidhuber (2000), is
adding “peephole connections.” This means that we let the gate layers
look at the cell state.

“peephole connections”

Another variation is to use coupled(耦合) forget and input gates.
Instead of separately deciding what to forget and what we should add
new information to, we make those decisions together. We only forget
when we’re going to input something in its place. We only input new
values to the state when we forget something older.

coupled

A slightly more dramatic variation on the LSTM is the Gated Recurrent
Unit, or GRU, introduced by Cho, et al. (2014). It combines the forget
and input gates into a single “update gate.” It also merges the cell
state and hidden state, and makes some other changes. The resulting
model is simpler than standard LSTM models, and has been growing
increasingly popular.

GRU

These are only a few of the most notable LSTM variants. There are lots
of others, like Depth Gated RNNs by Yao, et al. (2015). There’s also
some completely different approach to tackling long-term dependencies,
like Clockwork RNNs by Koutnik, et al. (2014).

长短期记忆的变种

Variants on Long Short Term Memory

What I’ve described so far is a pretty normal LSTM. But not all LSTMs
are the same as the above. In fact, it seems like almost every paper
involving LSTMs uses a slightly different version. The differences are
minor, but it’s worth mentioning some of them.

One popular LSTM variant, introduced by Gers & Schmidhuber (2000), is
adding “peephole connections.” This means that we let the gate layers
look at the cell state.

The above diagram adds peepholes to all the gates, but many papers
will give some peepholes and not others.

Another variation is to use coupled forget and input gates. Instead of
separately deciding what to forget and what we should add new
information to, we make those decisions together. We only forget when
we’re going to input something in its place. We only input new values
to the state when we forget something older.

GRU 比标准的 LSTM 模型更简单，因此正在变得流行起来。

A slightly more dramatic variation on the LSTM is the Gated Recurrent
Unit, or GRU, introduced by Cho, et al.
(2014)
. It combines the forget
and input gates into a single “update gate.” It also merges the cell
state and hidden state, and makes some other changes. The resulting
model is simpler than standard LSTM models, and has been growing
increasingly popular.

Koutnik 等人提出的 Clockwork RNN。

These are only a few of the most notable LSTM variants. There are lots
of others, like Depth Gated RNNs by Yao, et al.
(2015)
. There’s also some
completely different approach to tackling long-term dependencies, like
Clockwork RNNs by Koutnik, et al.
(2014)
.

Which of these variants is best? Do the differences matter? Greff, et
al. (2015)
do a nice comparison
of popular variants, finding that they’re all about the same.
Jozefowicz, et al.
(2015)

tested more than ten thousand RNN architectures, finding some that
worked better than LSTMs on certain tasks.

LSTM怎么做？

Long Short Term Memory networks – usually just called “LSTMs” – are a
special kind of RNN, capable of learning long-term dependencies. They
were introduced by Hochreiter & Schmidhuber (1997).

LSTMs are explicitly designed to avoid the long-term dependency
problem.Remembering information for long periods of time is
practically their default behavior, not something they struggle to
learn!

All recurrent neural networks have the form of a chain of repeating
modules of neural network. In standard RNNs, this repeating module
will have a very simple structure, such as a single tanh layer.

layer，如下图所示：

LSTM

[图片上传失败…(image-72c315-1521165904331)]

In the above diagram, each line carries an entire vector, from the
output of one node to the inputs of others. The pink circles represent
pointwise operations, like vector addition, while the yellow boxes are
learned neural network layers. Lines merging denote concatenation,
while a line forking denote its content being copied and the copies
going to different locations.

逐步解析 LSTM 的流程

Step-by-Step LSTM Walk Through

LSTM 的第一步要决定从细胞状态中舍弃哪些信息。这一决定由所谓“遗忘门层”的
S 形网络层做出。它接收 $$h_{t-1}$$

The first step in our LSTM is to decide what information we’re going
to throw away from the cell state. This decision is made by a sigmoid
layer called the “forget gate layer.” It looks at class=”math inline”>$$h_{t-1}$$ and class=”math inline”>$$x_t$$, and outputs a number between
$$0$$ and class=”math inline”>$$1$$ for each number in the cell state
$$C_{t-1}$$. A class=”math inline”>$$1$$ represents “completely keep this”
while a $$0$$ represents
“completely get rid of this.”

Let’s go back to our example of a language model trying to predict the
next word based on all the previous ones. In such a problem, the cell
state might include the gender of the present subject, so that the
correct pronouns can be used. When we see a new subject, we want to
forget the gender of the old subject.

S 形网络层确定哪些信息需要更新。第二部分，一个 tanh

The next step is to decide what new information we’re going to store
in the cell state. This has two parts. First, a sigmoid layer called
the “input gate layer” decides which values we’ll update. Next, a tanh
layer creates a vector of new candidate values, class=”math inline”>$$\tilde{C}_t$$, that could be added to
the state. In the next step, we’ll combine these two to create an
update to the state.

In the example of our language model, we’d want to add the gender of
the new subject to the cell state, to replace the old one we’re
forgetting.

$$i_t*\tilde{C}_t$$，这是新的候选值，根据我们对每个状态决定的更新值按比例进行缩放。

It’s now time to update the old cell state, class=”math inline”>$$C_{t-1}$$, into the new cell state
$$C_t$$. The previous steps
already decided what to do, we just need to actually do it.

We multiply the old state by class=”math inline”>$$f_t$$, forgetting the things we
decided to forget earlier. Then we add class=”math inline”>$$i_t*\tilde{C}_t$$. This is the new
candidate values, scaled by how much we decided to update each state
value.

In the case of the language model, this is where we’d actually drop
information, as we decided in the previous steps.

S

$$\tanh$$（把数值调整到 $$-1$$ 和 $$1$$ 之间）再和 S

Finally, we need to decide what we’re going to output. This output
will be based on our cell state, but will be a filtered version.
First, we run a sigmoid layer which decides what parts of the cell
state we’re going to output. Then, we put the cell state through class=”math inline”>$$\tanh$$ (to push the values to be
between $$-1$$ and class=”math inline”>$$1$$) and multiply it by the output of
the sigmoid gate, so that we only output the parts we decided to.

For the language model example, since it just saw a subject, it might
want to output information relevant to a verb, in case that’s what is
coming next. For example, it might output whether the subject is
singular or plural, so that we know what form a verb should be
conjugated into if that’s what follows next.

The Core Idea Behind LSTMs

The key to LSTMs is the cell state, the horizontal line running
through the top of the diagram.

The cell state is kind of like a conveyor belt(输送带). It runs
straight down the entire chain, with only some minor linear
interactions. It’s very easy for information to just flow along it
unchanged.

LSTM可以去除或增加cell state的信息,并被称为门(gates)的结构仔细调控。

Gates are a way to optionally let information through. They are
composed out of a sigmoid neural net layer and a pointwise
multiplication operation(逐点乘法运算).

forget gate layer

The sigmoid layer outputs numbers between zero and one, describing how
much of each component should be let through. A value of zero means
“let nothing through,” while a value of one means “let everything
through!”

An LSTM has three of these gates, to protect and control the cell
state.

Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)，原文以图文并茂的形式，深入浅出地为初学者介绍了
LSTM 网络。

Step-by-Step LSTM Walk Through

layer”的sigmoid layer完成。sigmoid
layer根据输入ht-1和xt，并为cell state
Ct-1中每个值输出一个介于0-1之间的值。当输出为 1

state可能包括当前主体的性别，因此可以使用正确的代词。

state的新信息。这步有两个部分。首先，被称为“input gate layer”的sigmoid
layer决定我们将更新哪些值。接下来，tanh层创建一个新的候选值向量Ct，可以添加到状态state中。在下一步中，我们将结合这两者来实现细胞状态cell
state的更新。

state中，以替换我们抛弃的旧主体性别信息。

input gate layer+ tanh layer

Ct。 之前的步骤已经决定要做什么，我们只需要真正做到这一点。

state，但将是一个过滤版本。 首先，我们运行一个sigmoid
layer，它决定我们要输出的cell state的哪些部分。 然后，将cell state

gate的输出，以便我们只输出决定输出的部分。

layer实现cell state的过滤，根据过滤版本的cell
state修改输出ht.

LSTM 网络

LSTM Networks

RNN，能够学习长期依赖性。由 Hochreiter 和
Schmidhuber（1997）提出的，并且在接下来的工作中被许多人改进和推广。LSTM

LSTM 被明确设计用来避免长期依赖性问题。长时间记住信息实际上是 LSTM

Long Short Term Memory networks – usually just called “LSTMs” – are a
special kind of RNN, capable of learning long-term dependencies. They
were introduced by Hochreiter & Schmidhuber
(1997)
, and
were refined and popularized by many people in following
work.1
They work tremendously well on a large variety of problems, and are
now widely used.

LSTMs are explicitly designed to avoid the long-term dependency
problem. Remembering information for long periods of time is
practically their default behavior, not something they struggle to
learn!

All recurrent neural networks have the form of a chain of repeating
modules of neural network. In standard RNNs, this repeating module
will have a very simple structure, such as a single tanh layer.

LSTM

LSTMs also have this chain like structure, but the repeating module
has a different structure. Instead of having a single neural network
layer, there are four, interacting in a very special way.

Don’t worry about the details of what’s going on. We’ll walk through
the LSTM diagram step by step later. For now, let’s just try to get
comfortable with the notation we’ll be using.

In the above diagram, each line carries an entire vector, from the
output of one node to the inputs of others. The pink circles represent
pointwise operations, like vector addition, while the yellow boxes are
learned neural network layers. Lines merging denote concatenation,
while a line forking denote its content being copied and the copies
going to different locations.

blog

鸣谢

Acknowledgments

Sutskever。我也非常感谢其他花时间帮助我的同事，包括 Dario Amodei 和
Jacob Steinhardt。我要特别感谢 Kyunghyun Cho

I’m grateful to a number of people for helping me better understand
LSTMs, commenting on the visualizations, and providing feedback on
this post.

feedback, especially Oriol
Vinyals
, Greg
, Jon
Shlens
, Luke
Vilnis
, and Ilya
Sutskever
. I’m also thankful to
many other friends and colleagues for taking the time to help me,
including Dario
Amodei
, and
Jacob Steinhardt. I’m
especially thankful to Kyunghyun Cho
for extremely thoughtful correspondence about my diagrams.

Before this post, I practiced explaining LSTMs during two seminar
series I taught on neural networks. Thanks to everyone who
participated in those for their patience with me, and for their
feedback.

1. In addition to the original authors, a lot of people contributed to
the modern LSTM. A non-comprehensive list is: Felix Gers, Fred
Cummins, Santiago Fernandez, Justin Bayer, Daan Wierstra, Julian
Togelius, Faustino Gomez, Matteo Gagliolo, and Alex
Graves
.

参考

发展展望

LSTM以后的发展方向：

1. Attention:Xu, et
al. (2015)
2. Grid LSTMs:Kalchbrenner, et
al. (2015)
3. RNN in generative models:Gregor, et
al. (2015)
,Chung,
et
al. (2015)
,Bayer
&
Osendorfer (2015)

长期依赖性问题

The Problem of Long-Term Dependencies

RNN

RNN 可以做到这一点，它们会非常有用。但它可以吗？嗯，这是有条件的。

“the clouds are in the sky

sky。在这种情况下，相关信息与它所在位置之间的距离很小，RNN

One of the appeals of RNNs is the idea that they might be able to
connect previous information to the present task, such as using
previous video frames might inform the understanding of the present
frame. If RNNs could do this, they’d be extremely useful. But can
they? It depends.

Sometimes, we only need to look at recent information to perform the
present task. For example, consider a language model trying to predict
the next word based on the previous ones. If we are trying to predict
the last word in “the clouds are in the sky,” we don’t need any
further context –– it’s pretty obvious the next word is going to be
sky. In such cases, where the gap between the relevant information and
the place that it’s needed is small, RNNs can learn to use the past
information.

speak fluent French.”

But there are also cases where we need more context. Consider trying
to predict the last word in the text “I grew up in France… I speak
fluent French.” Recent information suggests that the next word is
probably the name of a language, but if we want to narrow down which
language, we need the context of France, from further back. It’s
entirely possible for the gap between the relevant information and the
point where it is needed to become very large.

Unfortunately, as that gap grows, RNNs become unable to learn to
connect the information.

In theory, RNNs are absolutely capable of handling such “long-term
dependencies.” A human could carefully pick parameters for them to
solve toy problems of this form. Sadly, in practice, RNNs don’t seem
to be able to learn them. The problem was explored in depth by
Hochreiter (1991)
[German]

and Bengio, et al.
(1994)
,
who found some pretty fundamental reasons why it might be difficult.

Thankfully, LSTMs don’t have this problem!

1. 是什么？
2. 为什么？
3. 做什么？
4. 怎么做？

理解 LSTM 网络

Understanding LSTM Networks

【翻译】理解 LSTM

LSTM为什么产生？

The Problem of Long-Term Dependencies[1]

RNNs模型可以connect previous information to the present task,such as using previous video frames might inform the understanding of the present frame.

RNNs如何实现上述目标呢？这需要按情况而定。

”的最后一个词，我们不需要任何进一步的背景(上下文) –

Unfortunately, as that gap grows, RNNs become unable to learn to
connect the information.

dependencies.”,但是实际操作中，RNNs不能学习/训练这样的问题，即需要的过去信息n数量过大的情况下，RNNs将不再适用。The
problem was explored in depth by Hochreiter (1991)
[German]

and Bengio, et
al
.
(1994), who found some pretty fundamental reasons why it might be
difficult.

LSTM模型可以处理“long-term dependencies”的问题