Pytorch

torch.nn.utils.rnn:

pack_padded_sequence() pad_packed_sequence()

Notice:

• The padded embedding metrix must be sorted by the ground length of each sentence.
• The parameter "batch_first=True" controls the first demension of the embedding metrix is the batch_size (\(B\times T \times W_{emb}\)). Otherwise, the first demension is the length of the longest sentence (\(T\times B \times W_{emb}\))
• The two functions will inflence the padding word, especially on bidirectional RNN (the backward rnn will go through some padding words first and the forward rnn will go through some padding words last).

Examplt

a = torch.FloatTensor([[[2,3,4], [2, 3,1]], [[2,3,4], [2, 3,1]], [[2,4,5], [0, 0, 0]]])

packed_x = pack_padded_sequence(a, [2, 2, 1], batch_first=True)
# a[0,0,:] = torch.FloatTensor([0,0,0])

h0 = Variable(torch.randn(1, 3, 4))
lstm = nn.LSTM(3, 4,num_layers=1,batch_first=True,bidirectional=True)

rnn_packed, (h_last, c_last) = lstm(packed_x)

rnn_out, length = pad_packed_sequence(rnn_packed, batch_first=True)

Nan

Mask

When using mask on rnn outputs with \(-\infty\), it will cause the grad to be nan when back-propogating. But it's right when appling to attention. The \(softmax\) operation does not cause the nan problem.