Tutorial on word2vector using GloVe and Word2Vec
Tutorial on word2vector using GloVe and Word2Vec
2018-05-04 10:02:53
Some Important Reference Pages First:
Reference Page: https://github.com/IliaGavrilov/NeuralMachineTranslationBidirectionalLSTM/blob/master/1_Bidirectional_LSTM_Eng_to_French.ipynb
Glove Project Page: https://nlp.stanford.edu/projects/glove/
Word2Vec Project Page: https://code.google.com/archive/p/word2vec/
Pre-trained word2vec model: https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing
Gensim Tutorial: https://radimrehurek.com/gensim/models/word2vec.html
===================================
===== For the Glove
===================================
1. Download one of the pre-trained model from Glove project page and Unzip the files.
Wikipedia 2014 + Gigaword 5 (6B tokens, 400K vocab, uncased, 50d, 100d, 200d, & 300d vectors, 822 MB download): glove.6B.zip
Common Crawl (42B tokens, 1.9M vocab, uncased, 300d vectors, 1.75 GB download): glove.42B.300d.zip
Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB download): glove.840B.300d.zip
2. Install the needed packages:
pickle, numpy, re, pickle, collections, bcolz
3. Run the following demo to test the results (extract the feature of given single word).
Code:
import pickle
import numpy as np
import re, pickle, collections, bcolz # with open('./glove.840B.300d.txt', 'r', encoding="utf8") as f: with open('./glove.6B.200d.txt', 'r') as f:
lines = [line.split() for line in f] print('==>> begin to load Glove pre-trained models.')
glove_words = [elem[0] for elem in lines]
glove_words_idx = {elem:idx for idx, elem in enumerate(glove_words)} # is elem: idx equal to glove_words_idx[elem]=idx?
glove_vecs = np.stack(np.array(elem[1:], dtype=np.float32) for elem in lines) print('==>> save into .pkl files.')
pickle.dump(glove_words, open('./glove.6B.200d.txt'+'_glove_words.pkl', 'wb'))
pickle.dump(glove_words_idx, open('./glove.6B.200d.txt'+'_glove_words_idx.pkl', 'wb')) ## saving array using specific function.
def save_array(fname, arr):
c=bcolz.carray(arr, rootdir=fname, mode='w')
c.flush() save_array('./glove.6B.200d.txt'+'_glove_vecs'+'.dat', glove_vecs) def load_glove(loc):
return (bcolz.open(loc+'_glove_vecs.dat')[:],
pickle.load(open(loc+'_glove_words.pkl', 'rb')),
pickle.load(open(loc+'_glove_words_idx.pkl', 'rb'))) ###############################################
print('==>> Loading the glove.6B.200d.txt files.')
en_vecs, en_wv_word, en_wv_idx = load_glove('./glove.6B.200d.txt')
en_w2v = {w: en_vecs[en_wv_idx[w]] for w in en_wv_word}
n_en_vec, dim_en_vec = en_vecs.shape print('==>> shown one demo: "King"')
demo_vector = en_w2v['king']
print(demo_vector)
print("==>> Done !")
Results:
wangxiao@AHU$ python tutorial_Glove_word2vec.py
==>> begin to load Glove pre-trained models.
==>> save into .pkl files.
==>> Loading the glove.6B.200d.txt files.
==>> shown one demo: "King"
[-0.49346 -0.14768 0.32166001 0.056899 0.052572 0.20192 -0.13506 -0.030793 0.15614 -0.23004 -0.66376001 -0.27316001 0.10391 0.57334 -0.032355 -0.32765999 -0.27160001 0.32918999
0.41305 -0.18085 1.51670003 2.16490006 -0.10278 0.098019 -0.018946 0.027292 -0.79479998 0.36631 -0.33151001 0.28839999 0.10436 -0.19166 0.27326 -0.17519 -0.14985999 -0.072333
-0.54370999 -0.29728001 0.081491 -0.42673001 -0.36406001 -0.52034998 0.18455 0.44121 -0.32196 0.39172 0.11952 0.36978999 0.29229 -0.42954001 0.46653 -0.067243 0.31215999 -0.17216
0.48874 0.28029999 -0.17577 -0.35100999 0.020792 0.15974 0.21927001 -0.32499 0.086022 0.38927001 -0.65638 -0.67400998 -0.41896001 1.27090001 0.20857 0.28314999 0.58238 -0.14944001
0.3989 0.52680999 0.35714 -0.39100999 -0.55372 -0.56642002 -0.15762 -0.48004001 0.40448001 0.057518 -1.01569998 0.21754999 0.073296 0.15237001 -0.38361999 -0.75308001 -0.0060254 -0.26232001
-0.54101998 -0.34347001 0.11113 0.47685 -0.73229998 0.77596998 0.015216 -0.66327 -0.21144 -0.42964 -0.72689998 -0.067968 0.50601 0.039817 -0.27584001 -0.34794 -0.0474 0.50734001
-0.30777001 0.11594 -0.19211 0.3107 -0.60075003 0.22044 -0.36265001 -0.59442002 -1.20459998 0.10619 -0.60277998 0.21573 -0.35361999 0.55473 0.58094001 0.077259 1.0776 -0.1867
-1.51680005 0.32418001 0.83332998 0.17366 1.12320006 0.10863 0.55888999 0.30799001 0.084318 -0.43178001 -0.042287 -0.054615 0.054712 -0.80914003 -0.24429999 -0.076909 0.55216002 -0.71895999
0.83319002 0.020735 0.020472 -0.40279001 -0.28874001 0.23758 0.12576 -0.15165 -0.69419998 -0.25174001 0.29591 0.40290001 -1.0618 0.19847 -0.63463002 -0.70842999 0.067943 0.57366002
0.041122 0.17452 0.19430999 -0.28641 -1.13629997 0.45116001 -0.066518 0.82615 -0.45451999 -0.85652 0.18105 -0.24187 0.20152999 0.72298002 0.17415 -0.87327999 0.69814998 0.024706
0.26174 -0.0087155 -0.39348999 0.13801 -0.39298999 -0.23057 -0.22611 -0.14407 0.010511 -0.47389001 -0.15645 0.28601 -0.21772 -0.49535 0.022209 -0.23575 -0.22469001 -0.011578 0.52867001 -0.062309 ]
==>> Done !
4. Extract the feature of the whole sentense.
Given one sentense, such as "I Love Natural Language Processing", we can translate this sentense into a matrix representation. Specifically, we represent each word of the sentense with a vector which lies in a continuous space (as shown above). You can also see this blog: https://blog.csdn.net/fangqingan_java/article/details/50493948 to futher understand this process.
但是有如下的疑问：
1. 给定的句子，长短不一，每一个单词的维度固定，但是总的句子的长度不固定，怎么办？
只好用 padding 的方法了，那么，怎么进行 padding 呢？
For CNN architecture usually the input is (for each sentence) a vector of embedded words [GloVe(w0), GloVe(w1), ..., GloVe(wN)] of fixed length N.
So commonly you preset the maximum sentence length and just pad the tail of the input vector with zero vectors (just as you did). Marking the beginning and the end of the sentence is more relevant for the RNN, where the sentence length is expected to be variable and processing is done sequentially. Having said that, you can add a new dimension to the GloVe vector, setting it to zero for normal words, and arbitrarily to (say) 0.5 for BEGIN, and 1 for END and a random negative value for OOV.
As for unknown words, you should ;) encounter them very rear, otherwise you might consider training the embeddings by yourself. 那么，是什么意思呢？
对于 CNN 的网络来说，由于需要处理 网格状的 data，所以，需要将句子进行填充，以得到完整的 matrix，然后进行处理；
对于 RNN/LSTM 的网络来说，是可以按照时刻，顺序处理的，所以呢？就不需要 padding 了？
2. Here is another tutorial on Word2Vec using deep learning tools --- Keras.
from numpy import array
from keras.preprocessing.text import one_hot
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.embeddings import Embedding
# define documents
docs = ['Well done!',
'Good work',
'Great effort',
'nice work',
'Excellent!',
'Weak',
'Poor effort!',
'not good',
'poor work',
'Could have done better.']
# define class labels
labels = array([,,,,,,,,,])
# integer encode the documents
vocab_size =
encoded_docs = [one_hot(d, vocab_size) for d in docs]
print("==>> encoded_docs: ")
print(encoded_docs)
# pad documents to a max length of words
max_length =
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')
print("==>> padded_docs: ")
print(padded_docs)
# define the model
model = Sequential()
model.add(Embedding(vocab_size, , input_length=max_length))
model.add(Flatten())
model.add(Dense(, activation='sigmoid'))
# compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
# summarize the model
print("==>> model.summary()")
print(model.summary())
# fit the model
model.fit(padded_docs, labels, nb_epoch=, verbose=)
# evaluate the model
loss, accuracy = model.evaluate(padded_docs, labels, verbose=)
print("==>> the final Accuracy: ")
print('Accuracy: %f' % (accuracy*))
The output is:
Using TensorFlow backend.
==>> encoded_docs:
[[, ], [, ], [, ], [, ], [], [], [, ], [, ], [, ], [, , , ]]
==>> padded_docs:
[[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]]
WARNING:tensorflow:From /usr/local/lib/python2./dist-packages/keras/backend/tensorflow_backend.py:: calling reduce_prod (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python2./dist-packages/keras/backend/tensorflow_backend.py:: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
==>> model.summary()
____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
embedding_1 (Embedding) (None, , ) embedding_input_1[][]
____________________________________________________________________________________________________
flatten_1 (Flatten) (None, ) embedding_1[][]
____________________________________________________________________________________________________
dense_1 (Dense) (None, ) flatten_1[][]
====================================================================================================
Total params:
Trainable params:
Non-trainable params:
____________________________________________________________________________________________________
None
-- ::59.053198: I tensorflow/core/platform/cpu_feature_guard.cc:] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
==>> the final Accuracy:
Accuracy: 89.999998
3. Maybe translate the sentence into vectors using skip-thought vectors is another goog choice. you can also check this tutorial from: http://www.cnblogs.com/wangxiaocvpr/p/7277025.html
4. But we still want to use GloVe to extract the vector of each word and concatenate them into a long vector.
import os
import numpy as np
from collections import OrderedDict
import re, pickle, collections, bcolz
import pdb seq_home = '/dataset/language-dataset/'
seqlist_path = '/dataset/language-train-video-list.txt'
output_path = 'data/language-train-video-list.pkl' def load_glove(loc):
return (bcolz.open(loc+'_glove_vecs.dat')[:],
pickle.load(open(loc+'_glove_words.pkl', 'rb')),
pickle.load(open(loc+'_glove_words_idx.pkl', 'rb'))) pre_trained_model_path = '/glove/glove.6B.200d.txt' ###############################################
print('==>> Loading the glove.6B.200d.txt files.')
en_vecs, en_wv_word, en_wv_idx = load_glove(pre_trained_model_path)
en_w2v = {w: en_vecs[en_wv_idx[w]] for w in en_wv_word}
n_en_vec, dim_en_vec = en_vecs.shape with open(seqlist_path,'r') as fp:
seq_list = fp.read().splitlines() data = {}
for i,seq in enumerate(seq_list):
img_list = sorted([p for p in os.listdir(seq_home+seq+'/imgs/') if os.path.splitext(p)[1] == '.jpg']) ## image list
gt = np.loadtxt(seq_home+seq+'/groundtruth_rect.txt', delimiter=',') ## gt files
language_txt = open(seq_home+seq+'/language.txt', "rw+") ## natural language description files line = language_txt.readline()
print("==>> language: %s" % (line)) gloveVector = []
test_txtName = seq_home+seq+'/glove_vector.txt'
f=file(test_txtName, "w") word_list = line.split(' ')
for word_idx in range(len(word_list)):
current_word = word_list[word_idx]
current_GloVe_vector = en_w2v[current_word] ## vector dimension is: 200-D gloveVector = np.concatenate((gloveVector, current_GloVe_vector), axis=0)
f.write(str(current_GloVe_vector))
f.write('\n')
f.close() print(i) assert len(img_list) == len(gt), "Lengths do not match!!" if gt.shape[1]==8:
x_min = np.min(gt[:,[0,2,4,6]],axis=1)[:,None]
y_min = np.min(gt[:,[1,3,5,7]],axis=1)[:,None]
x_max = np.max(gt[:,[0,2,4,6]],axis=1)[:,None]
y_max = np.max(gt[:,[1,3,5,7]],axis=1)[:,None]
gt = np.concatenate((x_min, y_min, x_max-x_min, y_max-y_min),axis=1) data[seq] = {'images':img_list, 'gt':gt, 'gloveVector': gloveVector} with open(output_path, 'wb') as fp:
pickle.dump(data, fp, -1)
The Glove vector can be saved into a txt file for each video, as shown in following screenshots.
[ 0.19495 0.60610002 -0.077356 0.017301 -0.51370001 0.22421999
-0.80773002 0.022378 0.30256 1.06669998 -0.10918 0.57902998
0.23986 0.1691 0.0072384 0.42197999 -0.20458999 0.60271001
0.19188 -0.19616 0.33070001 3.20020008 -0.18104 0.20784
0.49511001 -0.42258999 0.022125 0.24379 0.16714001 -0.20909999
-0.12489 -0.51766998 -0.13569 -0.25979999 -0.17961 -0.47663
-0.89539999 -0.27138999 0.17746 0.45827001 0.21363001 0.22342999
-0.049342 0.34286001 -0.32315001 0.27469999 0.95993 -0.25979
0.21250001 -0.21373001 0.19809 0.15455 -0.48581001 0.38925001
0.33747 -0.27897999 0.19371 -0.45872 -0.054217 -0.24022999
0.59153003 0.12453 -0.21302 0.058223 -0.046671 -0.011614
-0.32025999 0.64120001 -0.28718999 0.035138 0.39287001 -0.030683
0.083529 -0.010964 -0.62427002 -0.13575 -0.38468999 0.11454
-0.61037999 0.12594999 -0.17633 -0.21415 -0.37013999 0.21763
0.055366 -0.25242001 -0.45475999 -0.28105 0.18911 -1.58539999
0.64841002 -0.34621999 0.59254003 -0.39034 -0.44258001 0.20562001
0.44395 0.23019999 -0.35018 -0.36090001 0.62993002 -0.34698999
-0.31964999 -0.17361 0.51714998 0.68493003 -0.15587001 1.42550004
-0.94313997 0.031951 -0.26210001 -0.089544 0.22437 -0.050374
0.035414 -0.070493 0.17037 -0.38598001 0.0095626 0.26363
0.72125 -0.13797 0.70602 -0.50839001 -0.49722001 -0.48706001
0.16254 0.025619 0.33572 -0.64160001 -0.32541999 0.21896
0.05331 0.082011 0.12038 0.088849 -0.04651 -0.085435
0.036835 -0.14695001 -0.25841001 -0.043812 0.053209 -0.48954999
1.73950005 0.99014997 0.09395 -0.20236 -0.050135 0.18337999
0.22713999 0.83146 -0.30974001 0.51995999 0.068264 -0.28237
-0.30096999 -0.031014 0.024615 0.4294 -0.085036 0.051913
0.31251001 -0.34443 -0.085145 0.024975 0.0082017 0.17241
-0.66000998 0.0058962 -0.055387 -0.22315 0.35721999 -0.18962
0.25819999 -0.24685 -0.79571998 -0.09436 -0.10271 0.13713001
1.48660004 -0.57113999 -0.52649999 -0.25181001 -0.40792 -0.18612
0.040009 0.11557 0.017987 0.27149001 -0.14252 -0.087756
0.15196 0.064926 0.01651 -0.25334001 0.27437001 0.24246
0.018494 0.22473 ]
[ 4.32489991e-01 5.34709990e-01 -1.83240008e-02 1.56369999e-01
6.69689998e-02 7.63140023e-02 -7.16499984e-01 2.41520002e-01
1.70919999e-01 8.96220028e-01 -2.00739995e-01 1.88370004e-01
4.53570008e-01 -1.86419994e-01 -3.41060013e-01 -3.10570002e-02
-4.90720011e-02 6.55830026e-01 -7.43409991e-03 1.54770002e-01
1.55849993e-01 3.11129999e+00 -4.12149996e-01 1.89439997e-01
1.86639994e-01 -2.00540006e-01 6.63070008e-02 3.90180014e-02
6.06740005e-02 -3.17880005e-01 1.82290003e-02 -6.80719972e-01
-1.33360000e-02 2.32099995e-01 -8.23459998e-02 -2.68420011e-01
-3.51870000e-01 -2.03860000e-01 6.69749975e-02 3.02689999e-01
1.84330001e-01 1.44429997e-01 1.38980001e-01 1.93629991e-02
-4.09139991e-01 2.24950001e-01 8.02020013e-01 -3.05290014e-01
2.22399995e-01 -6.10979982e-02 3.48910004e-01 1.13349997e-01
-4.14329991e-02 7.31640011e-02 3.17719996e-01 -2.59590000e-01
3.17759991e-01 -1.78080007e-01 2.12380007e-01 1.07529998e-01
4.37519997e-01 -1.81329995e-01 -1.53909996e-01 7.27130026e-02
4.14730012e-01 2.73079991e-01 -1.06940001e-01 7.45599985e-01
3.02549988e-01 1.49999997e-02 6.33790016e-01 1.03399999e-01
2.18640000e-01 -7.43470015e-03 -7.09949970e-01 4.60739993e-02
-8.34990025e-01 1.84519999e-02 -2.75200009e-01 -2.48000007e-02
-1.21320002e-01 3.29720005e-02 1.07320003e-01 2.18070000e-01
2.79079992e-02 -2.99050003e-01 -3.06030005e-01 -3.17970008e-01
2.10580006e-01 -1.16180003e+00 4.57720011e-01 -2.60210007e-01
3.76190007e-01 -2.16749996e-01 -1.68540001e-01 1.77729994e-01
-1.46709993e-01 2.67820001e-01 -5.86109996e-01 -4.98769999e-01
3.83679986e-01 2.51020014e-01 -3.90020013e-01 -6.41409993e-01
5.34929991e-01 2.76479989e-01 -4.32830006e-01 1.61350000e+00
-1.58399999e-01 -2.03970000e-01 -2.18089998e-01 -9.74659983e-04
-7.48440027e-02 3.78959998e-02 -4.03719991e-01 -1.74640000e-01
4.11469996e-01 -2.80400008e-01 -3.42570007e-01 5.72329983e-02
7.18890011e-01 -1.30419999e-01 5.75810015e-01 -3.27690005e-01
7.75609985e-02 -2.66130000e-01 4.82820012e-02 1.24679998e-01
1.29319996e-01 7.65379965e-02 -3.85809988e-01 3.72209996e-01
-1.67280003e-01 1.86330006e-01 -2.20280007e-01 3.27170014e-01
1.74089998e-01 -2.51549989e-01 -5.47109991e-02 -2.48549998e-01
-2.24260002e-01 -3.80129993e-01 5.53080022e-01 -1.52050003e-01
1.12820005e+00 3.70869994e-01 -4.62579988e-02 -3.56009990e-01
-1.72120005e-01 1.23460002e-01 6.58179998e-01 5.05930007e-01
-4.99610007e-01 1.97339997e-01 1.37759998e-01 3.98500008e-04
-1.38300002e-01 -6.67430013e-02 -7.20809996e-02 4.00720000e-01
-3.77620012e-01 -9.86199975e-02 2.04119995e-01 -3.42869997e-01
-2.01509997e-01 -1.18270002e-01 -1.60109997e-01 1.63570002e-01
-3.79029989e-01 -2.45529994e-01 -5.31699993e-02 -4.19999994e-02
1.82099998e-01 2.23959997e-01 4.50800002e-01 -2.03030005e-01
-6.16330028e-01 -2.02739999e-01 2.18419999e-01 3.66939992e-01
7.68440008e-01 -2.92189986e-01 -9.80089977e-02 -2.92939991e-01
-1.93189994e-01 1.45720005e-01 2.45150000e-01 -7.11840019e-02
3.97929996e-01 -3.33019998e-03 -4.01179999e-01 1.39760002e-01
8.27540010e-02 -1.29600003e-01 -3.05500001e-01 8.98869988e-03
2.26559997e-01 3.21410000e-01 -4.29780006e-01 4.74779993e-01]
[ 5.73459983e-01 5.41700006e-01 -2.34770000e-01 -3.62399995e-01
4.03699994e-01 1.13860004e-01 -4.49330002e-01 -3.09909999e-01
-5.34110004e-03 5.84259987e-01 -2.59559993e-02 4.93930012e-01
-3.72090004e-02 -2.84280002e-01 9.76959988e-02 -4.89069998e-01
2.60269996e-02 3.76489997e-01 5.77879995e-02 -4.68070000e-01
8.12880024e-02 3.28250003e+00 -6.36900008e-01 3.79559994e-01
3.81670007e-03 9.36070010e-02 -1.28549993e-01 1.73800007e-01
1.05219997e-01 2.86480010e-01 2.10889995e-01 -4.70759988e-01
2.77330000e-02 -1.98029995e-01 7.63280019e-02 -8.46289992e-01
-7.97079980e-01 -3.87430012e-01 -3.04220002e-02 -2.68489987e-01
4.85850006e-01 1.28950000e-01 3.83540004e-01 3.87219995e-01
-3.85239989e-01 1.90750003e-01 4.89980012e-01 1.32780001e-01
1.07920002e-02 2.67699987e-01 1.78120002e-01 -1.14330001e-01
-3.34939986e-01 8.73059988e-01 7.58750021e-01 -3.03779989e-01
-1.56259999e-01 1.20850001e-03 2.33219996e-01 2.79529989e-01
-1.84939995e-01 -1.41460001e-01 -1.89689994e-01 -3.83859985e-02
3.58740002e-01 6.55129999e-02 6.05649985e-02 6.63389981e-01
-8.32519978e-02 6.51630014e-02 5.17610013e-01 1.61709994e-01
4.60110009e-01 1.63880005e-01 -1.23989999e-01 3.11219990e-01
-1.54119998e-01 -1.09169997e-01 -4.25509989e-01 1.14179999e-01
2.51370013e-01 -5.61579987e-02 -2.59270012e-01 2.81630009e-01
-1.80939995e-02 1.60650000e-01 -4.85060006e-01 -9.89030004e-01
2.50220001e-01 -1.67359993e-01 4.14739996e-01 1.77010000e-01
4.24070001e-01 1.10880002e-01 -1.83599994e-01 -1.24100000e-01
-3.47799987e-01 9.90779996e-02 -2.23810002e-01 -1.12450004e-01
-2.11559996e-01 3.07060010e-03 -2.36070007e-01 2.72610001e-02
3.64300013e-01 3.99219990e-02 -1.83689997e-01 1.22660005e+00
-7.76400030e-01 -6.62249982e-01 1.57239996e-02 -1.49690002e-01
8.46489966e-02 2.68139988e-01 -1.67649999e-01 -3.19420010e-01
2.84940004e-01 -7.00000003e-02 1.20099997e-02 -1.22189999e-01
5.63099980e-01 -3.19999993e-01 5.01089990e-01 -1.02090001e-01
4.65750009e-01 -7.15420008e-01 1.72930002e-01 5.82589984e-01
7.83839971e-02 -3.38440016e-02 -2.51289994e-01 3.65029991e-01
3.15780006e-02 -6.57779992e-01 5.47499992e-02 8.71890008e-01
1.24550000e-01 -4.58770007e-01 -2.69650012e-01 -4.67790008e-01
-2.85780011e-03 1.78100005e-01 6.39689982e-01 1.39950007e-01
9.75960016e-01 1.18359998e-01 -6.39039993e-01 -1.54159993e-01
6.52619973e-02 2.43290007e-01 6.64759994e-01 2.50690013e-01
-1.02519996e-01 -3.28390002e-01 -8.55590031e-02 -1.27739999e-02
-1.94309995e-01 5.61389983e-01 -3.57329994e-01 -2.03439996e-01
-1.24130003e-01 -3.44309986e-01 -2.32960001e-01 -2.11870000e-01
8.53869990e-02 7.00630024e-02 -1.98029995e-01 -2.60230005e-02
-3.90370011e-01 8.00019979e-01 4.05770004e-01 -7.98629969e-02
3.52629989e-01 -3.40429991e-01 3.96759987e-01 2.28619993e-01
-3.50279987e-01 -4.73439991e-01 5.97419977e-01 -1.16570003e-01
1.05519998e+00 -4.15699989e-01 -8.05519968e-02 -5.65709993e-02
-1.66219994e-01 1.92739993e-01 -9.51749980e-02 -2.07810000e-01
1.56200007e-01 5.02309985e-02 -2.79150009e-01 4.37420011e-01
-3.12370002e-01 1.31940007e-01 -3.32780004e-01 1.88769996e-01
-2.34219998e-01 5.44179976e-01 -2.30690002e-01 3.49469990e-01]
[ -1.43299997e-01 -6.54269993e-01 -4.71419990e-01 -2.22780004e-01
-1.74260005e-01 6.43490016e-01 -7.60049999e-01 -8.42769966e-02
2.19850004e-01 -4.24650013e-01 -2.56919991e-02 -3.27909999e-02
-3.98149997e-01 -1.26000000e-02 6.31869972e-01 -6.31640017e-01
-2.70480007e-01 2.09920004e-01 5.72669983e-01 8.88589993e-02
2.24289998e-01 1.78419995e+00 8.73669982e-01 -2.23949999e-01
4.91869986e-01 6.86379969e-02 6.80689991e-01 1.85680002e-01
-2.85780013e-01 -1.06030002e-01 -3.07460010e-01 1.06180003e-02
-1.62290007e-01 -3.62700000e-02 8.02920014e-02 -2.17669994e-01
-7.92110026e-01 -7.06340015e-01 -2.89400011e-01 -1.51869997e-01
3.09009999e-01 -1.41849995e-01 -4.18920010e-01 -1.66299999e-01
-1.68510005e-01 1.04549997e-01 -3.49110007e-01 -1.21399999e+00
3.21929991e-01 1.88219994e-01 -3.50019991e-01 -2.23629996e-01
3.85280013e-01 1.40709996e-01 1.34890005e-01 3.89420003e-01
1.82109997e-01 1.32369995e-01 -2.60610014e-01 5.10110021e-01
1.26430005e-01 -2.94450015e-01 -1.34350002e-01 -1.44600004e-01
1.98740005e-01 -3.41720015e-01 2.40710005e-01 -6.08640015e-02
8.60409975e-01 8.92559998e-04 3.46020013e-01 -2.53780007e-01
2.53749996e-01 1.20920002e-01 -3.82739991e-01 1.55090000e-02
-6.41290024e-02 2.54729986e-01 -1.78489998e-01 2.37949997e-01
3.26880008e-01 2.03940004e-01 -5.65020025e-01 3.87109995e-01
-7.00220019e-02 -4.63999987e-01 -1.98210001e-01 -7.27079988e-01
5.36920011e-01 -9.09250021e-01 -2.36699998e-01 -6.20739982e-02
2.38769993e-01 3.24860007e-01 5.51190019e-01 -4.07079995e-01
1.40870005e-01 2.83569992e-01 -1.33379996e-01 -3.99740010e-01
-1.66620001e-01 4.43010002e-01 -6.85970008e-01 1.54200003e-01
6.58490002e-01 2.95630004e-02 2.61550009e-01 1.10969996e+00
-6.78719997e-01 -1.91960007e-01 7.45159984e-02 -3.53740007e-01
-7.15600014e-01 4.14079994e-01 -3.37280005e-01 2.05780007e-02
2.17340007e-01 6.48889989e-02 -3.51080000e-01 -2.44440004e-01
-1.98780000e-01 3.02210003e-01 -1.32180005e-01 -4.86490011e-01
-3.55599999e-01 2.83119995e-02 7.04949975e-01 7.35610008e-01
5.98779976e-01 -3.94160002e-01 -3.96349996e-01 2.55739987e-01
1.30939996e+00 -6.69730008e-02 4.72090006e-01 3.20030004e-01
-2.39020005e-01 1.87910005e-01 2.14310005e-01 3.10609996e-01
-7.22400010e-01 -2.59559989e-01 7.50709996e-02 5.45759976e-01
1.33249998e+00 5.97169995e-01 -7.56640017e-01 2.30540007e-01
3.45209986e-01 8.61710012e-01 5.65129995e-01 1.33000001e-01
4.27370012e-01 -4.36790008e-03 -5.15600026e-01 3.94329995e-01
-1.58559993e-01 -1.29820004e-01 -5.47240004e-02 -1.01789999e+00
6.39170036e-02 -6.47679985e-01 -7.53360018e-02 1.49210006e-01
5.53300023e-01 -6.24869987e-02 -2.01309994e-01 1.70489997e-01
-3.22829992e-01 2.43039995e-01 2.64449995e-02 -9.52059999e-02
3.80879998e-01 -3.69230002e-01 4.54270005e-01 -3.31169993e-01
-1.93750001e-02 1.59300007e-02 1.85659993e-02 -5.27350008e-01
1.05949998e+00 -4.31519985e-01 2.57169992e-01 -9.44539979e-02
7.53460005e-02 -2.06599995e-01 3.21940005e-01 -1.23549998e-02
4.72169995e-01 2.44020000e-01 -3.63059998e-01 1.71650007e-01
-3.50499988e-01 1.40990004e-01 1.18069999e-01 3.65170002e-01
-1.65889993e-01 -3.93799990e-02 -3.70829999e-01 2.03909993e-01]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[ 5.24869978e-01 -1.19410001e-01 -2.02419996e-01 -6.23929977e-01
-1.53799996e-01 -6.63369969e-02 -3.68499994e-01 2.86490005e-02
1.37950003e-01 -5.87819993e-01 6.02090001e-01 2.60540005e-02
7.07889974e-01 1.20329998e-01 -1.74310002e-02 4.03360009e-01
-3.19680005e-01 -2.50609994e-01 1.60889998e-01 2.47649997e-01
7.79359996e-01 2.74070001e+00 1.19589999e-01 -2.67529994e-01
-3.82809997e-01 -3.36580008e-01 1.41039997e-01 -4.65480000e-01
-8.92079994e-02 2.22540006e-01 -3.60740013e-02 -7.10140020e-02
6.23199999e-01 3.22770000e-01 4.15650010e-01 -3.68530005e-02
-5.82859993e-01 -6.26510024e-01 -3.26169990e-02 2.74789989e-01
-2.66950011e-01 5.27690016e-02 -1.09500003e+00 -1.99760008e-03
-7.49390006e-01 -1.89999994e-02 -1.87619999e-01 -5.19330025e-01
1.71590000e-01 4.40690011e-01 1.90789998e-01 -4.57339995e-02
-2.43069995e-02 2.32710004e-01 1.02179997e-01 -6.02790006e-02
-2.63680011e-01 -1.49090007e-01 4.33890015e-01 -8.51420015e-02
-6.61419988e-01 -6.23379983e-02 -1.37920007e-01 4.54079986e-01
-1.51400000e-01 1.14929996e-01 5.48650026e-01 2.82370001e-01
-2.55129993e-01 1.11149997e-01 -8.47479999e-02 -9.66319963e-02
4.88200009e-01 7.48070002e-01 -6.74910009e-01 3.21949989e-01
-8.98810029e-01 -3.58159989e-01 8.54640007e-02 -1.67390004e-01
-1.60109997e-01 -1.06349997e-01 -2.96649992e-01 -3.03389996e-01
2.29310006e-01 6.51580021e-02 1.70359999e-01 -2.08039999e-01
-1.12930000e-01 -1.58399999e-01 -1.88710004e-01 5.38689971e-01
-3.58050019e-02 3.26770008e-01 -3.18859994e-01 2.01079994e-02
-1.15209997e-01 -3.67400013e-02 5.97510003e-02 9.09829974e-01
1.55420005e-01 1.63939998e-01 3.57499987e-01 -4.75789994e-01
-3.53519991e-02 -1.30729997e+00 2.40319997e-01 1.24170005e+00
4.63180006e-01 -2.90019996e-02 6.63789988e-01 -2.20129997e-01
6.57410026e-01 -2.34209999e-01 -7.43539989e-01 8.95029977e-02
-1.41269997e-01 3.54860008e-01 -6.87049985e-01 -9.53029990e-02
-1.59720004e-01 -4.36020009e-02 5.39509989e-02 -1.33180007e-01
4.47530001e-01 1.41289994e-01 2.71119997e-02 1.18179999e-01
2.02419996e-01 -5.98179996e-01 -6.83469996e-02 -1.58150002e-01
2.42929995e-01 8.88189971e-02 -1.16219997e-01 -3.82679999e-01
1.30649999e-01 -3.00619990e-01 -8.46930027e-01 1.90159995e-02
6.41009986e-01 -8.42439979e-02 7.83680022e-01 -2.63660014e-01
1.98839998e+00 -7.38060027e-02 7.21970022e-01 4.84739989e-02
1.62070006e-01 3.40840012e-01 -1.85289994e-01 2.85659999e-01
-2.35440001e-01 -2.69939989e-01 -5.27239978e-01 2.74859995e-01
6.56930029e-01 2.20569998e-01 -6.80390000e-01 3.59100014e-01
2.51430005e-01 7.24399984e-02 2.58690000e-01 3.22510004e-01
3.57239991e-02 -3.88200015e-01 1.18890002e-01 -2.90010005e-01
-6.32229984e-01 -1.48519993e-01 -1.21040002e-01 -1.74060002e-01
2.48679996e-01 4.74539995e-01 -2.50710011e-01 -5.23679972e-01
-1.82730004e-01 -8.30529988e-01 6.15109980e-01 -3.84620011e-01
9.74919975e-01 4.49510008e-01 -5.42559981e-01 -3.18340003e-01
-5.07809997e-01 -2.73739994e-02 -1.26379997e-01 5.42479992e-01
4.85489994e-01 8.55289996e-02 -1.42509997e-01 8.77849981e-02
2.47189999e-01 -2.38490000e-01 6.14570007e-02 -4.23129983e-02
3.57470006e-01 9.73920003e-02 -3.58500004e-01 1.70269996e-01]
[ 7.54669979e-02 -2.92360008e-01 -2.60369986e-01 -2.81670004e-01
1.60970002e-01 -1.94720000e-01 -2.82059997e-01 4.91409987e-01
-4.11189999e-03 9.58790034e-02 3.33959997e-01 1.68740004e-02
2.06159994e-01 -1.57429993e-01 -1.61379993e-01 2.15379998e-01
-1.48359999e-01 2.54599992e-02 -4.36809987e-01 -6.36610016e-02
5.52609980e-01 3.08189988e+00 -1.17839999e-01 -4.61309999e-01
-2.76089996e-01 2.09480003e-01 -2.05440000e-01 -5.71550012e-01
3.34479988e-01 1.59130007e-01 2.54360004e-03 1.80040002e-01
1.34719998e-01 -9.74040031e-02 3.55369985e-01 -4.74280000e-01
-7.92569995e-01 -5.44179976e-01 2.42999997e-02 6.35990024e-01
1.23369999e-01 -1.29130006e-01 -2.65650004e-01 -2.49569997e-01
-5.21990001e-01 -4.05229986e-01 4.84030008e-01 1.83729995e-02
2.30389997e-01 6.21379986e-02 -1.92919999e-01 2.95060009e-01
-3.57930005e-01 1.67019993e-01 3.18679988e-01 -3.60540003e-01
-1.09779999e-01 -1.56320006e-01 4.59210008e-01 9.64749977e-02
-3.77000004e-01 -7.76150003e-02 -4.88990009e-01 2.05750003e-01
5.05429983e-01 5.34190014e-02 -2.59779990e-01 5.10420024e-01
9.75209996e-02 3.26059997e-01 1.43539995e-01 2.27149995e-03
4.86149997e-01 4.69379991e-01 -4.11240011e-01 -1.71480000e-01
-3.97439986e-01 -2.89009988e-01 -1.77560002e-01 3.70009989e-02
3.48300010e-01 1.59339994e-01 -7.42810011e-01 1.88970000e-01
4.36850004e-02 5.72080016e-01 -6.70159996e-01 -4.39470001e-02
-2.83360004e-01 -3.19959998e-01 -2.04040006e-01 -8.78980011e-02
-1.57240003e-01 2.18180008e-02 -5.67569971e-01 6.32960021e-01
-1.00970000e-01 -6.55760020e-02 5.82689978e-03 3.30349989e-02
3.97830009e-01 -3.11659992e-01 -6.10889971e-01 2.75590003e-01
1.00079998e-01 -4.19900000e-01 6.35599997e-03 1.87170005e+00
3.14729989e-01 -3.60040009e-01 8.13839972e-01 -2.17099994e-01
-1.84589997e-02 -2.26319999e-01 1.45850003e-01 -1.43500000e-01
-4.14239988e-02 5.59740007e-01 -6.67519987e-01 -2.19589993e-01
1.90109998e-01 3.30150008e-01 6.12900019e-01 4.67709988e-01
4.20260012e-01 -5.28190017e-01 2.31650006e-02 3.29100005e-02
4.73060012e-01 1.40060000e-02 -1.73960000e-01 -4.43619996e-01
4.13769990e-01 -2.06790000e-01 3.92830014e-01 3.02109987e-01
7.31339976e-02 4.21640016e-02 -9.27100003e-01 -4.76139992e-01
2.43100002e-01 -1.33790001e-01 -2.22379997e-01 -4.14570011e-02
1.58500004e+00 3.74810010e-01 2.59939991e-02 -2.42719993e-01
3.05779994e-01 1.46870002e-01 1.16659999e-01 -2.94179991e-02
-7.83390030e-02 -2.25119993e-01 1.33149996e-01 -6.48420006e-02
-2.86870003e-01 -1.05600003e-02 -3.46679986e-01 4.21449989e-02
-6.00409985e-01 8.24810028e-01 3.10220003e-01 1.64890006e-01
-7.29210004e-02 1.93939999e-01 -9.84980017e-02 -2.03830004e-02
-4.09090012e-01 -1.04039997e-01 1.91689998e-01 -1.59689993e-01
3.80259991e-01 6.28019989e-01 2.59499997e-01 -3.33669990e-01
-7.33330011e-01 -4.07429993e-01 6.84230030e-01 -6.63380027e-02
5.04360020e-01 -2.89829999e-01 -3.90859991e-01 -4.59310003e-02
-2.66240001e-01 -1.67699993e-01 -1.50370002e-01 1.48279995e-01
-2.81430006e-01 -1.70870006e-01 -2.55760014e-01 -5.62830009e-02
-1.66500002e-01 3.51060003e-01 4.10320014e-02 2.73110002e-01
3.00200004e-02 1.64649993e-01 -8.41889977e-02 5.75059988e-02]
[ 5.46909988e-01 -7.55890012e-01 -9.20799971e-01 -8.20680022e-01
1.48800001e-01 -1.32200003e-01 2.49499991e-03 5.39979994e-01
-3.12929988e-01 -2.12009996e-02 -2.96559989e-01 1.51110003e-02
-3.76980007e-02 -4.07290012e-01 3.36799994e-02 -2.90179998e-01
-5.64790010e-01 6.47960007e-01 3.96770000e-01 -1.91139996e-01
6.98249996e-01 1.61090004e+00 8.42899978e-01 -1.45980000e-01
-1.13940001e-01 -5.39680004e-01 -7.22540021e-01 -1.47310004e-01
-6.78520024e-01 1.93159997e-01 4.17819992e-03 -3.96059990e-01
8.99320021e-02 3.30909997e-01 -4.09649983e-02 4.59470004e-01
-3.07020009e-01 -2.61469990e-01 5.39489985e-01 6.60130024e-01
-5.68049997e-02 -3.70750010e-01 -1.46880001e-03 -1.48770005e-01
2.13000000e-01 -1.36710003e-01 5.12350023e-01 -5.27249992e-01
2.73449998e-02 3.00159991e-01 4.59829986e-01 3.65709990e-01
2.23989993e-01 1.47589996e-01 -9.97050032e-02 2.26520002e-01
-5.95120013e-01 -4.34839994e-01 -5.26629984e-02 -3.05240005e-01
-1.91280007e-01 -4.08630013e-01 -2.83719987e-01 6.89159989e-01
-7.18529999e-01 2.89909989e-01 -1.31809995e-01 -8.75229985e-02
2.70280004e-01 3.31449993e-02 1.93159997e-01 -2.50629991e-01
1.13150001e-01 4.88560013e-02 -4.41439986e-01 -8.88149977e-01
-3.45840007e-01 -5.47370017e-01 2.13919997e-01 1.46469995e-01
-3.76450002e-01 5.63120008e-01 4.44130003e-01 -6.05080016e-02
5.85870028e-01 4.50850010e-01 -1.27309993e-01 6.86409995e-02
3.78729999e-01 -7.15900004e-01 2.72450000e-01 2.16839999e-01
-5.09050012e-01 2.82240003e-01 5.28559983e-01 2.97140002e-01
-3.33429992e-01 -1.62200004e-01 -3.97210002e-01 3.37130010e-01
6.61469996e-01 -4.09590006e-01 -1.97520003e-01 -8.77860010e-01
-4.36379999e-01 1.79399997e-01 3.39769991e-03 9.73770022e-01
7.32789993e-01 -3.44499983e-02 -4.76709992e-01 -4.99819994e-01
4.97110009e-01 -7.77419984e-01 -3.60000014e-01 -1.58899993e-01
1.19869998e-02 -1.39750004e-01 -5.91250002e-01 -1.58280000e-01
9.19710025e-02 4.62440014e-01 -1.17980000e-02 8.29930007e-01
7.96280026e-01 -5.64369977e-01 -1.02080004e-02 -3.86090010e-01
-6.74889982e-01 -3.42109986e-02 1.99829996e-01 2.01010004e-01
7.43009984e-01 -3.58200014e-01 1.07579999e-01 -7.77289987e-01
3.24770004e-01 -7.43889987e-01 -7.90560007e-01 4.95799989e-01
3.70090008e-01 -1.55349998e-02 1.20260000e+00 -6.90900028e-01
1.49619997e+00 4.31109995e-01 2.64299989e-01 -5.99219978e-01
-4.51359987e-01 7.66879976e-01 -1.43020004e-02 1.43340006e-01
1.19379997e-01 -5.56930006e-01 -9.95339990e-01 8.20089996e-01
-3.02410007e-01 9.95709971e-02 -4.92139995e-01 2.25950003e-01
5.18890023e-01 6.51669979e-01 -7.13440031e-02 3.06109991e-03
-2.95789987e-01 -4.40490007e-01 -1.61009997e-01 2.52220005e-01
-3.50809991e-01 -7.96020031e-02 4.33560014e-01 -1.43889993e-01
-1.26969993e-01 1.01450002e+00 -6.24639988e-02 -5.60909986e-01
-6.52670026e-01 1.86299995e-01 -3.76190007e-01 -2.40750000e-01
2.50979990e-01 -2.06090003e-01 -8.57209980e-01 -5.84480017e-02
-1.23539999e-01 -7.35830009e-01 -4.83509988e-01 2.75110006e-01
-1.10220000e-01 3.17330003e-01 1.87999994e-01 1.05719995e+00
5.85460007e-01 3.13740000e-02 2.11569995e-01 1.13799997e-01
4.65829998e-01 -3.05020005e-01 -4.51409996e-01 2.87349999e-01]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[-0.34636 -0.88984001 -0.50321001 -0.43516001 0.54518998 0.17437001
-0.093541 0.16141 -0.46575999 -0.22 -0.31415001 -0.13484
-0.37617999 -0.67635 0.78820002 -0.33384001 -0.42414001 0.32367
0.50670999 0.21540999 0.43296 1.49049997 0.31795001 -0.15196
0.2579 -0.35666001 -0.63880002 -0.086453 -0.94755 0.19203
0.31161001 -0.74491 -0.59210998 0.4332 -0.064934 -0.48862001
0.35600999 -0.44780999 -0.015773 0.18203001 0.051751 -0.2854
-0.14727999 0.1513 -0.33048001 0.27135 1.16659999 -0.36662
0.090829 0.87301999 -0.13505 0.21204001 0.57270002 0.54259002
-0.50335002 0.16767 -0.82530999 -0.45962 -0.42642 -0.2164
0.088689 -0.15061 -0.16785 -0.31794 -0.69608998 0.40715
-0.29190999 -0.042072 0.90051001 0.35947999 0.030644 -0.028853
0.086821 0.74741 -0.52008998 0.20655 0.44053999 -0.11865
-0.15449999 -0.22457001 -0.15453 0.16101 -0.30825001 -0.28479999
-0.50823998 0.48732999 -0.012029 0.034592 0.48304 -0.56752002
-0.057299 0.22070999 -0.34200001 -0.060634 0.95499998 -0.60952997
0.59577 -0.11553 -0.67475998 0.52658999 0.82163 0.35126001
0.15521 0.12135 0.38191 0.24228001 -0.51485997 1.14810002
0.07281 0.23024 -0.68901998 -0.17606001 -0.24308001 -0.13686
-0.13467 0.059625 -0.68668997 0.15907 0.11983 -0.024954
0.34898001 0.15456 0.047524 0.23616999 0.54784 -1.01380002
0.10351 0.26865 -0.064867 0.23893 0.026141 0.081648
0.74479997 -0.67208999 0.23351 -0.55070001 -0.14296 -0.30303001
-0.40233999 0.012984 0.86865002 -0.14025 1.13900006 -0.093339
1.56060004 0.41804001 0.54067999 -0.43340999 -0.090589 0.56682003
-0.21291 0.45693001 -0.64519 -0.05866 0.21477 0.45563
-0.15218 0.36307001 -0.25441 -0.72013998 0.52191001 0.55509001
-0.073841 0.44994 -0.11501 0.1911 0.077304 0.18629
0.60244 0.028085 0.17228 -0.24455 0.04822 0.51318002
-0.06824 0.35515001 -0.80987 -0.42732999 -0.72728997 0.47817001
0.87611997 -0.18855 0.30390999 -0.14161 0.26699001 -0.81572002
-0.67589998 -0.34687999 0.53188998 0.75443 -0.083874 0.77434999
0.081108 -0.29840001 -0.24409001 -0.14574 -0.1186 0.085964
0.48076999 -0.13097 ]
[ 1.07439995 -0.49325001 -0.23126 -0.197 -0.087485 -0.16806
-0.11092 0.42857999 -0.16893999 0.0094633 -0.50453001 -0.40006
0.31169 0.50295001 -0.48537001 -0.20689 -0.62162 0.38407999
0.22182 0.051087 -0.018028 1.37919998 0.3163 -0.17425001
0.11344 -0.42723 -0.28510001 -0.1246 -0.2024 0.18217
-0.37834001 -0.22421999 0.38877001 0.20065001 -0.29708999 0.77051002
0.13087 -0.25084001 0.54755002 0.38086 0.28174999 -0.15691
-0.71987998 0.24118 0.073913 -0.46965 1.02180004 0.049863
0.036841 0.54706001 -0.15903001 0.53780001 -0.10335 0.51644999
-0.25512001 -0.18553001 -0.51804 -0.24337 0.57081997 -0.39017001
-0.17132001 -0.14939 -0.1724 0.91408002 -0.45778 0.40143001
0.075224 -0.4104 -0.1714 -0.63552999 0.60185999 -0.3193
-0.46136999 0.030652 0.32890001 -0.2472 -0.49831 -0.90982997
-0.057251 0.20213 -0.51845998 0.46320999 0.032707 0.29872999
1.11189997 -0.35784999 0.34929001 -0.51739001 0.25235 -1.0309
0.21052 0.06349 -0.10589 0.43222001 0.20389 0.065589
-0.62914002 0.1096 -0.86363 0.44760999 0.43114999 0.041376
-0.42429999 -0.080897 0.093115 0.22603001 0.31176999 0.83004999
-0.25659001 0.013861 0.38326001 -0.52025998 0.30410001 -0.52507001
-0.78566003 -0.046498 0.41154999 -0.21447 -0.24202999 -0.24732
1.01129997 0.067517 0.18331 -0.17636 0.49777001 -0.21067999
0.0037579 0.22881 -0.15993001 -0.13421001 -0.27379999 -0.20734
0.13407999 -0.57762003 -0.66526997 -0.42083001 0.65882999 -0.53825998
-0.50585997 0.51735002 0.25468999 -0.83724999 0.83806998 -0.42219999
1.0776 0.065962 0.48954999 -0.78614002 -0.19338 0.097524
0.27215999 0.037038 0.61778998 -0.29506999 -0.97285002 0.53106999
-0.32765001 -0.045966 -0.75436997 0.024904 0.64872003 0.023095
0.32062 0.35837999 -0.091125 -0.10866 0.33048001 -0.1162
-0.40981999 0.43928999 -0.16706 0.26047 0.090957 0.92714
0.099946 -0.29513001 -0.35341001 0.33693001 -0.42203999 -0.065625
0.54738998 -0.41751 -0.86232001 -0.65891999 -0.41549 0.067035
-0.41558 0.15092 0.17556 0.94068003 0.22527 0.65908003
0.15809 0.061199 0.63612998 -0.17089 -0.017591 -0.054003
-0.69072002 0.65179998]
[ 1.03529997e-01 7.20499977e-02 -2.93029994e-02 -4.46799994e-01
-8.61259997e-02 7.40030035e-02 -4.65499997e-01 -6.18570000e-02
-5.03650010e-01 1.95480004e-01 -1.03349999e-01 7.75929987e-01
-1.72040001e-01 -4.53520000e-01 2.63040006e-01 1.64340004e-01
2.80279994e-01 2.89330006e-01 3.10360014e-01 1.27750002e-02
6.79520011e-01 2.62770009e+00 3.89640003e-01 -4.49900001e-01
2.17969999e-01 9.16400030e-02 -1.67940006e-01 7.72420019e-02
2.91269988e-01 1.20530002e-01 -5.49659990e-02 9.16600004e-02
1.31300002e-01 -9.33270007e-02 -3.53009999e-01 -5.03880024e-01
-7.29799986e-01 -3.61380011e-01 -2.99659997e-01 2.07839999e-02
-7.03599975e-02 -6.72269985e-02 -3.62650007e-02 -1.46009997e-02
-5.98580018e-02 -1.63020000e-01 3.00660014e-01 -9.28120017e-02
-6.69799969e-02 1.43830001e-01 1.03950001e-01 -1.93039998e-02
-4.07020003e-01 8.86749983e-01 2.67349988e-01 -1.23829998e-01
2.73739994e-02 3.44639987e-01 6.04049981e-01 2.80640006e-01
1.41320005e-01 2.46429995e-01 -4.48760003e-01 5.42909980e-01
1.96759999e-01 -4.94709998e-01 2.68779993e-02 2.69100010e-01
2.53410012e-01 9.88679975e-02 1.77919999e-01 -3.22290003e-01
-1.05930001e-01 1.89520001e-01 -2.57629991e-01 2.43619993e-01
-2.45560005e-01 -1.36539996e-01 -1.08170003e-01 -4.21220005e-01
-1.61640003e-01 -3.41199994e-01 -1.47389993e-01 -2.16409996e-01
-5.62399998e-02 7.15879977e-01 6.40570000e-02 -3.07790011e-01
6.57369971e-01 -6.36910021e-01 2.64039993e-01 2.15059996e-01
1.83850005e-01 3.32610011e-01 -6.30220026e-02 -2.22450003e-01
6.31980002e-02 -4.47950006e-01 -2.52279997e-01 1.97699994e-01
-3.55479985e-01 1.19139999e-01 -2.99149990e-01 1.29390001e-01
-4.30770010e-01 -1.37700006e-01 2.38429993e-01 1.45529997e+00
-1.13710001e-01 -3.79790008e-01 -2.83129990e-01 -4.64819998e-01
-1.92760006e-01 1.98490005e-02 2.70090014e-01 -1.70849994e-01
-5.20099998e-02 -2.03219995e-01 -3.27439994e-01 -5.07570028e-01
2.98289992e-02 1.80350006e-01 3.05330008e-01 2.40700006e-01
4.66120005e-01 -7.12530017e-01 5.96719980e-01 2.13310003e-01
3.48639995e-01 3.80190015e-01 1.38260007e-01 -7.33380020e-02
2.26349995e-01 -4.55599993e-01 8.19810014e-03 3.47319990e-01
-1.36079997e-01 -6.59820020e-01 -2.79430002e-01 4.81799990e-02
-3.97129990e-02 2.86449999e-01 1.41259998e-01 -3.94299999e-02
1.44519997e+00 4.59950000e-01 7.92850032e-02 -3.55580002e-01
3.09360009e-02 -2.50809994e-02 5.64870015e-02 9.00759995e-02
5.20309992e-02 -3.99529994e-01 3.40330005e-01 -4.17400002e-01
-3.82189989e-01 2.22049996e-01 1.09520003e-01 -1.64539993e-01
1.20609999e-01 1.60919994e-01 -3.27459991e-01 2.45800003e-01
-2.28320006e-02 2.74109989e-01 -2.10689995e-02 3.91460001e-01
-5.61020017e-01 6.95510030e-01 -3.55260000e-02 4.71640006e-02
6.71909988e-01 4.81240004e-01 -2.78840009e-02 5.05490005e-01
-5.41859984e-01 -3.52369994e-01 -3.12009990e-01 -1.76020002e-03
9.59439993e-01 -5.03639996e-01 4.39889997e-01 4.71810013e-01
4.25799996e-01 -5.92209995e-01 -3.96219999e-01 1.89040005e-02
3.33819985e-02 -2.90149987e-01 -1.22079998e-01 2.76309997e-02
-2.66250014e-01 -1.97340008e-02 2.31020004e-01 8.76149982e-02
-7.69149978e-03 1.90050006e-02 -4.42119986e-01 -6.81999996e-02]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[ 0.041052 -0.54705 -0.72193998 -0.31235999 -0.43849 0.10691
-0.50641 -0.45401001 -0.28623 0.018973 0.020495 0.42860001
-0.0057162 -0.21272001 0.71864998 -0.091906 -0.55365002 0.39133
0.15351 -0.27454001 0.56528002 3.04830003 0.30467001 -0.37893
0.37865999 0.13925999 -0.11482 0.48212999 -0.30522999 0.43125001
-0.09667 0.069156 0.31426001 0.26350999 -0.31189999 -0.39881
-0.55656999 -0.35934001 -0.25402001 0.072061 -0.12966999 -0.11247
-0.041192 -0.042619 -0.07848 0.31992 0.41635999 0.26131001
-0.18175 -0.1279 0.21332 -0.41973001 -0.50444001 0.37705001
0.83955002 -0.34571001 -0.43000999 -0.18653999 -0.061082 -0.087612
0.092833 0.52604997 -0.57611001 -0.19328 0.20576 0.24607
0.1631 -0.18138 0.032592 0.19169 0.73565 -0.25718999
0.30072999 0.56699002 -0.21544001 0.18933 -0.12287 -0.65759999
0.021702 0.1041 0.098952 -0.43171999 -0.27517 -0.15448
0.31301001 -0.032041 -0.090526 0.14489999 0.68151999 -0.88242
0.30816999 -0.62702 0.12274 0.014773 -0.16887 0.56159002
0.022004 0.52085 -0.22685 0.09633 0.26956999 0.30489001
0.018463 0.31009001 0.04198 0.32381999 0.13153 0.89722002
0.15475 -0.38806999 -0.52993 -0.35383999 -0.0913 0.57230002
0.48067001 0.24438 0.074138 -0.019957 -0.35030001 -0.034695
-0.12045 0.39998999 -0.37015 -0.53040999 0.10655 -0.44973001
0.43105 -0.44937 0.48675999 0.43836999 0.043421 0.52675003
0.61176002 0.26704001 0.59239 0.23650999 0.12841 -0.10665
-0.46715 -0.039081 -0.24921 0.030486 0.092933 -0.04841
1.83580005 0.077535 -0.11494 -0.13668001 -0.23983 0.31948
0.19205999 0.38894001 0.34755 -0.038804 0.19541 -0.37099999
-0.027576 -0.24127001 -0.16868 0.032815 0.08139 0.054121
-0.42785999 0.26447001 0.054847 -0.21765999 0.015181 0.57656002
0.24033 0.62076002 -0.019055 -0.31509 0.76424998 0.35168999
-0.28512001 0.15175 0.11238 -0.60829997 0.35087001 0.19140001
0.51753998 0.20893 -0.63814002 0.19403 0.24493 0.46606001
-0.32235 0.37286001 -0.19508 0.13237999 -0.35420999 0.22849
0.36032 -0.0050241 -0.051955 -0.37755999 -0.087065 0.3592
0.11564 0.44372001]
[ 4.71520007e-01 -5.87790012e-01 -6.76150024e-01 -4.67000008e-01
1.11709997e-01 3.26370001e-01 -4.65070009e-01 -8.05180013e-01
-1.65340006e-01 -1.13849998e-01 -1.36849999e-01 2.56980002e-01
3.36279988e-01 1.90149993e-01 1.32440001e-01 2.37419993e-01
-1.46239996e+00 8.59059989e-01 5.53650022e-01 1.94330007e-01
2.73930013e-01 1.05669999e+00 6.24970019e-01 -4.30469990e-01
7.14770019e-01 -4.73030001e-01 -8.97960007e-01 2.56910007e-02
-6.42499983e-01 2.15990007e-01 -1.22769997e-01 -5.36949992e-01
5.91489971e-01 6.28649965e-02 1.51260002e-02 -6.57150000e-02
1.61709994e-01 -8.86740014e-02 -7.09370002e-02 6.12349987e-01
1.38689995e-01 -3.67980003e-01 -9.46219981e-01 1.30669996e-01
-2.82139987e-01 -3.02709997e-01 4.05889988e-01 -2.11899996e-01
1.74940005e-01 2.38450006e-01 3.41769993e-01 4.50269997e-01
-7.82140017e-01 1.64210007e-01 7.19319999e-01 -6.80140018e-01
-4.93660003e-01 3.67380008e-02 2.62410015e-01 -8.48299980e-01
-6.59759998e-01 4.04370010e-01 -2.61209998e-02 5.83829999e-01
-3.28000009e-01 6.39530003e-01 1.20350003e-01 7.21519988e-04
8.28130007e-01 -3.83879989e-01 5.35929978e-01 -4.59630013e-01
-5.12839973e-01 1.74339995e-01 2.06220001e-01 -8.01329970e-01
-4.74339992e-01 -4.32810009e-01 -6.11400008e-01 1.71409994e-01
5.54369986e-01 6.11240007e-02 6.43959999e-01 -5.23599982e-01
1.35130000e+00 -1.40279993e-01 3.67210001e-01 -3.68629992e-01
7.95690000e-01 -1.01139998e+00 -1.47060007e-01 4.48889993e-02
4.06240001e-02 5.33150017e-01 3.40680003e-01 2.50710011e-01
-1.26489997e-01 -1.15050003e-01 -1.48660004e-01 7.65860021e-01
9.80419964e-02 6.28759980e-01 -4.09599990e-01 -3.33020017e-02
-3.77389997e-01 -2.54250001e-02 -8.92150030e-02 1.63540006e+00
5.04270017e-01 -3.86139989e-01 -1.25259995e-01 1.65910006e-01
-2.19990000e-01 -6.84350014e-01 -2.99389988e-01 -1.25550002e-01
3.63970011e-01 3.89310002e-01 -9.63360012e-01 -7.42670000e-02
3.48879993e-01 2.36190006e-01 -8.27549994e-01 3.19779992e-01
-3.00359994e-01 -4.01740015e-01 7.04670012e-01 -3.32989991e-01
-1.26369998e-01 -3.72830003e-01 -9.09730017e-01 -1.33279994e-01
-5.32779992e-02 -3.47909987e-01 2.48980001e-01 -4.34430003e-01
-2.42050007e-01 -6.21890008e-01 -1.27769995e+00 -9.66310024e-01
4.20859993e-01 -4.12339985e-01 1.20589995e+00 -2.55379993e-02
1.01170003e+00 -1.11219997e-03 5.92599988e-01 -5.58459997e-01
-3.69489998e-01 5.58549985e-02 -6.60969973e-01 3.91559988e-01
3.09260011e-01 -6.28539994e-02 -1.14900005e+00 4.69440013e-01
-5.05439997e-01 -1.75500005e-01 -3.80259991e-01 -1.97349995e-01
7.31180012e-01 -1.77149996e-01 4.18940008e-01 4.23940003e-01
-5.16149998e-02 -1.20180003e-01 3.96189988e-01 -2.36780003e-01
1.19029999e-01 2.93020010e-01 -1.81150004e-01 4.59089994e-01
1.93489999e-01 4.75919992e-01 -3.67170006e-01 -8.76540027e-04
-6.15379997e-02 -4.12099995e-02 -1.95240006e-01 -6.47260016e-03
4.30709988e-01 6.84989989e-02 -4.27579999e-01 -3.15990001e-01
-2.59119987e-01 -6.58209980e-01 -1.92790002e-01 5.57280004e-01
-2.24089995e-01 2.21340001e-01 4.68760014e-01 6.37290001e-01
6.58949971e-01 8.79120007e-02 -3.06369990e-01 -3.22079986e-01
6.47899985e-01 1.75490007e-01 -5.78580022e-01 8.94800007e-01]
===================================
===== For the word2vec
===================================
1. Download and unzip the pre-trained model of GoogleNews-vectors-negative300.bin.gz.
2. Install the gensim tools:
sudo pip install --upgrade gensim
3. Code for vector extraction from given sentence.
import gensim
print("==>> loading the pre-trained word2vec model: GoogleNews-vectors-negative300.bin")
dictFileName = './GoogleNews-vectors-negative300.bin'
wv = gensim.models.KeyedVectors.load_word2vec_format(dictFileName, binary=True)
The Output is:
==>> loading the pre-trained word2vec model: GoogleNews-vectors-negative300.bin
INFO:gensim.models.utils_any2vec:loading projection weights from ./GoogleNews-vectors-negative300.bin
INFO:gensim.models.utils_any2vec:loaded (3000000, 300) matrix from ./GoogleNews-vectors-negative300.bin
INFO:root:Data statistic
INFO:root:train_labels:19337
INFO:root:test_labels:20632
INFO:root:train_sentences:19337
INFO:root:dev_sentences:2000
INFO:root:test_sentences:20632
INFO:root:dev_labels:2000
embed_size:300
vocab_size:3000000
vocab_path = ''data/bilstm.vocab''
index_to_word = [key for key in wv.vocab]
word_to_index = {}
for index, word in enumerate(index_to_word):
word_to_index[word] = index
with open(vocab_path, "w") as f:
f.write(json.dumps(word_to_index))
Let's take a deep understanding on the bidirectional-LSTM-for-text-classification-master
class BiLSTM(nn.Module):
def __init__(self, embedding_matrix, hidden_size=150, num_layer=2, embedding_freeze=False):
super(BiLSTM,self).__init__() # embedding layer
vocab_size = embedding_matrix.shape[0]
embed_size = embedding_matrix.shape[1]
self.hidden_size = hidden_size
self.num_layer = num_layer
self.embed = nn.Embedding(vocab_size, embed_size)
self.embed.weight = nn.Parameter(torch.from_numpy(embedding_matrix).type(torch.FloatTensor), requires_grad=not embedding_freeze)
self.embed_dropout = nn.Dropout(p=0.3)
self.custom_params = []
if embedding_freeze == False:
self.custom_params.append(self.embed.weight) # The first LSTM layer
self.lstm1 = nn.LSTM(embed_size, self.hidden_size, num_layer, dropout=0.3, bidirectional=True)
for param in self.lstm1.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param) self.lstm1_dropout = nn.Dropout(p=0.3) # The second LSTM layer
self.lstm2 = nn.LSTM(2*self.hidden_size, self.hidden_size, num_layer, dropout=0.3, bidirectional=True)
for param in self.lstm2.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param)
self.lstm2_dropout = nn.Dropout(p=0.3) # Attention
self.attention = nn.Linear(2*self.hidden_size,1)
self.attention_dropout = nn.Dropout(p=0.5) # Fully-connected layer
self.fc = weight_norm(nn.Linear(2*self.hidden_size,3))
for param in self.fc.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param) self.hidden1=self.init_hidden()
self.hidden2=self.init_hidden() def init_hidden(self, batch_size=3):
if torch.cuda.is_available():
return (Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)).cuda(),
Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)).cuda())
else:
return (Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)),
Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size))) def forward(self, sentences): print("==>> sentences: ", sentences) # get embedding vectors of input
padded_sentences, lengths = torch.nn.utils.rnn.pad_packed_sequence(sentences, padding_value=int(0), batch_first=True)
print("==>> padded_sentences: ", padded_sentences) embeds = self.embed(padded_sentences)
print("==>> embeds: ", embeds) # pdb.set_trace() noise = Variable(torch.zeros(embeds.shape).cuda())
noise.data.normal_(std=0.3)
embeds += noise
embeds = self.embed_dropout(embeds)
# add noise packed_embeds = torch.nn.utils.rnn.pack_padded_sequence(embeds, lengths, batch_first=True) print("==>> packed_embeds: ", packed_embeds) # First LSTM layer
# self.hidden = num_layers*num_directions batch_size hidden_size
packed_out_lstm1, self.hidden1 = self.lstm1(packed_embeds, self.hidden1)
padded_out_lstm1, lengths = torch.nn.utils.rnn.pad_packed_sequence(packed_out_lstm1, padding_value=int(0))
padded_out_lstm1 = self.lstm1_dropout(padded_out_lstm1)
packed_out_lstm1 = torch.nn.utils.rnn.pack_padded_sequence(padded_out_lstm1, lengths) pdb.set_trace() # Second LSTM layer
packed_out_lstm2, self.hidden2 = self.lstm2(packed_out_lstm1, self.hidden2)
padded_out_lstm2, lengths = torch.nn.utils.rnn.pad_packed_sequence(packed_out_lstm2, padding_value=int(0), batch_first=True)
padded_out_lstm2 = self.lstm2_dropout(padded_out_lstm2) # attention
unnormalize_weight = F.tanh(torch.squeeze(self.attention(padded_out_lstm2), 2))
unnormalize_weight = F.softmax(unnormalize_weight, dim=1)
unnormalize_weight = torch.nn.utils.rnn.pack_padded_sequence(unnormalize_weight, lengths, batch_first=True)
unnormalize_weight, lengths = torch.nn.utils.rnn.pad_packed_sequence(unnormalize_weight, padding_value=0.0, batch_first=True)
logging.debug("unnormalize_weight size: %s" % (str(unnormalize_weight.size())))
normalize_weight = torch.nn.functional.normalize(unnormalize_weight, p=1, dim=1)
normalize_weight = normalize_weight.view(normalize_weight.size(0), 1, -1)
weighted_sum = torch.squeeze(normalize_weight.bmm(padded_out_lstm2), 1) # fully connected layer
output = self.fc(self.attention_dropout(weighted_sum))
return output
==>> Some Testing:
(a). len(wv.vocab) = 300,0000
(b). wv.vocab is what ? Something like this:
{ ... , u'fivemonth': <gensim.models.keyedvectors.Vocab object at 0x7f90945bd810>,
u'retractable_roofs_Indians': <gensim.models.keyedvectors.Vocab object at 0x7f90785f5690>,
u'Dac_Lac_province': <gensim.models.keyedvectors.Vocab object at 0x7f908d8eda10>,
u'Kenneth_Klinge': <gensim.models.keyedvectors.Vocab object at 0x7f9081563410>}
(c). index_to_word: count the words from the pre-trained model.
{ ... , u"Lina'la_Sin_Casino", u'fivemonth', u'retractable_roofs_Indians', u'Dac_Lac_province', u'Kenneth_Klinge'}
(d). word_to_index: give each word a index as following
{ ... , u'fivemonth': 2999996, u'Pidduck': 2999978, u'Dac_Lac_province': 2999998, u'Kenneth_Klinge': 2999999 }
(e). dataset is:
[ ... , [1837614, 1569052, 1837614, 1288695, 2221039, 2323218, 1837614, 1837614, 2029395, 1612781, 311032, 1524921, 1837614, 2973515, 2033866, 882731, 2462275, 2809106, 1479961, 826019, 73590, 953550, 1837614],
[1524921, 1113778, 1837614, 318169, 1837614, 1954969, 196613, 943118, 1837614, 2687790, 291413, 2774825, 2366038, 296869, 1468080, 856987, 1802099, 724308, 1207907, 2264894, 2206446, 812434],
[564298, 477983, 1837614, 1449153, 1837614, 211925, 2206446, 481834, 488597, 280760, 2072822, 1344872, 1791678, 2458776, 2965810, 2168205, 387112, 2656471, 1391, 1837614, 1801696, 2093846, 1210651],
[2493381, 133883, 2441902, 1014220, 1837614, 2597880, 1756105, 2651537, 1391, 2641114, 2517536, 1109601, 122269, 1782479, 2965835, 488597, 1767716, 753333, 564298, 2380935, 228060, 1837614, 371618],
[1837614, 1344872, 2458776, 2965810, 1837614, 2015408, 1837614, 528014, 1991322, 1837614, 908982, 1484130, 2349526, 988689, 753336, 1837614, 364492, 2183116, 826019, 73590, 953550, 1837614],
[1837614, 2673785, 1990947, 1219831, 2635341, 1247040, 1837614, 799543, 1990947, 1219831, 2722301, 1837614, 1427513, 969099, 2157673, 1430111, 1837614]]
(f). the variable during the training process:
# ('==>> sentences: ', PackedSequence(data=tensor(
# [ 4.0230e+05, 1.8376e+06, 2.0185e+06, 1.8376e+06, 2.8157e+06,
# 1.8376e+06, 1.8376e+06, 1.0394e+06, 1.8376e+06, 2.9841e+06,
# 4.4713e+05, 1.1352e+06, 2.3532e+06, 1.8376e+06, 1.8376e+06,
# 1.8376e+06, 1.9550e+06, 3.8429e+04, 6.2537e+05, 2.3764e+05,
# 1.8376e+06, 1.5428e+06, 1.4214e+06], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])))
# ('==>> padded_sentences: ', tensor(
# [[ 4.0230e+05, 1.8376e+06, 2.0185e+06, 1.8376e+06, 2.8157e+06,
# 1.8376e+06, 1.8376e+06, 1.0394e+06, 1.8376e+06, 2.9841e+06,
# 4.4713e+05, 1.1352e+06, 2.3532e+06, 1.8376e+06, 1.8376e+06,
# 1.8376e+06, 1.9550e+06, 3.8429e+04, 6.2537e+05, 2.3764e+05,
# 1.8376e+06, 1.5428e+06, 1.4214e+06]], device='cuda:0'))
# length: 23
# ('==>> embeds: ', tensor([[
# [-0.0684, 0.1826, -0.1777, ..., 0.1904, -0.1021, 0.1729],
# [ 0.0801, 0.1050, 0.0498, ..., 0.0037, 0.0476, -0.0688],
# [-0.1982, -0.0693, 0.1230, ..., -0.1357, -0.0306, 0.1104],
# ...,
# [ 0.0801, 0.1050, 0.0498, ..., 0.0037, 0.0476, -0.0688],
# [-0.0518, -0.0299, 0.0415, ..., 0.0776, -0.1660, 0.1602],
# [-0.0532, -0.0004, 0.0337, ..., -0.2373, -0.1709, 0.0233]]], device='cuda:0'))
# ('==>> packed_embeds: ', PackedSequence(data=tensor([
# [-0.3647, 0.2966, -0.2359, ..., -0.0000, 0.2657, -0.4302],
# [ 1.1699, 0.0000, 0.3312, ..., 0.5714, 0.1930, -0.2267],
# [-0.0627, -1.0548, 0.4966, ..., -0.5135, -0.0150, -0.0000],
# ...,
# [-0.6065, -0.7562, 0.3320, ..., -0.5854, -0.2089, -0.5737],
# [-0.0000, 0.4390, 0.0000, ..., 0.6891, 0.0250, -0.0000],
# [ 0.6909, -0.0000, -0.0000, ..., 0.1867, 0.0594, -0.2385]], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])))
# packed_out_lstm1
# PackedSequence(data=tensor([
# [-0.0000, -0.0000, 0.1574, ..., 0.1864, -0.0901, -0.3205],
# [-0.0000, -0.3490, 0.1774, ..., 0.1677, -0.0000, -0.3688],
# [-0.3055, -0.0000, 0.2240, ..., 0.0000, -0.0927, -0.0000],
# ...,
# [-0.3188, -0.4134, 0.1339, ..., 0.3161, -0.0000, -0.3846],
# [-0.3355, -0.4365, 0.1575, ..., 0.2775, -0.0886, -0.4015],
# [-0.0000, -0.0000, 0.2452, ..., 0.1763, -0.0000, -0.2748]], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]))
## self.hidden1
# self.hidden1
# (tensor([
# [[-0.0550, -0.4266, 0.0498, 0.0750, 0.3122, 0.0499, -0.2899,
# -0.0980, -0.1776, 0.4376, 0.1555, 0.7167, -0.0162, 0.0081,
# -0.3512, 0.0250, 0.0948, -0.3596, 0.0670, -0.6623, 0.0026,
# -0.0262, 0.1934, 0.3206, -0.1941, 0.3229, -0.6488, -0.3858,
# 0.3169, -0.0216, -0.1136, 0.1411, -0.0296, -0.2860, -0.1277,
# -0.0154, 0.2620, -0.3553, 0.0637, 0.8245, 0.6886, -0.0048,
# -0.1717, 0.2495, -0.8636, -0.0217, -0.2365, 0.1324, 0.1790,
# -0.1515, 0.3530, -0.1644, 0.0073, 0.6709, 0.1577, -0.0190,
# 0.0384, 0.4871, -0.7375, -0.1804, 0.3034, -0.3516, -0.2870,
# 0.6387, 0.0414, 0.6983, -0.3211, 0.0449, 0.2127, -0.0421,
# -0.3454, -0.8145, -0.3629, -0.0828, -0.1558, 0.4048, -0.4971,
# -0.7495, 0.0622, -0.3318, 0.3913, -0.0322, -0.0678, 0.0307,
# -0.0153, -0.1535, -0.2719, 0.0128, -0.1521, -0.2924, 0.7109,
# -0.8551, 0.0330, 0.0482, 0.2410, -0.0655, -0.2496, 0.1816,
# 0.4963, -0.7593, -0.0022, -0.1122, 0.6857, -0.5693, 0.5805,
# 0.7660, 0.4430, 0.0243, -0.0313, 0.0780, -0.2419, 0.0745,
# -0.0119, 0.5761, -0.0285, -0.1085, -0.1783, -0.0706, 0.1243,
# 0.6333, 0.0296, 0.5557, 0.2717, -0.0071, 0.0503, -0.0405,
# -0.4542, 0.8905, 0.4492, 0.8809, 0.7021, 0.8616, 0.2825,
# -0.2114, 0.3026, -0.1384, 0.1252, 0.4989, 0.2236, -0.5374,
# -0.1352, -0.0561, 0.0378, -0.5291, -0.1004, -0.3723, 0.0836,
# 0.3500, 0.0542, 0.2013]], # [[-0.0041, -0.0040, -0.1428, 0.2783, -0.1378, -0.4242, 0.1000,
# 0.1641, -0.0175, -0.0896, 0.3241, 0.3513, -0.4675, -0.1250,
# 0.0546, 0.2400, -0.0997, -0.5614, 0.2026, -0.1505, 0.0833,
# -0.3128, -0.4646, -0.0778, 0.2204, 0.5597, -0.7004, 0.0419,
# -0.3699, -0.5748, 0.7741, -0.7220, 0.0494, 0.3430, 0.1389,
# 0.0178, 0.0136, 0.0273, 0.1559, 0.4333, 0.2411, 0.0804,
# -0.0202, 0.6204, -0.4104, 0.5382, 0.1804, 0.0179, 0.1118,
# 0.3084, 0.1894, 0.0092, -0.2182, 0.0022, 0.4377, -0.0575,
# 0.0906, -0.0531, 0.0936, -0.1203, -0.0836, -0.1718, 0.2059,
# 0.7192, 0.0833, 0.3712, 0.2354, -0.1141, -0.0993, -0.0433,
# -0.1474, -0.0078, 0.0834, 0.0864, 0.9305, -0.3387, 0.0818,
# 0.3775, 0.1609, -0.0246, 0.2563, -0.1253, -0.0897, 0.0322,
# -0.0848, 0.0673, -0.1051, 0.0205, 0.0183, -0.0007, 0.1229,
# -0.3388, -0.0948, 0.0335, 0.0450, -0.2747, 0.2763, -0.2691,
# 0.6240, -0.0018, 0.2048, 0.3943, 0.2015, -0.5962, -0.0069,
# -0.3460, -0.7910, 0.3002, -0.4653, -0.0611, 0.6912, -0.8154,
# -0.0443, -0.0189, -0.1265, -0.1202, 0.0013, -0.5983, 0.0879,
# 0.0752, 0.8593, 0.0357, 0.0953, -0.0525, 0.2069, -0.6292,
# -0.0456, -0.7646, 0.0166, 0.0584, 0.0142, 0.0575, -0.1658,
# -0.4304, 0.3228, 0.4094, 0.0149, 0.1478, 0.1447, 0.4192,
# -0.0783, -0.0683, 0.0259, 0.0665, 0.6224, 0.3775, 0.0247,
# 0.1710, -0.3622, 0.0931]], # [[-0.2315, -0.2179, 0.1716, -0.6143, 0.4329, 0.2288, 0.1208,
# -0.3435, 0.6314, 0.0106, 0.1470, -0.0915, -0.3019, 0.1302,
# 0.2325, 0.1794, 0.0145, -0.6598, 0.0062, -0.0743, 0.0232,
# 0.9310, -0.8155, -0.0449, -0.5504, 0.5746, 0.3607, 0.4053,
# -0.1887, -0.5448, 0.1522, -0.0189, -0.4852, 0.6322, 0.0011,
# -0.1590, -0.0054, 0.2972, -0.0270, 0.0047, 0.2944, -0.0629,
# -0.1138, 0.1349, 0.5449, 0.8018, -0.0218, 0.0523, 0.3262,
# -0.0506, 0.2821, -0.0661, -0.7165, -0.3653, 0.3321, -0.0255,
# -0.0551, -0.1826, 0.6027, -0.1995, 0.0598, 0.0205, -0.1769,
# -0.0789, 0.4500, -0.1641, 0.5002, -0.0716, -0.3708, -0.0020,
# -0.5195, -0.0896, 0.1421, 0.1149, 0.3407, 0.2649, 0.0858,
# 0.2778, -0.3768, 0.6176, -0.2148, 0.5444, 0.3009, 0.4848,
# -0.1174, 0.0019, 0.6213, 0.2524, -0.0816, 0.4639, 0.4747,
# -0.7812, 0.2435, -0.0867, 0.1617, 0.2194, 0.0426, 0.1393,
# -0.0448, 0.0506, -0.5524, 0.0707, 0.2226, -0.0337, 0.7445,
# -0.4516, 0.1107, -0.2617, -0.1914, 0.7238, -0.2689, 0.0110,
# -0.3139, -0.0027, -0.5964, -0.9012, -0.4319, 0.0112, -0.0306,
# 0.4002, -0.1117, -0.1021, 0.1652, -0.2872, 0.3640, 0.2162,
# -0.3843, -0.0869, -0.1623, 0.0297, -0.0048, -0.0735, -0.0886,
# -0.4138, 0.2325, -0.4248, 0.3354, 0.0712, -0.4079, 0.0821,
# 0.1413, 0.2241, -0.1938, -0.0807, 0.3551, -0.0814, 0.1438,
# -0.6870, -0.3647, 0.0276]], # [[ 0.0258, 0.3281, -0.8145, -0.0476, -0.2886, -0.8013, 0.2135,
# 0.1541, 0.2069, 0.1345, -0.0171, -0.0228, -0.5237, 0.4917,
# 0.5187, 0.1402, 0.0928, -0.0373, 0.2698, 0.1259, 0.0021,
# -0.1624, -0.4100, 0.5377, -0.1013, -0.5658, -0.1015, 0.5609,
# 0.1661, 0.5731, 0.0012, -0.1766, 0.0743, -0.3630, 0.1082,
# 0.4643, 0.0175, -0.0260, 0.3810, -0.6425, -0.5515, 0.8800,
# -0.1158, -0.5741, 0.0463, 0.4033, 0.0803, 0.0403, 0.1159,
# 0.4471, 0.0294, 0.2899, 0.0248, -0.1772, 0.6600, -0.2252,
# -0.4896, -0.1285, -0.2377, -0.4179, -0.4056, -0.3224, -0.6855,
# -0.2703, 0.2971, 0.1259, -0.0456, -0.2495, 0.8141, 0.4453,
# 0.7480, -0.0578, 0.8023, -0.3586, -0.5229, 0.2299, 0.9668,
# -0.0717, 0.5355, -0.0743, 0.5246, 0.1604, -0.1464, -0.0757,
# 0.0414, -0.0861, 0.2245, 0.1247, 0.0676, -0.2053, 0.0113,
# 0.7875, -0.0308, 0.2025, 0.1289, -0.0020, -0.3099, 0.5317,
# -0.0117, 0.0928, -0.4100, -0.6184, 0.1171, 0.0216, -0.1266,
# 0.1640, 0.0821, -0.4097, -0.0691, 0.5805, 0.1692, -0.2021,
# 0.5971, 0.1172, -0.6535, -0.0579, 0.1177, 0.1123, -0.1943,
# 0.0488, -0.1305, -0.4859, -0.2758, -0.2972, -0.0605, -0.0029,
# -0.1508, 0.0375, -0.5942, -0.2139, -0.0335, -0.2320, -0.1152,
# -0.2054, -0.2643, -0.1770, 0.1245, 0.6334, -0.0363, 0.0264,
# -0.3348, -0.0434, -0.3794, -0.0913, 0.1293, -0.6537, 0.6490,
# 0.1305, -0.0631, -0.2243]]], device='cuda:0'),
# tensor([[[ -0.0775, -4.6346, 7.9077, 0.1164, 0.8626, 0.4240,
# -0.9286, -0.1612, -0.6049, 0.6771, 0.7443, 1.7457,
# -0.3930, 0.0112, -13.5393, 0.0317, 0.1236, -0.8475,
# 0.1212, -1.3623, 0.0117, -0.3297, 0.9009, 0.3840,
# -0.5885, 0.7411, -8.9613, -1.1402, 0.4511, -0.0753,
# -0.3107, 1.6518, -0.0870, -3.1360, -12.0200, -0.0464,
# 0.2756, -0.6695, 0.1604, 4.8299, 6.4623, -0.9555,
# -0.6904, 0.4469, -11.3343, -0.1669, -0.2747, 0.1590,
# 0.5829, -0.3345, 2.1731, -0.5636, 0.0207, 0.9874,
# 0.6291, -1.2261, 0.1946, 1.1287, -1.5759, -0.1875,
# 0.5550, -1.7350, -0.8235, 1.5122, 0.2019, 5.5143,
# -3.8153, 0.6771, 0.3011, -0.2994, -0.7320, -1.5857,
# -0.4785, -0.5584, -0.3226, 1.1932, -4.3901, -1.6923,
# 0.3526, -1.0625, 0.9279, -0.1843, -0.4376, 2.2389,
# -0.1558, -0.3959, -1.2987, 0.0279, -0.4938, -0.3364,
# 3.2596, -2.2647, 0.1448, 0.0726, 0.3968, -0.1885,
# -0.3960, 0.2141, 0.6785, -2.1622, -0.0043, -0.7516,
# 0.9367, -0.7092, 4.1853, 1.3348, 0.6993, 0.2043,
# -0.0916, 0.1392, -1.5672, 0.0867, -0.0346, 0.9226,
# -0.0470, -0.6870, -0.3002, -0.1131, 0.7785, 1.0582,
# 0.0914, 2.8785, 0.8164, -0.1048, 0.0573, -0.0499,
# -0.5990, 3.1714, 0.7925, 1.7461, 1.3243, 2.9236,
# 0.8966, -0.4455, 0.8763, -0.3036, 0.3302, 2.6581,
# 0.4608, -0.7280, -2.9457, -0.1973, 0.0585, -0.6555,
# -0.6621, -0.4549, 0.5812, 0.4495, 0.1350, 1.8521]], # [[ -0.0053, -0.0843, -0.1763, 0.3929, -0.1668, -0.6609,
# 0.1269, 0.2214, -0.0208, -0.3571, 0.7532, 0.7496,
# -4.9288, -0.5457, 0.3557, 0.4795, -0.2318, -0.9659,
# 0.6826, -1.6542, 0.4917, -0.3956, -1.5164, -0.2274,
# 0.6779, 1.1201, -3.1397, 0.0434, -0.4993, -0.8809,
# 6.1257, -5.6283, 0.4273, 1.5070, 0.6624, 0.1289,
# 0.2180, 0.9920, 0.1646, 0.8828, 0.5732, 0.3255,
# -0.0679, 0.9843, -1.8408, 1.0547, 0.1959, 0.0748,
# 0.1907, 0.4751, 0.3174, 0.0747, -0.6487, 0.0377,
# 0.5554, -0.4095, 0.2593, -0.0568, 0.3751, -0.3646,
# -0.2031, -0.3284, 0.4058, 1.2788, 0.1348, 1.8184,
# 0.8482, -0.7494, -0.2395, -0.4352, -0.1584, -0.0105,
# 0.2676, 0.3763, 2.1413, -1.3001, 0.3923, 1.6432,
# 0.2987, -1.2708, 5.5667, -0.1727, -0.5106, 0.5180,
# -6.7258, 0.5001, -0.3052, 0.0843, 0.0474, -0.2306,
# 0.1908, -2.2523, -1.5432, 0.2809, 0.7099, -0.4145,
# 0.7393, -0.6529, 0.7825, -0.0019, 1.3337, 2.2042,
# 9.2887, -0.8515, -0.0610, -0.6146, -1.5616, 0.3592,
# -1.3585, -0.2641, 1.4763, -3.2525, -0.5447, -0.0453,
# -1.0416, -2.4657, 0.0556, -0.7654, 0.2062, 0.0855,
# 3.0740, 0.0952, 0.4923, -0.1772, 0.9173, -4.2004,
# -0.1298, -2.4266, 0.0181, 0.5039, 0.0399, 7.9909,
# -0.5778, -2.9112, 0.4854, 1.2364, 0.0686, 0.6365,
# 0.1869, 0.6050, -0.1246, -0.1848, 0.5406, 0.2110,
# 1.2367, 1.9466, 0.0302, 0.2002, -0.5902, 0.1069]], # [[ -0.3559, -0.2859, 0.2699, -1.4359, 0.9814, 0.2811,
# 0.8539, -2.6654, 2.5455, 0.0434, 0.5947, -0.5325,
# -0.4638, 0.5487, 1.4376, 0.9863, 0.7429, -1.8308,
# 0.0402, -0.2282, 0.0366, 12.7877, -1.2491, -0.1437,
# -1.4960, 0.7364, 0.8599, 1.8343, -3.5117, -1.2758,
# 0.2930, -0.0472, -0.7527, 0.9555, 0.0446, -0.3389,
# -0.1985, 1.7953, -0.5702, 0.0141, 0.6166, -0.0924,
# -0.5182, 0.5146, 2.0801, 2.7460, -0.2606, 0.2090,
# 0.9266, -0.4758, 0.9961, -0.1723, -1.2069, -1.1735,
# 0.3683, -0.4933, -0.0604, -0.2354, 0.8239, -5.4226,
# 0.0854, 0.1185, -0.2656, -0.2689, 0.6047, -0.6246,
# 1.0131, -0.1673, -0.4990, -0.0690, -0.6092, -0.5205,
# 0.1808, 0.3061, 0.3924, 0.5868, 0.1452, 2.8930,
# -0.6085, 1.6086, -0.4763, 5.0389, 1.1569, 3.4060,
# -0.7565, 0.0247, 0.8477, 0.3714, -0.1043, 1.5607,
# 4.0700, -1.8363, 0.4370, -0.3571, 0.7268, 0.3435,
# 0.0972, 7.1477, -0.1486, 0.3342, -0.9733, 0.2311,
# 0.6104, -0.4988, 2.8838, -1.3387, 1.4291, -0.4121,
# -0.6722, 2.6834, -0.5188, 0.0428, -0.3452, -0.0131,
# -0.9004, -3.0346, -0.9254, 0.0150, -0.0386, 1.0639,
# -0.2444, -0.4562, 4.1626, -1.9304, 1.0662, 2.0683,
# -0.9553, -0.6434, -1.6777, 0.0702, -0.0113, -0.5503,
# -0.1873, -0.6916, 1.0729, -0.8234, 0.6421, 0.3022,
# -0.6065, 0.1016, 0.7792, 0.2533, -0.2670, -0.1314,
# 0.7515, -0.9859, 0.5050, -1.0552, -0.5632, 1.0697]], # [[ 0.3612, 2.4727, -4.6103, -1.6459, -1.7761, -1.4302,
# 0.2737, 0.3302, 4.0617, 0.7206, -0.0749, -0.8146,
# -1.0134, 0.8741, 1.9300, 0.5426, 0.1386, -1.3920,
# 0.4602, 1.3387, 0.0068, -0.3648, -7.6665, 0.9011,
# -0.3286, -1.5220, -0.2155, 0.7959, 4.0746, 0.9382,
# 0.0023, -0.2666, 0.4571, -1.9530, 0.3216, 2.1178,
# 0.4043, -0.0309, 2.5116, -1.2250, -0.9842, 5.0822,
# -4.1296, -5.3579, 0.9115, 0.4843, 0.1365, 0.0491,
# 0.1446, 0.6523, 0.0765, 0.3761, 0.0310, -0.4825,
# 7.1485, -0.4211, -3.7914, -0.2492, -0.3775, -0.4745,
# -1.3320, -1.8203, -1.0266, -0.4446, 2.2385, 0.6003,
# -0.1759, -1.9601, 2.3865, 1.3325, 4.8762, -0.2398,
# 7.5251, -0.4380, -2.3422, 0.6013, 13.8362, -0.4112,
# 2.3579, -0.1720, 1.0265, 0.6521, -0.7363, -0.7864,
# 0.2986, -0.1298, 0.5078, 0.1386, 1.4856, -0.3133,
# 0.9933, 1.5144, -0.0433, 1.0841, 0.3962, -0.0024,
# -0.3937, 2.2719, -0.0198, 1.6771, -1.2469, -0.8017,
# 0.1607, 0.0244, -0.1429, 0.9912, 0.1635, -1.2396,
# -0.1615, 1.0921, 0.8146, -0.3309, 0.8553, 0.4243,
# -3.5547, -0.1382, 0.1513, 0.4036, -0.2505, 0.2295,
# -0.6219, -0.7644, -0.7568, -0.4494, -0.0775, -0.0178,
# -0.2550, 0.2258, -2.7895, -0.3362, -0.2364, -0.9864,
# -1.3459, -1.8118, -0.4397, -0.8312, 0.3526, 1.3541,
# -0.0467, 1.6161, -0.4478, -0.5202, -0.4164, -0.8265,
# 0.1626, -4.2044, 3.2649, 0.2940, -0.8260, -0.4956]]], device='cuda:0'))
######## Padding functions used in pytorch. #########
1. torch.nn.utils.rnn.PackedSequence(*args)
Holds the data and list of batch_sizes of a packed sequence.
All RNN modules accept packed sequences as inputs.
2. torch.nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first=False)
Packs a Tensor containing padded sequences of variable length.
Input can be of size T*B** where T is the length of the longest sequence, B is the batch size, and the * is any number of dimensions.
If batch_first is True B*T** inputs are expected. The sequences should be sorted by length in a decreasing order.
3. torch.nn.utils.rnn.pad_packed_sequence(sequence, batch_first=False, padding_values=0.0, total_length=None)
Pads a packed batch of variable length sequences.
It is an inverse operation to pack_padded_sequence().
Batch elements will be ordered decreasingly by their length.
Note: total_length is useful to implement the pack sequence -> rnn -> unpack sequence .
Return: Tuple of tensor containing the padded sequence, and a tensor containing the list of lengths of each sequence in the batch.
4. torch.nn.utils.rnn.pad_sequence(sequence, batch_first=False, padding_value=0)
Pad a list of variable length Tensors with zero.
5. torch.nn.utils.rnn.pack_sequence(sequences)
Packs a list of variable length Tensors.
6. Tutorials on these functions.
(1). https://zhuanlan.zhihu.com/p/34418001
(2). https://zhuanlan.zhihu.com/p/28472545
总结起来就是：在利用 recurrent neural network 处理变长的句子序列时，我们可以配套的使用：
torch.nn.utils.rnn.pack_padded_sequence () 来对一个 mini-batch 中的句子进行 padding；
torch.nn.utils.rnn.pad_packed_sequence () 来避免 padding 对句子表示的影响。
===
Tutorial on word2vector using GloVe and Word2Vec的更多相关文章
- 词向量(one-hot/SVD/NNLM/Word2Vec/GloVe)
目录 词向量简介 1. 基于one-hot编码的词向量方法 2. 统计语言模型 3. 从分布式表征到SVD分解 3.1 分布式表征(Distribution) 3.2 奇异值分解(SVD) 3.3 基 ...
- word2vec 和 glove 模型的区别
2019-09-09 15:36:13 问题描述:word2vec 和 glove 这两个生成 word embedding 的算法有什么区别. 问题求解: GloVe (global vectors ...
- [白话解析] 带你一起梳理Word2vec相关概念
[白话解析] 带你一起梳理Word2vec相关概念 0x00 摘要 本文将尽量使用易懂的方式,尽可能不涉及数学公式,而是从整体的思路上来说,运用感性直觉的思考来帮大家梳理Word2vec相关概念. 0 ...
- NLP︱高级词向量表达（一）——GloVe（理论、相关测评结果、R&python实现、相关应用）
有很多改进版的word2vec,但是目前还是word2vec最流行,但是Glove也有很多在提及,笔者在自己实验的时候,发现Glove也还是有很多优点以及可以深入研究对比的地方的,所以对其进行了一定的 ...
- Getting Started with Word2Vec
Getting Started with Word2Vec 1. Source by Google Project with Code: https://code.google.com/archive ...
- 四步理解GloVe！(附代码实现)
1. 说说GloVe 正如GloVe论文的标题而言,GloVe的全称叫Global Vectors for Word Representation,它是一个基于全局词频统计(count-based & ...
- 词嵌入之GloVe
什么是GloVe GloVe(Global Vectors for Word Representation)是一个基于全局词频统计(count-based & overall statisti ...
- RNN 与 LSTM 的应用
之前已经介绍过关于 Recurrent Neural Nnetwork 与 Long Short-Trem Memory 的网络结构与参数求解算法( 递归神经网络(Recurrent Neural N ...
- 从Word Embedding到Bert模型—自然语言处理中的预训练技术发展史（转载）
转载 https://zhuanlan.zhihu.com/p/49271699 首发于深度学习前沿笔记 写文章 从Word Embedding到Bert模型—自然语言处理中的预训练技术发展史 张 ...
随机推荐
- 【转】查看sqlserver被锁的表以及如何解锁
查看被锁表: select request_session_id spid,OBJECT_NAME(resource_associated_entity_id) tableName from sys. ...
- codeforces 975C Valhalla Siege
题意: 有n个巫师站成一列,每个巫师有自己的血量. 一个人射箭攻击他们,每次造成若干点伤害,巫师按照给定的顺序承受伤害,如果伤害大了,那么死掉,伤害落到下一个巫师身上. 如果一轮攻击之后,所有的巫师都 ...
- 浏览器页面请求js、css大文件处理
当页面引用一个比较大的js和css文件时,会出现较大下载延迟,占用带宽的问题,如果一个应用里有很多这样的js或CSS文件,那么就需要优化了. 比如ext-all.js有1.4M,页面引用这个文件,正常 ...
- linux常用命令：cp 命令
cp命令用来复制文件或者目录,是Linux系统中最常用的命令之一.一般情况下,shell会设置一个别名,在命令行下复制文件时,如果目标文件已经存在,就会询问是否覆盖,不管你是否使用-i参数.但是如果是 ...
- HashMap集合存储自定义类
第一种情况,key为String,value为自定义类person类: 输出结果,key重复的被去掉了,key重复的那个value值之前的被最后一个覆盖了: 第二种情况,key为自定义类person类 ...
- let的使用 优先于闭包
let声明的变量在{}中使用,变量的作用域限制在块级域中 举例:使用js动态给ul添加li对象并点击第几项,显示当前点击是第几个 错误代码 window.onload = function(){ va ...
- SQL知识点、SQL语句学习
一. 数据库简介和创建1. 系统数据库在安装好SQL SERVER后,系统会自动安装5个用于维护系统正常运行的系统数据库: (1)master:记录了SQL SERVER实例的所有系统级消息,包括实例 ...
- java-web的mybatis的学习
idea开发必须是把Mapper文件与配置文件放到Resources标记的classpath目录下,eclips好像放到哪都行指定好路径就可以了, maven里面做好配置resources的路径,不然 ...
- easyui dialog 表单提交，弹框初始化赋值，dialog实现
//定义弹框html<div id="delete" class="easyui-dialog" title="Delete" dat ...
- strcpy和memcpy,memmove函数的区别
strcpy和memcpy的区别 strcpy和memcpy都是标准C库函数,它们有下面的特点. strcpy提供了字符串的复制.即strcpy只用于字符串复制,并且它不仅复制字符串内容之外,还会复制 ...