Tutorial on word2vector using GloVe and Word2Vec
Tutorial on word2vector using GloVe and Word2Vec
2018-05-04 10:02:53
Some Important Reference Pages First:
Reference Page: https://github.com/IliaGavrilov/NeuralMachineTranslationBidirectionalLSTM/blob/master/1_Bidirectional_LSTM_Eng_to_French.ipynb
Glove Project Page: https://nlp.stanford.edu/projects/glove/
Word2Vec Project Page: https://code.google.com/archive/p/word2vec/
Pre-trained word2vec model: https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing
Gensim Tutorial: https://radimrehurek.com/gensim/models/word2vec.html
===================================
===== For the Glove
===================================
1. Download one of the pre-trained model from Glove project page and Unzip the files.
Wikipedia 2014 + Gigaword 5 (6B tokens, 400K vocab, uncased, 50d, 100d, 200d, & 300d vectors, 822 MB download): glove.6B.zip
Common Crawl (42B tokens, 1.9M vocab, uncased, 300d vectors, 1.75 GB download): glove.42B.300d.zip
Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB download): glove.840B.300d.zip
2. Install the needed packages:
pickle, numpy, re, pickle, collections, bcolz
3. Run the following demo to test the results (extract the feature of given single word).
Code:
import pickle
import numpy as np
import re, pickle, collections, bcolz # with open('./glove.840B.300d.txt', 'r', encoding="utf8") as f: with open('./glove.6B.200d.txt', 'r') as f:
lines = [line.split() for line in f] print('==>> begin to load Glove pre-trained models.')
glove_words = [elem[0] for elem in lines]
glove_words_idx = {elem:idx for idx, elem in enumerate(glove_words)} # is elem: idx equal to glove_words_idx[elem]=idx?
glove_vecs = np.stack(np.array(elem[1:], dtype=np.float32) for elem in lines) print('==>> save into .pkl files.')
pickle.dump(glove_words, open('./glove.6B.200d.txt'+'_glove_words.pkl', 'wb'))
pickle.dump(glove_words_idx, open('./glove.6B.200d.txt'+'_glove_words_idx.pkl', 'wb')) ## saving array using specific function.
def save_array(fname, arr):
c=bcolz.carray(arr, rootdir=fname, mode='w')
c.flush() save_array('./glove.6B.200d.txt'+'_glove_vecs'+'.dat', glove_vecs) def load_glove(loc):
return (bcolz.open(loc+'_glove_vecs.dat')[:],
pickle.load(open(loc+'_glove_words.pkl', 'rb')),
pickle.load(open(loc+'_glove_words_idx.pkl', 'rb'))) ###############################################
print('==>> Loading the glove.6B.200d.txt files.')
en_vecs, en_wv_word, en_wv_idx = load_glove('./glove.6B.200d.txt')
en_w2v = {w: en_vecs[en_wv_idx[w]] for w in en_wv_word}
n_en_vec, dim_en_vec = en_vecs.shape print('==>> shown one demo: "King"')
demo_vector = en_w2v['king']
print(demo_vector)
print("==>> Done !")
Results:
wangxiao@AHU$ python tutorial_Glove_word2vec.py
==>> begin to load Glove pre-trained models.
==>> save into .pkl files.
==>> Loading the glove.6B.200d.txt files.
==>> shown one demo: "King"
[-0.49346 -0.14768 0.32166001 0.056899 0.052572 0.20192 -0.13506 -0.030793 0.15614 -0.23004 -0.66376001 -0.27316001 0.10391 0.57334 -0.032355 -0.32765999 -0.27160001 0.32918999
0.41305 -0.18085 1.51670003 2.16490006 -0.10278 0.098019 -0.018946 0.027292 -0.79479998 0.36631 -0.33151001 0.28839999 0.10436 -0.19166 0.27326 -0.17519 -0.14985999 -0.072333
-0.54370999 -0.29728001 0.081491 -0.42673001 -0.36406001 -0.52034998 0.18455 0.44121 -0.32196 0.39172 0.11952 0.36978999 0.29229 -0.42954001 0.46653 -0.067243 0.31215999 -0.17216
0.48874 0.28029999 -0.17577 -0.35100999 0.020792 0.15974 0.21927001 -0.32499 0.086022 0.38927001 -0.65638 -0.67400998 -0.41896001 1.27090001 0.20857 0.28314999 0.58238 -0.14944001
0.3989 0.52680999 0.35714 -0.39100999 -0.55372 -0.56642002 -0.15762 -0.48004001 0.40448001 0.057518 -1.01569998 0.21754999 0.073296 0.15237001 -0.38361999 -0.75308001 -0.0060254 -0.26232001
-0.54101998 -0.34347001 0.11113 0.47685 -0.73229998 0.77596998 0.015216 -0.66327 -0.21144 -0.42964 -0.72689998 -0.067968 0.50601 0.039817 -0.27584001 -0.34794 -0.0474 0.50734001
-0.30777001 0.11594 -0.19211 0.3107 -0.60075003 0.22044 -0.36265001 -0.59442002 -1.20459998 0.10619 -0.60277998 0.21573 -0.35361999 0.55473 0.58094001 0.077259 1.0776 -0.1867
-1.51680005 0.32418001 0.83332998 0.17366 1.12320006 0.10863 0.55888999 0.30799001 0.084318 -0.43178001 -0.042287 -0.054615 0.054712 -0.80914003 -0.24429999 -0.076909 0.55216002 -0.71895999
0.83319002 0.020735 0.020472 -0.40279001 -0.28874001 0.23758 0.12576 -0.15165 -0.69419998 -0.25174001 0.29591 0.40290001 -1.0618 0.19847 -0.63463002 -0.70842999 0.067943 0.57366002
0.041122 0.17452 0.19430999 -0.28641 -1.13629997 0.45116001 -0.066518 0.82615 -0.45451999 -0.85652 0.18105 -0.24187 0.20152999 0.72298002 0.17415 -0.87327999 0.69814998 0.024706
0.26174 -0.0087155 -0.39348999 0.13801 -0.39298999 -0.23057 -0.22611 -0.14407 0.010511 -0.47389001 -0.15645 0.28601 -0.21772 -0.49535 0.022209 -0.23575 -0.22469001 -0.011578 0.52867001 -0.062309 ]
==>> Done !
4. Extract the feature of the whole sentense.
Given one sentense, such as "I Love Natural Language Processing", we can translate this sentense into a matrix representation. Specifically, we represent each word of the sentense with a vector which lies in a continuous space (as shown above). You can also see this blog: https://blog.csdn.net/fangqingan_java/article/details/50493948 to futher understand this process.
但是有如下的疑问：
1. 给定的句子，长短不一，每一个单词的维度固定，但是总的句子的长度不固定，怎么办？
只好用 padding 的方法了，那么，怎么进行 padding 呢？
For CNN architecture usually the input is (for each sentence) a vector of embedded words [GloVe(w0), GloVe(w1), ..., GloVe(wN)] of fixed length N.
So commonly you preset the maximum sentence length and just pad the tail of the input vector with zero vectors (just as you did). Marking the beginning and the end of the sentence is more relevant for the RNN, where the sentence length is expected to be variable and processing is done sequentially. Having said that, you can add a new dimension to the GloVe vector, setting it to zero for normal words, and arbitrarily to (say) 0.5 for BEGIN, and 1 for END and a random negative value for OOV.
As for unknown words, you should ;) encounter them very rear, otherwise you might consider training the embeddings by yourself. 那么，是什么意思呢？
对于 CNN 的网络来说，由于需要处理 网格状的 data，所以，需要将句子进行填充，以得到完整的 matrix，然后进行处理；
对于 RNN/LSTM 的网络来说，是可以按照时刻，顺序处理的，所以呢？就不需要 padding 了？
2. Here is another tutorial on Word2Vec using deep learning tools --- Keras.
from numpy import array
from keras.preprocessing.text import one_hot
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.embeddings import Embedding
# define documents
docs = ['Well done!',
'Good work',
'Great effort',
'nice work',
'Excellent!',
'Weak',
'Poor effort!',
'not good',
'poor work',
'Could have done better.']
# define class labels
labels = array([,,,,,,,,,])
# integer encode the documents
vocab_size =
encoded_docs = [one_hot(d, vocab_size) for d in docs]
print("==>> encoded_docs: ")
print(encoded_docs)
# pad documents to a max length of words
max_length =
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')
print("==>> padded_docs: ")
print(padded_docs)
# define the model
model = Sequential()
model.add(Embedding(vocab_size, , input_length=max_length))
model.add(Flatten())
model.add(Dense(, activation='sigmoid'))
# compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
# summarize the model
print("==>> model.summary()")
print(model.summary())
# fit the model
model.fit(padded_docs, labels, nb_epoch=, verbose=)
# evaluate the model
loss, accuracy = model.evaluate(padded_docs, labels, verbose=)
print("==>> the final Accuracy: ")
print('Accuracy: %f' % (accuracy*))
The output is:
Using TensorFlow backend.
==>> encoded_docs:
[[, ], [, ], [, ], [, ], [], [], [, ], [, ], [, ], [, , , ]]
==>> padded_docs:
[[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]]
WARNING:tensorflow:From /usr/local/lib/python2./dist-packages/keras/backend/tensorflow_backend.py:: calling reduce_prod (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python2./dist-packages/keras/backend/tensorflow_backend.py:: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
==>> model.summary()
____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
embedding_1 (Embedding) (None, , ) embedding_input_1[][]
____________________________________________________________________________________________________
flatten_1 (Flatten) (None, ) embedding_1[][]
____________________________________________________________________________________________________
dense_1 (Dense) (None, ) flatten_1[][]
====================================================================================================
Total params:
Trainable params:
Non-trainable params:
____________________________________________________________________________________________________
None
-- ::59.053198: I tensorflow/core/platform/cpu_feature_guard.cc:] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
==>> the final Accuracy:
Accuracy: 89.999998
3. Maybe translate the sentence into vectors using skip-thought vectors is another goog choice. you can also check this tutorial from: http://www.cnblogs.com/wangxiaocvpr/p/7277025.html
4. But we still want to use GloVe to extract the vector of each word and concatenate them into a long vector.
import os
import numpy as np
from collections import OrderedDict
import re, pickle, collections, bcolz
import pdb seq_home = '/dataset/language-dataset/'
seqlist_path = '/dataset/language-train-video-list.txt'
output_path = 'data/language-train-video-list.pkl' def load_glove(loc):
return (bcolz.open(loc+'_glove_vecs.dat')[:],
pickle.load(open(loc+'_glove_words.pkl', 'rb')),
pickle.load(open(loc+'_glove_words_idx.pkl', 'rb'))) pre_trained_model_path = '/glove/glove.6B.200d.txt' ###############################################
print('==>> Loading the glove.6B.200d.txt files.')
en_vecs, en_wv_word, en_wv_idx = load_glove(pre_trained_model_path)
en_w2v = {w: en_vecs[en_wv_idx[w]] for w in en_wv_word}
n_en_vec, dim_en_vec = en_vecs.shape with open(seqlist_path,'r') as fp:
seq_list = fp.read().splitlines() data = {}
for i,seq in enumerate(seq_list):
img_list = sorted([p for p in os.listdir(seq_home+seq+'/imgs/') if os.path.splitext(p)[1] == '.jpg']) ## image list
gt = np.loadtxt(seq_home+seq+'/groundtruth_rect.txt', delimiter=',') ## gt files
language_txt = open(seq_home+seq+'/language.txt', "rw+") ## natural language description files line = language_txt.readline()
print("==>> language: %s" % (line)) gloveVector = []
test_txtName = seq_home+seq+'/glove_vector.txt'
f=file(test_txtName, "w") word_list = line.split(' ')
for word_idx in range(len(word_list)):
current_word = word_list[word_idx]
current_GloVe_vector = en_w2v[current_word] ## vector dimension is: 200-D gloveVector = np.concatenate((gloveVector, current_GloVe_vector), axis=0)
f.write(str(current_GloVe_vector))
f.write('\n')
f.close() print(i) assert len(img_list) == len(gt), "Lengths do not match!!" if gt.shape[1]==8:
x_min = np.min(gt[:,[0,2,4,6]],axis=1)[:,None]
y_min = np.min(gt[:,[1,3,5,7]],axis=1)[:,None]
x_max = np.max(gt[:,[0,2,4,6]],axis=1)[:,None]
y_max = np.max(gt[:,[1,3,5,7]],axis=1)[:,None]
gt = np.concatenate((x_min, y_min, x_max-x_min, y_max-y_min),axis=1) data[seq] = {'images':img_list, 'gt':gt, 'gloveVector': gloveVector} with open(output_path, 'wb') as fp:
pickle.dump(data, fp, -1)
The Glove vector can be saved into a txt file for each video, as shown in following screenshots.
[ 0.19495 0.60610002 -0.077356 0.017301 -0.51370001 0.22421999
-0.80773002 0.022378 0.30256 1.06669998 -0.10918 0.57902998
0.23986 0.1691 0.0072384 0.42197999 -0.20458999 0.60271001
0.19188 -0.19616 0.33070001 3.20020008 -0.18104 0.20784
0.49511001 -0.42258999 0.022125 0.24379 0.16714001 -0.20909999
-0.12489 -0.51766998 -0.13569 -0.25979999 -0.17961 -0.47663
-0.89539999 -0.27138999 0.17746 0.45827001 0.21363001 0.22342999
-0.049342 0.34286001 -0.32315001 0.27469999 0.95993 -0.25979
0.21250001 -0.21373001 0.19809 0.15455 -0.48581001 0.38925001
0.33747 -0.27897999 0.19371 -0.45872 -0.054217 -0.24022999
0.59153003 0.12453 -0.21302 0.058223 -0.046671 -0.011614
-0.32025999 0.64120001 -0.28718999 0.035138 0.39287001 -0.030683
0.083529 -0.010964 -0.62427002 -0.13575 -0.38468999 0.11454
-0.61037999 0.12594999 -0.17633 -0.21415 -0.37013999 0.21763
0.055366 -0.25242001 -0.45475999 -0.28105 0.18911 -1.58539999
0.64841002 -0.34621999 0.59254003 -0.39034 -0.44258001 0.20562001
0.44395 0.23019999 -0.35018 -0.36090001 0.62993002 -0.34698999
-0.31964999 -0.17361 0.51714998 0.68493003 -0.15587001 1.42550004
-0.94313997 0.031951 -0.26210001 -0.089544 0.22437 -0.050374
0.035414 -0.070493 0.17037 -0.38598001 0.0095626 0.26363
0.72125 -0.13797 0.70602 -0.50839001 -0.49722001 -0.48706001
0.16254 0.025619 0.33572 -0.64160001 -0.32541999 0.21896
0.05331 0.082011 0.12038 0.088849 -0.04651 -0.085435
0.036835 -0.14695001 -0.25841001 -0.043812 0.053209 -0.48954999
1.73950005 0.99014997 0.09395 -0.20236 -0.050135 0.18337999
0.22713999 0.83146 -0.30974001 0.51995999 0.068264 -0.28237
-0.30096999 -0.031014 0.024615 0.4294 -0.085036 0.051913
0.31251001 -0.34443 -0.085145 0.024975 0.0082017 0.17241
-0.66000998 0.0058962 -0.055387 -0.22315 0.35721999 -0.18962
0.25819999 -0.24685 -0.79571998 -0.09436 -0.10271 0.13713001
1.48660004 -0.57113999 -0.52649999 -0.25181001 -0.40792 -0.18612
0.040009 0.11557 0.017987 0.27149001 -0.14252 -0.087756
0.15196 0.064926 0.01651 -0.25334001 0.27437001 0.24246
0.018494 0.22473 ]
[ 4.32489991e-01 5.34709990e-01 -1.83240008e-02 1.56369999e-01
6.69689998e-02 7.63140023e-02 -7.16499984e-01 2.41520002e-01
1.70919999e-01 8.96220028e-01 -2.00739995e-01 1.88370004e-01
4.53570008e-01 -1.86419994e-01 -3.41060013e-01 -3.10570002e-02
-4.90720011e-02 6.55830026e-01 -7.43409991e-03 1.54770002e-01
1.55849993e-01 3.11129999e+00 -4.12149996e-01 1.89439997e-01
1.86639994e-01 -2.00540006e-01 6.63070008e-02 3.90180014e-02
6.06740005e-02 -3.17880005e-01 1.82290003e-02 -6.80719972e-01
-1.33360000e-02 2.32099995e-01 -8.23459998e-02 -2.68420011e-01
-3.51870000e-01 -2.03860000e-01 6.69749975e-02 3.02689999e-01
1.84330001e-01 1.44429997e-01 1.38980001e-01 1.93629991e-02
-4.09139991e-01 2.24950001e-01 8.02020013e-01 -3.05290014e-01
2.22399995e-01 -6.10979982e-02 3.48910004e-01 1.13349997e-01
-4.14329991e-02 7.31640011e-02 3.17719996e-01 -2.59590000e-01
3.17759991e-01 -1.78080007e-01 2.12380007e-01 1.07529998e-01
4.37519997e-01 -1.81329995e-01 -1.53909996e-01 7.27130026e-02
4.14730012e-01 2.73079991e-01 -1.06940001e-01 7.45599985e-01
3.02549988e-01 1.49999997e-02 6.33790016e-01 1.03399999e-01
2.18640000e-01 -7.43470015e-03 -7.09949970e-01 4.60739993e-02
-8.34990025e-01 1.84519999e-02 -2.75200009e-01 -2.48000007e-02
-1.21320002e-01 3.29720005e-02 1.07320003e-01 2.18070000e-01
2.79079992e-02 -2.99050003e-01 -3.06030005e-01 -3.17970008e-01
2.10580006e-01 -1.16180003e+00 4.57720011e-01 -2.60210007e-01
3.76190007e-01 -2.16749996e-01 -1.68540001e-01 1.77729994e-01
-1.46709993e-01 2.67820001e-01 -5.86109996e-01 -4.98769999e-01
3.83679986e-01 2.51020014e-01 -3.90020013e-01 -6.41409993e-01
5.34929991e-01 2.76479989e-01 -4.32830006e-01 1.61350000e+00
-1.58399999e-01 -2.03970000e-01 -2.18089998e-01 -9.74659983e-04
-7.48440027e-02 3.78959998e-02 -4.03719991e-01 -1.74640000e-01
4.11469996e-01 -2.80400008e-01 -3.42570007e-01 5.72329983e-02
7.18890011e-01 -1.30419999e-01 5.75810015e-01 -3.27690005e-01
7.75609985e-02 -2.66130000e-01 4.82820012e-02 1.24679998e-01
1.29319996e-01 7.65379965e-02 -3.85809988e-01 3.72209996e-01
-1.67280003e-01 1.86330006e-01 -2.20280007e-01 3.27170014e-01
1.74089998e-01 -2.51549989e-01 -5.47109991e-02 -2.48549998e-01
-2.24260002e-01 -3.80129993e-01 5.53080022e-01 -1.52050003e-01
1.12820005e+00 3.70869994e-01 -4.62579988e-02 -3.56009990e-01
-1.72120005e-01 1.23460002e-01 6.58179998e-01 5.05930007e-01
-4.99610007e-01 1.97339997e-01 1.37759998e-01 3.98500008e-04
-1.38300002e-01 -6.67430013e-02 -7.20809996e-02 4.00720000e-01
-3.77620012e-01 -9.86199975e-02 2.04119995e-01 -3.42869997e-01
-2.01509997e-01 -1.18270002e-01 -1.60109997e-01 1.63570002e-01
-3.79029989e-01 -2.45529994e-01 -5.31699993e-02 -4.19999994e-02
1.82099998e-01 2.23959997e-01 4.50800002e-01 -2.03030005e-01
-6.16330028e-01 -2.02739999e-01 2.18419999e-01 3.66939992e-01
7.68440008e-01 -2.92189986e-01 -9.80089977e-02 -2.92939991e-01
-1.93189994e-01 1.45720005e-01 2.45150000e-01 -7.11840019e-02
3.97929996e-01 -3.33019998e-03 -4.01179999e-01 1.39760002e-01
8.27540010e-02 -1.29600003e-01 -3.05500001e-01 8.98869988e-03
2.26559997e-01 3.21410000e-01 -4.29780006e-01 4.74779993e-01]
[ 5.73459983e-01 5.41700006e-01 -2.34770000e-01 -3.62399995e-01
4.03699994e-01 1.13860004e-01 -4.49330002e-01 -3.09909999e-01
-5.34110004e-03 5.84259987e-01 -2.59559993e-02 4.93930012e-01
-3.72090004e-02 -2.84280002e-01 9.76959988e-02 -4.89069998e-01
2.60269996e-02 3.76489997e-01 5.77879995e-02 -4.68070000e-01
8.12880024e-02 3.28250003e+00 -6.36900008e-01 3.79559994e-01
3.81670007e-03 9.36070010e-02 -1.28549993e-01 1.73800007e-01
1.05219997e-01 2.86480010e-01 2.10889995e-01 -4.70759988e-01
2.77330000e-02 -1.98029995e-01 7.63280019e-02 -8.46289992e-01
-7.97079980e-01 -3.87430012e-01 -3.04220002e-02 -2.68489987e-01
4.85850006e-01 1.28950000e-01 3.83540004e-01 3.87219995e-01
-3.85239989e-01 1.90750003e-01 4.89980012e-01 1.32780001e-01
1.07920002e-02 2.67699987e-01 1.78120002e-01 -1.14330001e-01
-3.34939986e-01 8.73059988e-01 7.58750021e-01 -3.03779989e-01
-1.56259999e-01 1.20850001e-03 2.33219996e-01 2.79529989e-01
-1.84939995e-01 -1.41460001e-01 -1.89689994e-01 -3.83859985e-02
3.58740002e-01 6.55129999e-02 6.05649985e-02 6.63389981e-01
-8.32519978e-02 6.51630014e-02 5.17610013e-01 1.61709994e-01
4.60110009e-01 1.63880005e-01 -1.23989999e-01 3.11219990e-01
-1.54119998e-01 -1.09169997e-01 -4.25509989e-01 1.14179999e-01
2.51370013e-01 -5.61579987e-02 -2.59270012e-01 2.81630009e-01
-1.80939995e-02 1.60650000e-01 -4.85060006e-01 -9.89030004e-01
2.50220001e-01 -1.67359993e-01 4.14739996e-01 1.77010000e-01
4.24070001e-01 1.10880002e-01 -1.83599994e-01 -1.24100000e-01
-3.47799987e-01 9.90779996e-02 -2.23810002e-01 -1.12450004e-01
-2.11559996e-01 3.07060010e-03 -2.36070007e-01 2.72610001e-02
3.64300013e-01 3.99219990e-02 -1.83689997e-01 1.22660005e+00
-7.76400030e-01 -6.62249982e-01 1.57239996e-02 -1.49690002e-01
8.46489966e-02 2.68139988e-01 -1.67649999e-01 -3.19420010e-01
2.84940004e-01 -7.00000003e-02 1.20099997e-02 -1.22189999e-01
5.63099980e-01 -3.19999993e-01 5.01089990e-01 -1.02090001e-01
4.65750009e-01 -7.15420008e-01 1.72930002e-01 5.82589984e-01
7.83839971e-02 -3.38440016e-02 -2.51289994e-01 3.65029991e-01
3.15780006e-02 -6.57779992e-01 5.47499992e-02 8.71890008e-01
1.24550000e-01 -4.58770007e-01 -2.69650012e-01 -4.67790008e-01
-2.85780011e-03 1.78100005e-01 6.39689982e-01 1.39950007e-01
9.75960016e-01 1.18359998e-01 -6.39039993e-01 -1.54159993e-01
6.52619973e-02 2.43290007e-01 6.64759994e-01 2.50690013e-01
-1.02519996e-01 -3.28390002e-01 -8.55590031e-02 -1.27739999e-02
-1.94309995e-01 5.61389983e-01 -3.57329994e-01 -2.03439996e-01
-1.24130003e-01 -3.44309986e-01 -2.32960001e-01 -2.11870000e-01
8.53869990e-02 7.00630024e-02 -1.98029995e-01 -2.60230005e-02
-3.90370011e-01 8.00019979e-01 4.05770004e-01 -7.98629969e-02
3.52629989e-01 -3.40429991e-01 3.96759987e-01 2.28619993e-01
-3.50279987e-01 -4.73439991e-01 5.97419977e-01 -1.16570003e-01
1.05519998e+00 -4.15699989e-01 -8.05519968e-02 -5.65709993e-02
-1.66219994e-01 1.92739993e-01 -9.51749980e-02 -2.07810000e-01
1.56200007e-01 5.02309985e-02 -2.79150009e-01 4.37420011e-01
-3.12370002e-01 1.31940007e-01 -3.32780004e-01 1.88769996e-01
-2.34219998e-01 5.44179976e-01 -2.30690002e-01 3.49469990e-01]
[ -1.43299997e-01 -6.54269993e-01 -4.71419990e-01 -2.22780004e-01
-1.74260005e-01 6.43490016e-01 -7.60049999e-01 -8.42769966e-02
2.19850004e-01 -4.24650013e-01 -2.56919991e-02 -3.27909999e-02
-3.98149997e-01 -1.26000000e-02 6.31869972e-01 -6.31640017e-01
-2.70480007e-01 2.09920004e-01 5.72669983e-01 8.88589993e-02
2.24289998e-01 1.78419995e+00 8.73669982e-01 -2.23949999e-01
4.91869986e-01 6.86379969e-02 6.80689991e-01 1.85680002e-01
-2.85780013e-01 -1.06030002e-01 -3.07460010e-01 1.06180003e-02
-1.62290007e-01 -3.62700000e-02 8.02920014e-02 -2.17669994e-01
-7.92110026e-01 -7.06340015e-01 -2.89400011e-01 -1.51869997e-01
3.09009999e-01 -1.41849995e-01 -4.18920010e-01 -1.66299999e-01
-1.68510005e-01 1.04549997e-01 -3.49110007e-01 -1.21399999e+00
3.21929991e-01 1.88219994e-01 -3.50019991e-01 -2.23629996e-01
3.85280013e-01 1.40709996e-01 1.34890005e-01 3.89420003e-01
1.82109997e-01 1.32369995e-01 -2.60610014e-01 5.10110021e-01
1.26430005e-01 -2.94450015e-01 -1.34350002e-01 -1.44600004e-01
1.98740005e-01 -3.41720015e-01 2.40710005e-01 -6.08640015e-02
8.60409975e-01 8.92559998e-04 3.46020013e-01 -2.53780007e-01
2.53749996e-01 1.20920002e-01 -3.82739991e-01 1.55090000e-02
-6.41290024e-02 2.54729986e-01 -1.78489998e-01 2.37949997e-01
3.26880008e-01 2.03940004e-01 -5.65020025e-01 3.87109995e-01
-7.00220019e-02 -4.63999987e-01 -1.98210001e-01 -7.27079988e-01
5.36920011e-01 -9.09250021e-01 -2.36699998e-01 -6.20739982e-02
2.38769993e-01 3.24860007e-01 5.51190019e-01 -4.07079995e-01
1.40870005e-01 2.83569992e-01 -1.33379996e-01 -3.99740010e-01
-1.66620001e-01 4.43010002e-01 -6.85970008e-01 1.54200003e-01
6.58490002e-01 2.95630004e-02 2.61550009e-01 1.10969996e+00
-6.78719997e-01 -1.91960007e-01 7.45159984e-02 -3.53740007e-01
-7.15600014e-01 4.14079994e-01 -3.37280005e-01 2.05780007e-02
2.17340007e-01 6.48889989e-02 -3.51080000e-01 -2.44440004e-01
-1.98780000e-01 3.02210003e-01 -1.32180005e-01 -4.86490011e-01
-3.55599999e-01 2.83119995e-02 7.04949975e-01 7.35610008e-01
5.98779976e-01 -3.94160002e-01 -3.96349996e-01 2.55739987e-01
1.30939996e+00 -6.69730008e-02 4.72090006e-01 3.20030004e-01
-2.39020005e-01 1.87910005e-01 2.14310005e-01 3.10609996e-01
-7.22400010e-01 -2.59559989e-01 7.50709996e-02 5.45759976e-01
1.33249998e+00 5.97169995e-01 -7.56640017e-01 2.30540007e-01
3.45209986e-01 8.61710012e-01 5.65129995e-01 1.33000001e-01
4.27370012e-01 -4.36790008e-03 -5.15600026e-01 3.94329995e-01
-1.58559993e-01 -1.29820004e-01 -5.47240004e-02 -1.01789999e+00
6.39170036e-02 -6.47679985e-01 -7.53360018e-02 1.49210006e-01
5.53300023e-01 -6.24869987e-02 -2.01309994e-01 1.70489997e-01
-3.22829992e-01 2.43039995e-01 2.64449995e-02 -9.52059999e-02
3.80879998e-01 -3.69230002e-01 4.54270005e-01 -3.31169993e-01
-1.93750001e-02 1.59300007e-02 1.85659993e-02 -5.27350008e-01
1.05949998e+00 -4.31519985e-01 2.57169992e-01 -9.44539979e-02
7.53460005e-02 -2.06599995e-01 3.21940005e-01 -1.23549998e-02
4.72169995e-01 2.44020000e-01 -3.63059998e-01 1.71650007e-01
-3.50499988e-01 1.40990004e-01 1.18069999e-01 3.65170002e-01
-1.65889993e-01 -3.93799990e-02 -3.70829999e-01 2.03909993e-01]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[ 5.24869978e-01 -1.19410001e-01 -2.02419996e-01 -6.23929977e-01
-1.53799996e-01 -6.63369969e-02 -3.68499994e-01 2.86490005e-02
1.37950003e-01 -5.87819993e-01 6.02090001e-01 2.60540005e-02
7.07889974e-01 1.20329998e-01 -1.74310002e-02 4.03360009e-01
-3.19680005e-01 -2.50609994e-01 1.60889998e-01 2.47649997e-01
7.79359996e-01 2.74070001e+00 1.19589999e-01 -2.67529994e-01
-3.82809997e-01 -3.36580008e-01 1.41039997e-01 -4.65480000e-01
-8.92079994e-02 2.22540006e-01 -3.60740013e-02 -7.10140020e-02
6.23199999e-01 3.22770000e-01 4.15650010e-01 -3.68530005e-02
-5.82859993e-01 -6.26510024e-01 -3.26169990e-02 2.74789989e-01
-2.66950011e-01 5.27690016e-02 -1.09500003e+00 -1.99760008e-03
-7.49390006e-01 -1.89999994e-02 -1.87619999e-01 -5.19330025e-01
1.71590000e-01 4.40690011e-01 1.90789998e-01 -4.57339995e-02
-2.43069995e-02 2.32710004e-01 1.02179997e-01 -6.02790006e-02
-2.63680011e-01 -1.49090007e-01 4.33890015e-01 -8.51420015e-02
-6.61419988e-01 -6.23379983e-02 -1.37920007e-01 4.54079986e-01
-1.51400000e-01 1.14929996e-01 5.48650026e-01 2.82370001e-01
-2.55129993e-01 1.11149997e-01 -8.47479999e-02 -9.66319963e-02
4.88200009e-01 7.48070002e-01 -6.74910009e-01 3.21949989e-01
-8.98810029e-01 -3.58159989e-01 8.54640007e-02 -1.67390004e-01
-1.60109997e-01 -1.06349997e-01 -2.96649992e-01 -3.03389996e-01
2.29310006e-01 6.51580021e-02 1.70359999e-01 -2.08039999e-01
-1.12930000e-01 -1.58399999e-01 -1.88710004e-01 5.38689971e-01
-3.58050019e-02 3.26770008e-01 -3.18859994e-01 2.01079994e-02
-1.15209997e-01 -3.67400013e-02 5.97510003e-02 9.09829974e-01
1.55420005e-01 1.63939998e-01 3.57499987e-01 -4.75789994e-01
-3.53519991e-02 -1.30729997e+00 2.40319997e-01 1.24170005e+00
4.63180006e-01 -2.90019996e-02 6.63789988e-01 -2.20129997e-01
6.57410026e-01 -2.34209999e-01 -7.43539989e-01 8.95029977e-02
-1.41269997e-01 3.54860008e-01 -6.87049985e-01 -9.53029990e-02
-1.59720004e-01 -4.36020009e-02 5.39509989e-02 -1.33180007e-01
4.47530001e-01 1.41289994e-01 2.71119997e-02 1.18179999e-01
2.02419996e-01 -5.98179996e-01 -6.83469996e-02 -1.58150002e-01
2.42929995e-01 8.88189971e-02 -1.16219997e-01 -3.82679999e-01
1.30649999e-01 -3.00619990e-01 -8.46930027e-01 1.90159995e-02
6.41009986e-01 -8.42439979e-02 7.83680022e-01 -2.63660014e-01
1.98839998e+00 -7.38060027e-02 7.21970022e-01 4.84739989e-02
1.62070006e-01 3.40840012e-01 -1.85289994e-01 2.85659999e-01
-2.35440001e-01 -2.69939989e-01 -5.27239978e-01 2.74859995e-01
6.56930029e-01 2.20569998e-01 -6.80390000e-01 3.59100014e-01
2.51430005e-01 7.24399984e-02 2.58690000e-01 3.22510004e-01
3.57239991e-02 -3.88200015e-01 1.18890002e-01 -2.90010005e-01
-6.32229984e-01 -1.48519993e-01 -1.21040002e-01 -1.74060002e-01
2.48679996e-01 4.74539995e-01 -2.50710011e-01 -5.23679972e-01
-1.82730004e-01 -8.30529988e-01 6.15109980e-01 -3.84620011e-01
9.74919975e-01 4.49510008e-01 -5.42559981e-01 -3.18340003e-01
-5.07809997e-01 -2.73739994e-02 -1.26379997e-01 5.42479992e-01
4.85489994e-01 8.55289996e-02 -1.42509997e-01 8.77849981e-02
2.47189999e-01 -2.38490000e-01 6.14570007e-02 -4.23129983e-02
3.57470006e-01 9.73920003e-02 -3.58500004e-01 1.70269996e-01]
[ 7.54669979e-02 -2.92360008e-01 -2.60369986e-01 -2.81670004e-01
1.60970002e-01 -1.94720000e-01 -2.82059997e-01 4.91409987e-01
-4.11189999e-03 9.58790034e-02 3.33959997e-01 1.68740004e-02
2.06159994e-01 -1.57429993e-01 -1.61379993e-01 2.15379998e-01
-1.48359999e-01 2.54599992e-02 -4.36809987e-01 -6.36610016e-02
5.52609980e-01 3.08189988e+00 -1.17839999e-01 -4.61309999e-01
-2.76089996e-01 2.09480003e-01 -2.05440000e-01 -5.71550012e-01
3.34479988e-01 1.59130007e-01 2.54360004e-03 1.80040002e-01
1.34719998e-01 -9.74040031e-02 3.55369985e-01 -4.74280000e-01
-7.92569995e-01 -5.44179976e-01 2.42999997e-02 6.35990024e-01
1.23369999e-01 -1.29130006e-01 -2.65650004e-01 -2.49569997e-01
-5.21990001e-01 -4.05229986e-01 4.84030008e-01 1.83729995e-02
2.30389997e-01 6.21379986e-02 -1.92919999e-01 2.95060009e-01
-3.57930005e-01 1.67019993e-01 3.18679988e-01 -3.60540003e-01
-1.09779999e-01 -1.56320006e-01 4.59210008e-01 9.64749977e-02
-3.77000004e-01 -7.76150003e-02 -4.88990009e-01 2.05750003e-01
5.05429983e-01 5.34190014e-02 -2.59779990e-01 5.10420024e-01
9.75209996e-02 3.26059997e-01 1.43539995e-01 2.27149995e-03
4.86149997e-01 4.69379991e-01 -4.11240011e-01 -1.71480000e-01
-3.97439986e-01 -2.89009988e-01 -1.77560002e-01 3.70009989e-02
3.48300010e-01 1.59339994e-01 -7.42810011e-01 1.88970000e-01
4.36850004e-02 5.72080016e-01 -6.70159996e-01 -4.39470001e-02
-2.83360004e-01 -3.19959998e-01 -2.04040006e-01 -8.78980011e-02
-1.57240003e-01 2.18180008e-02 -5.67569971e-01 6.32960021e-01
-1.00970000e-01 -6.55760020e-02 5.82689978e-03 3.30349989e-02
3.97830009e-01 -3.11659992e-01 -6.10889971e-01 2.75590003e-01
1.00079998e-01 -4.19900000e-01 6.35599997e-03 1.87170005e+00
3.14729989e-01 -3.60040009e-01 8.13839972e-01 -2.17099994e-01
-1.84589997e-02 -2.26319999e-01 1.45850003e-01 -1.43500000e-01
-4.14239988e-02 5.59740007e-01 -6.67519987e-01 -2.19589993e-01
1.90109998e-01 3.30150008e-01 6.12900019e-01 4.67709988e-01
4.20260012e-01 -5.28190017e-01 2.31650006e-02 3.29100005e-02
4.73060012e-01 1.40060000e-02 -1.73960000e-01 -4.43619996e-01
4.13769990e-01 -2.06790000e-01 3.92830014e-01 3.02109987e-01
7.31339976e-02 4.21640016e-02 -9.27100003e-01 -4.76139992e-01
2.43100002e-01 -1.33790001e-01 -2.22379997e-01 -4.14570011e-02
1.58500004e+00 3.74810010e-01 2.59939991e-02 -2.42719993e-01
3.05779994e-01 1.46870002e-01 1.16659999e-01 -2.94179991e-02
-7.83390030e-02 -2.25119993e-01 1.33149996e-01 -6.48420006e-02
-2.86870003e-01 -1.05600003e-02 -3.46679986e-01 4.21449989e-02
-6.00409985e-01 8.24810028e-01 3.10220003e-01 1.64890006e-01
-7.29210004e-02 1.93939999e-01 -9.84980017e-02 -2.03830004e-02
-4.09090012e-01 -1.04039997e-01 1.91689998e-01 -1.59689993e-01
3.80259991e-01 6.28019989e-01 2.59499997e-01 -3.33669990e-01
-7.33330011e-01 -4.07429993e-01 6.84230030e-01 -6.63380027e-02
5.04360020e-01 -2.89829999e-01 -3.90859991e-01 -4.59310003e-02
-2.66240001e-01 -1.67699993e-01 -1.50370002e-01 1.48279995e-01
-2.81430006e-01 -1.70870006e-01 -2.55760014e-01 -5.62830009e-02
-1.66500002e-01 3.51060003e-01 4.10320014e-02 2.73110002e-01
3.00200004e-02 1.64649993e-01 -8.41889977e-02 5.75059988e-02]
[ 5.46909988e-01 -7.55890012e-01 -9.20799971e-01 -8.20680022e-01
1.48800001e-01 -1.32200003e-01 2.49499991e-03 5.39979994e-01
-3.12929988e-01 -2.12009996e-02 -2.96559989e-01 1.51110003e-02
-3.76980007e-02 -4.07290012e-01 3.36799994e-02 -2.90179998e-01
-5.64790010e-01 6.47960007e-01 3.96770000e-01 -1.91139996e-01
6.98249996e-01 1.61090004e+00 8.42899978e-01 -1.45980000e-01
-1.13940001e-01 -5.39680004e-01 -7.22540021e-01 -1.47310004e-01
-6.78520024e-01 1.93159997e-01 4.17819992e-03 -3.96059990e-01
8.99320021e-02 3.30909997e-01 -4.09649983e-02 4.59470004e-01
-3.07020009e-01 -2.61469990e-01 5.39489985e-01 6.60130024e-01
-5.68049997e-02 -3.70750010e-01 -1.46880001e-03 -1.48770005e-01
2.13000000e-01 -1.36710003e-01 5.12350023e-01 -5.27249992e-01
2.73449998e-02 3.00159991e-01 4.59829986e-01 3.65709990e-01
2.23989993e-01 1.47589996e-01 -9.97050032e-02 2.26520002e-01
-5.95120013e-01 -4.34839994e-01 -5.26629984e-02 -3.05240005e-01
-1.91280007e-01 -4.08630013e-01 -2.83719987e-01 6.89159989e-01
-7.18529999e-01 2.89909989e-01 -1.31809995e-01 -8.75229985e-02
2.70280004e-01 3.31449993e-02 1.93159997e-01 -2.50629991e-01
1.13150001e-01 4.88560013e-02 -4.41439986e-01 -8.88149977e-01
-3.45840007e-01 -5.47370017e-01 2.13919997e-01 1.46469995e-01
-3.76450002e-01 5.63120008e-01 4.44130003e-01 -6.05080016e-02
5.85870028e-01 4.50850010e-01 -1.27309993e-01 6.86409995e-02
3.78729999e-01 -7.15900004e-01 2.72450000e-01 2.16839999e-01
-5.09050012e-01 2.82240003e-01 5.28559983e-01 2.97140002e-01
-3.33429992e-01 -1.62200004e-01 -3.97210002e-01 3.37130010e-01
6.61469996e-01 -4.09590006e-01 -1.97520003e-01 -8.77860010e-01
-4.36379999e-01 1.79399997e-01 3.39769991e-03 9.73770022e-01
7.32789993e-01 -3.44499983e-02 -4.76709992e-01 -4.99819994e-01
4.97110009e-01 -7.77419984e-01 -3.60000014e-01 -1.58899993e-01
1.19869998e-02 -1.39750004e-01 -5.91250002e-01 -1.58280000e-01
9.19710025e-02 4.62440014e-01 -1.17980000e-02 8.29930007e-01
7.96280026e-01 -5.64369977e-01 -1.02080004e-02 -3.86090010e-01
-6.74889982e-01 -3.42109986e-02 1.99829996e-01 2.01010004e-01
7.43009984e-01 -3.58200014e-01 1.07579999e-01 -7.77289987e-01
3.24770004e-01 -7.43889987e-01 -7.90560007e-01 4.95799989e-01
3.70090008e-01 -1.55349998e-02 1.20260000e+00 -6.90900028e-01
1.49619997e+00 4.31109995e-01 2.64299989e-01 -5.99219978e-01
-4.51359987e-01 7.66879976e-01 -1.43020004e-02 1.43340006e-01
1.19379997e-01 -5.56930006e-01 -9.95339990e-01 8.20089996e-01
-3.02410007e-01 9.95709971e-02 -4.92139995e-01 2.25950003e-01
5.18890023e-01 6.51669979e-01 -7.13440031e-02 3.06109991e-03
-2.95789987e-01 -4.40490007e-01 -1.61009997e-01 2.52220005e-01
-3.50809991e-01 -7.96020031e-02 4.33560014e-01 -1.43889993e-01
-1.26969993e-01 1.01450002e+00 -6.24639988e-02 -5.60909986e-01
-6.52670026e-01 1.86299995e-01 -3.76190007e-01 -2.40750000e-01
2.50979990e-01 -2.06090003e-01 -8.57209980e-01 -5.84480017e-02
-1.23539999e-01 -7.35830009e-01 -4.83509988e-01 2.75110006e-01
-1.10220000e-01 3.17330003e-01 1.87999994e-01 1.05719995e+00
5.85460007e-01 3.13740000e-02 2.11569995e-01 1.13799997e-01
4.65829998e-01 -3.05020005e-01 -4.51409996e-01 2.87349999e-01]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[-0.34636 -0.88984001 -0.50321001 -0.43516001 0.54518998 0.17437001
-0.093541 0.16141 -0.46575999 -0.22 -0.31415001 -0.13484
-0.37617999 -0.67635 0.78820002 -0.33384001 -0.42414001 0.32367
0.50670999 0.21540999 0.43296 1.49049997 0.31795001 -0.15196
0.2579 -0.35666001 -0.63880002 -0.086453 -0.94755 0.19203
0.31161001 -0.74491 -0.59210998 0.4332 -0.064934 -0.48862001
0.35600999 -0.44780999 -0.015773 0.18203001 0.051751 -0.2854
-0.14727999 0.1513 -0.33048001 0.27135 1.16659999 -0.36662
0.090829 0.87301999 -0.13505 0.21204001 0.57270002 0.54259002
-0.50335002 0.16767 -0.82530999 -0.45962 -0.42642 -0.2164
0.088689 -0.15061 -0.16785 -0.31794 -0.69608998 0.40715
-0.29190999 -0.042072 0.90051001 0.35947999 0.030644 -0.028853
0.086821 0.74741 -0.52008998 0.20655 0.44053999 -0.11865
-0.15449999 -0.22457001 -0.15453 0.16101 -0.30825001 -0.28479999
-0.50823998 0.48732999 -0.012029 0.034592 0.48304 -0.56752002
-0.057299 0.22070999 -0.34200001 -0.060634 0.95499998 -0.60952997
0.59577 -0.11553 -0.67475998 0.52658999 0.82163 0.35126001
0.15521 0.12135 0.38191 0.24228001 -0.51485997 1.14810002
0.07281 0.23024 -0.68901998 -0.17606001 -0.24308001 -0.13686
-0.13467 0.059625 -0.68668997 0.15907 0.11983 -0.024954
0.34898001 0.15456 0.047524 0.23616999 0.54784 -1.01380002
0.10351 0.26865 -0.064867 0.23893 0.026141 0.081648
0.74479997 -0.67208999 0.23351 -0.55070001 -0.14296 -0.30303001
-0.40233999 0.012984 0.86865002 -0.14025 1.13900006 -0.093339
1.56060004 0.41804001 0.54067999 -0.43340999 -0.090589 0.56682003
-0.21291 0.45693001 -0.64519 -0.05866 0.21477 0.45563
-0.15218 0.36307001 -0.25441 -0.72013998 0.52191001 0.55509001
-0.073841 0.44994 -0.11501 0.1911 0.077304 0.18629
0.60244 0.028085 0.17228 -0.24455 0.04822 0.51318002
-0.06824 0.35515001 -0.80987 -0.42732999 -0.72728997 0.47817001
0.87611997 -0.18855 0.30390999 -0.14161 0.26699001 -0.81572002
-0.67589998 -0.34687999 0.53188998 0.75443 -0.083874 0.77434999
0.081108 -0.29840001 -0.24409001 -0.14574 -0.1186 0.085964
0.48076999 -0.13097 ]
[ 1.07439995 -0.49325001 -0.23126 -0.197 -0.087485 -0.16806
-0.11092 0.42857999 -0.16893999 0.0094633 -0.50453001 -0.40006
0.31169 0.50295001 -0.48537001 -0.20689 -0.62162 0.38407999
0.22182 0.051087 -0.018028 1.37919998 0.3163 -0.17425001
0.11344 -0.42723 -0.28510001 -0.1246 -0.2024 0.18217
-0.37834001 -0.22421999 0.38877001 0.20065001 -0.29708999 0.77051002
0.13087 -0.25084001 0.54755002 0.38086 0.28174999 -0.15691
-0.71987998 0.24118 0.073913 -0.46965 1.02180004 0.049863
0.036841 0.54706001 -0.15903001 0.53780001 -0.10335 0.51644999
-0.25512001 -0.18553001 -0.51804 -0.24337 0.57081997 -0.39017001
-0.17132001 -0.14939 -0.1724 0.91408002 -0.45778 0.40143001
0.075224 -0.4104 -0.1714 -0.63552999 0.60185999 -0.3193
-0.46136999 0.030652 0.32890001 -0.2472 -0.49831 -0.90982997
-0.057251 0.20213 -0.51845998 0.46320999 0.032707 0.29872999
1.11189997 -0.35784999 0.34929001 -0.51739001 0.25235 -1.0309
0.21052 0.06349 -0.10589 0.43222001 0.20389 0.065589
-0.62914002 0.1096 -0.86363 0.44760999 0.43114999 0.041376
-0.42429999 -0.080897 0.093115 0.22603001 0.31176999 0.83004999
-0.25659001 0.013861 0.38326001 -0.52025998 0.30410001 -0.52507001
-0.78566003 -0.046498 0.41154999 -0.21447 -0.24202999 -0.24732
1.01129997 0.067517 0.18331 -0.17636 0.49777001 -0.21067999
0.0037579 0.22881 -0.15993001 -0.13421001 -0.27379999 -0.20734
0.13407999 -0.57762003 -0.66526997 -0.42083001 0.65882999 -0.53825998
-0.50585997 0.51735002 0.25468999 -0.83724999 0.83806998 -0.42219999
1.0776 0.065962 0.48954999 -0.78614002 -0.19338 0.097524
0.27215999 0.037038 0.61778998 -0.29506999 -0.97285002 0.53106999
-0.32765001 -0.045966 -0.75436997 0.024904 0.64872003 0.023095
0.32062 0.35837999 -0.091125 -0.10866 0.33048001 -0.1162
-0.40981999 0.43928999 -0.16706 0.26047 0.090957 0.92714
0.099946 -0.29513001 -0.35341001 0.33693001 -0.42203999 -0.065625
0.54738998 -0.41751 -0.86232001 -0.65891999 -0.41549 0.067035
-0.41558 0.15092 0.17556 0.94068003 0.22527 0.65908003
0.15809 0.061199 0.63612998 -0.17089 -0.017591 -0.054003
-0.69072002 0.65179998]
[ 1.03529997e-01 7.20499977e-02 -2.93029994e-02 -4.46799994e-01
-8.61259997e-02 7.40030035e-02 -4.65499997e-01 -6.18570000e-02
-5.03650010e-01 1.95480004e-01 -1.03349999e-01 7.75929987e-01
-1.72040001e-01 -4.53520000e-01 2.63040006e-01 1.64340004e-01
2.80279994e-01 2.89330006e-01 3.10360014e-01 1.27750002e-02
6.79520011e-01 2.62770009e+00 3.89640003e-01 -4.49900001e-01
2.17969999e-01 9.16400030e-02 -1.67940006e-01 7.72420019e-02
2.91269988e-01 1.20530002e-01 -5.49659990e-02 9.16600004e-02
1.31300002e-01 -9.33270007e-02 -3.53009999e-01 -5.03880024e-01
-7.29799986e-01 -3.61380011e-01 -2.99659997e-01 2.07839999e-02
-7.03599975e-02 -6.72269985e-02 -3.62650007e-02 -1.46009997e-02
-5.98580018e-02 -1.63020000e-01 3.00660014e-01 -9.28120017e-02
-6.69799969e-02 1.43830001e-01 1.03950001e-01 -1.93039998e-02
-4.07020003e-01 8.86749983e-01 2.67349988e-01 -1.23829998e-01
2.73739994e-02 3.44639987e-01 6.04049981e-01 2.80640006e-01
1.41320005e-01 2.46429995e-01 -4.48760003e-01 5.42909980e-01
1.96759999e-01 -4.94709998e-01 2.68779993e-02 2.69100010e-01
2.53410012e-01 9.88679975e-02 1.77919999e-01 -3.22290003e-01
-1.05930001e-01 1.89520001e-01 -2.57629991e-01 2.43619993e-01
-2.45560005e-01 -1.36539996e-01 -1.08170003e-01 -4.21220005e-01
-1.61640003e-01 -3.41199994e-01 -1.47389993e-01 -2.16409996e-01
-5.62399998e-02 7.15879977e-01 6.40570000e-02 -3.07790011e-01
6.57369971e-01 -6.36910021e-01 2.64039993e-01 2.15059996e-01
1.83850005e-01 3.32610011e-01 -6.30220026e-02 -2.22450003e-01
6.31980002e-02 -4.47950006e-01 -2.52279997e-01 1.97699994e-01
-3.55479985e-01 1.19139999e-01 -2.99149990e-01 1.29390001e-01
-4.30770010e-01 -1.37700006e-01 2.38429993e-01 1.45529997e+00
-1.13710001e-01 -3.79790008e-01 -2.83129990e-01 -4.64819998e-01
-1.92760006e-01 1.98490005e-02 2.70090014e-01 -1.70849994e-01
-5.20099998e-02 -2.03219995e-01 -3.27439994e-01 -5.07570028e-01
2.98289992e-02 1.80350006e-01 3.05330008e-01 2.40700006e-01
4.66120005e-01 -7.12530017e-01 5.96719980e-01 2.13310003e-01
3.48639995e-01 3.80190015e-01 1.38260007e-01 -7.33380020e-02
2.26349995e-01 -4.55599993e-01 8.19810014e-03 3.47319990e-01
-1.36079997e-01 -6.59820020e-01 -2.79430002e-01 4.81799990e-02
-3.97129990e-02 2.86449999e-01 1.41259998e-01 -3.94299999e-02
1.44519997e+00 4.59950000e-01 7.92850032e-02 -3.55580002e-01
3.09360009e-02 -2.50809994e-02 5.64870015e-02 9.00759995e-02
5.20309992e-02 -3.99529994e-01 3.40330005e-01 -4.17400002e-01
-3.82189989e-01 2.22049996e-01 1.09520003e-01 -1.64539993e-01
1.20609999e-01 1.60919994e-01 -3.27459991e-01 2.45800003e-01
-2.28320006e-02 2.74109989e-01 -2.10689995e-02 3.91460001e-01
-5.61020017e-01 6.95510030e-01 -3.55260000e-02 4.71640006e-02
6.71909988e-01 4.81240004e-01 -2.78840009e-02 5.05490005e-01
-5.41859984e-01 -3.52369994e-01 -3.12009990e-01 -1.76020002e-03
9.59439993e-01 -5.03639996e-01 4.39889997e-01 4.71810013e-01
4.25799996e-01 -5.92209995e-01 -3.96219999e-01 1.89040005e-02
3.33819985e-02 -2.90149987e-01 -1.22079998e-01 2.76309997e-02
-2.66250014e-01 -1.97340008e-02 2.31020004e-01 8.76149982e-02
-7.69149978e-03 1.90050006e-02 -4.42119986e-01 -6.81999996e-02]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[ 0.041052 -0.54705 -0.72193998 -0.31235999 -0.43849 0.10691
-0.50641 -0.45401001 -0.28623 0.018973 0.020495 0.42860001
-0.0057162 -0.21272001 0.71864998 -0.091906 -0.55365002 0.39133
0.15351 -0.27454001 0.56528002 3.04830003 0.30467001 -0.37893
0.37865999 0.13925999 -0.11482 0.48212999 -0.30522999 0.43125001
-0.09667 0.069156 0.31426001 0.26350999 -0.31189999 -0.39881
-0.55656999 -0.35934001 -0.25402001 0.072061 -0.12966999 -0.11247
-0.041192 -0.042619 -0.07848 0.31992 0.41635999 0.26131001
-0.18175 -0.1279 0.21332 -0.41973001 -0.50444001 0.37705001
0.83955002 -0.34571001 -0.43000999 -0.18653999 -0.061082 -0.087612
0.092833 0.52604997 -0.57611001 -0.19328 0.20576 0.24607
0.1631 -0.18138 0.032592 0.19169 0.73565 -0.25718999
0.30072999 0.56699002 -0.21544001 0.18933 -0.12287 -0.65759999
0.021702 0.1041 0.098952 -0.43171999 -0.27517 -0.15448
0.31301001 -0.032041 -0.090526 0.14489999 0.68151999 -0.88242
0.30816999 -0.62702 0.12274 0.014773 -0.16887 0.56159002
0.022004 0.52085 -0.22685 0.09633 0.26956999 0.30489001
0.018463 0.31009001 0.04198 0.32381999 0.13153 0.89722002
0.15475 -0.38806999 -0.52993 -0.35383999 -0.0913 0.57230002
0.48067001 0.24438 0.074138 -0.019957 -0.35030001 -0.034695
-0.12045 0.39998999 -0.37015 -0.53040999 0.10655 -0.44973001
0.43105 -0.44937 0.48675999 0.43836999 0.043421 0.52675003
0.61176002 0.26704001 0.59239 0.23650999 0.12841 -0.10665
-0.46715 -0.039081 -0.24921 0.030486 0.092933 -0.04841
1.83580005 0.077535 -0.11494 -0.13668001 -0.23983 0.31948
0.19205999 0.38894001 0.34755 -0.038804 0.19541 -0.37099999
-0.027576 -0.24127001 -0.16868 0.032815 0.08139 0.054121
-0.42785999 0.26447001 0.054847 -0.21765999 0.015181 0.57656002
0.24033 0.62076002 -0.019055 -0.31509 0.76424998 0.35168999
-0.28512001 0.15175 0.11238 -0.60829997 0.35087001 0.19140001
0.51753998 0.20893 -0.63814002 0.19403 0.24493 0.46606001
-0.32235 0.37286001 -0.19508 0.13237999 -0.35420999 0.22849
0.36032 -0.0050241 -0.051955 -0.37755999 -0.087065 0.3592
0.11564 0.44372001]
[ 4.71520007e-01 -5.87790012e-01 -6.76150024e-01 -4.67000008e-01
1.11709997e-01 3.26370001e-01 -4.65070009e-01 -8.05180013e-01
-1.65340006e-01 -1.13849998e-01 -1.36849999e-01 2.56980002e-01
3.36279988e-01 1.90149993e-01 1.32440001e-01 2.37419993e-01
-1.46239996e+00 8.59059989e-01 5.53650022e-01 1.94330007e-01
2.73930013e-01 1.05669999e+00 6.24970019e-01 -4.30469990e-01
7.14770019e-01 -4.73030001e-01 -8.97960007e-01 2.56910007e-02
-6.42499983e-01 2.15990007e-01 -1.22769997e-01 -5.36949992e-01
5.91489971e-01 6.28649965e-02 1.51260002e-02 -6.57150000e-02
1.61709994e-01 -8.86740014e-02 -7.09370002e-02 6.12349987e-01
1.38689995e-01 -3.67980003e-01 -9.46219981e-01 1.30669996e-01
-2.82139987e-01 -3.02709997e-01 4.05889988e-01 -2.11899996e-01
1.74940005e-01 2.38450006e-01 3.41769993e-01 4.50269997e-01
-7.82140017e-01 1.64210007e-01 7.19319999e-01 -6.80140018e-01
-4.93660003e-01 3.67380008e-02 2.62410015e-01 -8.48299980e-01
-6.59759998e-01 4.04370010e-01 -2.61209998e-02 5.83829999e-01
-3.28000009e-01 6.39530003e-01 1.20350003e-01 7.21519988e-04
8.28130007e-01 -3.83879989e-01 5.35929978e-01 -4.59630013e-01
-5.12839973e-01 1.74339995e-01 2.06220001e-01 -8.01329970e-01
-4.74339992e-01 -4.32810009e-01 -6.11400008e-01 1.71409994e-01
5.54369986e-01 6.11240007e-02 6.43959999e-01 -5.23599982e-01
1.35130000e+00 -1.40279993e-01 3.67210001e-01 -3.68629992e-01
7.95690000e-01 -1.01139998e+00 -1.47060007e-01 4.48889993e-02
4.06240001e-02 5.33150017e-01 3.40680003e-01 2.50710011e-01
-1.26489997e-01 -1.15050003e-01 -1.48660004e-01 7.65860021e-01
9.80419964e-02 6.28759980e-01 -4.09599990e-01 -3.33020017e-02
-3.77389997e-01 -2.54250001e-02 -8.92150030e-02 1.63540006e+00
5.04270017e-01 -3.86139989e-01 -1.25259995e-01 1.65910006e-01
-2.19990000e-01 -6.84350014e-01 -2.99389988e-01 -1.25550002e-01
3.63970011e-01 3.89310002e-01 -9.63360012e-01 -7.42670000e-02
3.48879993e-01 2.36190006e-01 -8.27549994e-01 3.19779992e-01
-3.00359994e-01 -4.01740015e-01 7.04670012e-01 -3.32989991e-01
-1.26369998e-01 -3.72830003e-01 -9.09730017e-01 -1.33279994e-01
-5.32779992e-02 -3.47909987e-01 2.48980001e-01 -4.34430003e-01
-2.42050007e-01 -6.21890008e-01 -1.27769995e+00 -9.66310024e-01
4.20859993e-01 -4.12339985e-01 1.20589995e+00 -2.55379993e-02
1.01170003e+00 -1.11219997e-03 5.92599988e-01 -5.58459997e-01
-3.69489998e-01 5.58549985e-02 -6.60969973e-01 3.91559988e-01
3.09260011e-01 -6.28539994e-02 -1.14900005e+00 4.69440013e-01
-5.05439997e-01 -1.75500005e-01 -3.80259991e-01 -1.97349995e-01
7.31180012e-01 -1.77149996e-01 4.18940008e-01 4.23940003e-01
-5.16149998e-02 -1.20180003e-01 3.96189988e-01 -2.36780003e-01
1.19029999e-01 2.93020010e-01 -1.81150004e-01 4.59089994e-01
1.93489999e-01 4.75919992e-01 -3.67170006e-01 -8.76540027e-04
-6.15379997e-02 -4.12099995e-02 -1.95240006e-01 -6.47260016e-03
4.30709988e-01 6.84989989e-02 -4.27579999e-01 -3.15990001e-01
-2.59119987e-01 -6.58209980e-01 -1.92790002e-01 5.57280004e-01
-2.24089995e-01 2.21340001e-01 4.68760014e-01 6.37290001e-01
6.58949971e-01 8.79120007e-02 -3.06369990e-01 -3.22079986e-01
6.47899985e-01 1.75490007e-01 -5.78580022e-01 8.94800007e-01]
===================================
===== For the word2vec
===================================
1. Download and unzip the pre-trained model of GoogleNews-vectors-negative300.bin.gz.
2. Install the gensim tools:
sudo pip install --upgrade gensim
3. Code for vector extraction from given sentence.
import gensim
print("==>> loading the pre-trained word2vec model: GoogleNews-vectors-negative300.bin")
dictFileName = './GoogleNews-vectors-negative300.bin'
wv = gensim.models.KeyedVectors.load_word2vec_format(dictFileName, binary=True)
The Output is:
==>> loading the pre-trained word2vec model: GoogleNews-vectors-negative300.bin
INFO:gensim.models.utils_any2vec:loading projection weights from ./GoogleNews-vectors-negative300.bin
INFO:gensim.models.utils_any2vec:loaded (3000000, 300) matrix from ./GoogleNews-vectors-negative300.bin
INFO:root:Data statistic
INFO:root:train_labels:19337
INFO:root:test_labels:20632
INFO:root:train_sentences:19337
INFO:root:dev_sentences:2000
INFO:root:test_sentences:20632
INFO:root:dev_labels:2000
embed_size:300
vocab_size:3000000
vocab_path = ''data/bilstm.vocab''
index_to_word = [key for key in wv.vocab]
word_to_index = {}
for index, word in enumerate(index_to_word):
word_to_index[word] = index
with open(vocab_path, "w") as f:
f.write(json.dumps(word_to_index))
Let's take a deep understanding on the bidirectional-LSTM-for-text-classification-master
class BiLSTM(nn.Module):
def __init__(self, embedding_matrix, hidden_size=150, num_layer=2, embedding_freeze=False):
super(BiLSTM,self).__init__() # embedding layer
vocab_size = embedding_matrix.shape[0]
embed_size = embedding_matrix.shape[1]
self.hidden_size = hidden_size
self.num_layer = num_layer
self.embed = nn.Embedding(vocab_size, embed_size)
self.embed.weight = nn.Parameter(torch.from_numpy(embedding_matrix).type(torch.FloatTensor), requires_grad=not embedding_freeze)
self.embed_dropout = nn.Dropout(p=0.3)
self.custom_params = []
if embedding_freeze == False:
self.custom_params.append(self.embed.weight) # The first LSTM layer
self.lstm1 = nn.LSTM(embed_size, self.hidden_size, num_layer, dropout=0.3, bidirectional=True)
for param in self.lstm1.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param) self.lstm1_dropout = nn.Dropout(p=0.3) # The second LSTM layer
self.lstm2 = nn.LSTM(2*self.hidden_size, self.hidden_size, num_layer, dropout=0.3, bidirectional=True)
for param in self.lstm2.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param)
self.lstm2_dropout = nn.Dropout(p=0.3) # Attention
self.attention = nn.Linear(2*self.hidden_size,1)
self.attention_dropout = nn.Dropout(p=0.5) # Fully-connected layer
self.fc = weight_norm(nn.Linear(2*self.hidden_size,3))
for param in self.fc.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param) self.hidden1=self.init_hidden()
self.hidden2=self.init_hidden() def init_hidden(self, batch_size=3):
if torch.cuda.is_available():
return (Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)).cuda(),
Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)).cuda())
else:
return (Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)),
Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size))) def forward(self, sentences): print("==>> sentences: ", sentences) # get embedding vectors of input
padded_sentences, lengths = torch.nn.utils.rnn.pad_packed_sequence(sentences, padding_value=int(0), batch_first=True)
print("==>> padded_sentences: ", padded_sentences) embeds = self.embed(padded_sentences)
print("==>> embeds: ", embeds) # pdb.set_trace() noise = Variable(torch.zeros(embeds.shape).cuda())
noise.data.normal_(std=0.3)
embeds += noise
embeds = self.embed_dropout(embeds)
# add noise packed_embeds = torch.nn.utils.rnn.pack_padded_sequence(embeds, lengths, batch_first=True) print("==>> packed_embeds: ", packed_embeds) # First LSTM layer
# self.hidden = num_layers*num_directions batch_size hidden_size
packed_out_lstm1, self.hidden1 = self.lstm1(packed_embeds, self.hidden1)
padded_out_lstm1, lengths = torch.nn.utils.rnn.pad_packed_sequence(packed_out_lstm1, padding_value=int(0))
padded_out_lstm1 = self.lstm1_dropout(padded_out_lstm1)
packed_out_lstm1 = torch.nn.utils.rnn.pack_padded_sequence(padded_out_lstm1, lengths) pdb.set_trace() # Second LSTM layer
packed_out_lstm2, self.hidden2 = self.lstm2(packed_out_lstm1, self.hidden2)
padded_out_lstm2, lengths = torch.nn.utils.rnn.pad_packed_sequence(packed_out_lstm2, padding_value=int(0), batch_first=True)
padded_out_lstm2 = self.lstm2_dropout(padded_out_lstm2) # attention
unnormalize_weight = F.tanh(torch.squeeze(self.attention(padded_out_lstm2), 2))
unnormalize_weight = F.softmax(unnormalize_weight, dim=1)
unnormalize_weight = torch.nn.utils.rnn.pack_padded_sequence(unnormalize_weight, lengths, batch_first=True)
unnormalize_weight, lengths = torch.nn.utils.rnn.pad_packed_sequence(unnormalize_weight, padding_value=0.0, batch_first=True)
logging.debug("unnormalize_weight size: %s" % (str(unnormalize_weight.size())))
normalize_weight = torch.nn.functional.normalize(unnormalize_weight, p=1, dim=1)
normalize_weight = normalize_weight.view(normalize_weight.size(0), 1, -1)
weighted_sum = torch.squeeze(normalize_weight.bmm(padded_out_lstm2), 1) # fully connected layer
output = self.fc(self.attention_dropout(weighted_sum))
return output
==>> Some Testing:
(a). len(wv.vocab) = 300,0000
(b). wv.vocab is what ? Something like this:
{ ... , u'fivemonth': <gensim.models.keyedvectors.Vocab object at 0x7f90945bd810>,
u'retractable_roofs_Indians': <gensim.models.keyedvectors.Vocab object at 0x7f90785f5690>,
u'Dac_Lac_province': <gensim.models.keyedvectors.Vocab object at 0x7f908d8eda10>,
u'Kenneth_Klinge': <gensim.models.keyedvectors.Vocab object at 0x7f9081563410>}
(c). index_to_word: count the words from the pre-trained model.
{ ... , u"Lina'la_Sin_Casino", u'fivemonth', u'retractable_roofs_Indians', u'Dac_Lac_province', u'Kenneth_Klinge'}
(d). word_to_index: give each word a index as following
{ ... , u'fivemonth': 2999996, u'Pidduck': 2999978, u'Dac_Lac_province': 2999998, u'Kenneth_Klinge': 2999999 }
(e). dataset is:
[ ... , [1837614, 1569052, 1837614, 1288695, 2221039, 2323218, 1837614, 1837614, 2029395, 1612781, 311032, 1524921, 1837614, 2973515, 2033866, 882731, 2462275, 2809106, 1479961, 826019, 73590, 953550, 1837614],
[1524921, 1113778, 1837614, 318169, 1837614, 1954969, 196613, 943118, 1837614, 2687790, 291413, 2774825, 2366038, 296869, 1468080, 856987, 1802099, 724308, 1207907, 2264894, 2206446, 812434],
[564298, 477983, 1837614, 1449153, 1837614, 211925, 2206446, 481834, 488597, 280760, 2072822, 1344872, 1791678, 2458776, 2965810, 2168205, 387112, 2656471, 1391, 1837614, 1801696, 2093846, 1210651],
[2493381, 133883, 2441902, 1014220, 1837614, 2597880, 1756105, 2651537, 1391, 2641114, 2517536, 1109601, 122269, 1782479, 2965835, 488597, 1767716, 753333, 564298, 2380935, 228060, 1837614, 371618],
[1837614, 1344872, 2458776, 2965810, 1837614, 2015408, 1837614, 528014, 1991322, 1837614, 908982, 1484130, 2349526, 988689, 753336, 1837614, 364492, 2183116, 826019, 73590, 953550, 1837614],
[1837614, 2673785, 1990947, 1219831, 2635341, 1247040, 1837614, 799543, 1990947, 1219831, 2722301, 1837614, 1427513, 969099, 2157673, 1430111, 1837614]]
(f). the variable during the training process:
# ('==>> sentences: ', PackedSequence(data=tensor(
# [ 4.0230e+05, 1.8376e+06, 2.0185e+06, 1.8376e+06, 2.8157e+06,
# 1.8376e+06, 1.8376e+06, 1.0394e+06, 1.8376e+06, 2.9841e+06,
# 4.4713e+05, 1.1352e+06, 2.3532e+06, 1.8376e+06, 1.8376e+06,
# 1.8376e+06, 1.9550e+06, 3.8429e+04, 6.2537e+05, 2.3764e+05,
# 1.8376e+06, 1.5428e+06, 1.4214e+06], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])))
# ('==>> padded_sentences: ', tensor(
# [[ 4.0230e+05, 1.8376e+06, 2.0185e+06, 1.8376e+06, 2.8157e+06,
# 1.8376e+06, 1.8376e+06, 1.0394e+06, 1.8376e+06, 2.9841e+06,
# 4.4713e+05, 1.1352e+06, 2.3532e+06, 1.8376e+06, 1.8376e+06,
# 1.8376e+06, 1.9550e+06, 3.8429e+04, 6.2537e+05, 2.3764e+05,
# 1.8376e+06, 1.5428e+06, 1.4214e+06]], device='cuda:0'))
# length: 23
# ('==>> embeds: ', tensor([[
# [-0.0684, 0.1826, -0.1777, ..., 0.1904, -0.1021, 0.1729],
# [ 0.0801, 0.1050, 0.0498, ..., 0.0037, 0.0476, -0.0688],
# [-0.1982, -0.0693, 0.1230, ..., -0.1357, -0.0306, 0.1104],
# ...,
# [ 0.0801, 0.1050, 0.0498, ..., 0.0037, 0.0476, -0.0688],
# [-0.0518, -0.0299, 0.0415, ..., 0.0776, -0.1660, 0.1602],
# [-0.0532, -0.0004, 0.0337, ..., -0.2373, -0.1709, 0.0233]]], device='cuda:0'))
# ('==>> packed_embeds: ', PackedSequence(data=tensor([
# [-0.3647, 0.2966, -0.2359, ..., -0.0000, 0.2657, -0.4302],
# [ 1.1699, 0.0000, 0.3312, ..., 0.5714, 0.1930, -0.2267],
# [-0.0627, -1.0548, 0.4966, ..., -0.5135, -0.0150, -0.0000],
# ...,
# [-0.6065, -0.7562, 0.3320, ..., -0.5854, -0.2089, -0.5737],
# [-0.0000, 0.4390, 0.0000, ..., 0.6891, 0.0250, -0.0000],
# [ 0.6909, -0.0000, -0.0000, ..., 0.1867, 0.0594, -0.2385]], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])))
# packed_out_lstm1
# PackedSequence(data=tensor([
# [-0.0000, -0.0000, 0.1574, ..., 0.1864, -0.0901, -0.3205],
# [-0.0000, -0.3490, 0.1774, ..., 0.1677, -0.0000, -0.3688],
# [-0.3055, -0.0000, 0.2240, ..., 0.0000, -0.0927, -0.0000],
# ...,
# [-0.3188, -0.4134, 0.1339, ..., 0.3161, -0.0000, -0.3846],
# [-0.3355, -0.4365, 0.1575, ..., 0.2775, -0.0886, -0.4015],
# [-0.0000, -0.0000, 0.2452, ..., 0.1763, -0.0000, -0.2748]], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]))
## self.hidden1
# self.hidden1
# (tensor([
# [[-0.0550, -0.4266, 0.0498, 0.0750, 0.3122, 0.0499, -0.2899,
# -0.0980, -0.1776, 0.4376, 0.1555, 0.7167, -0.0162, 0.0081,
# -0.3512, 0.0250, 0.0948, -0.3596, 0.0670, -0.6623, 0.0026,
# -0.0262, 0.1934, 0.3206, -0.1941, 0.3229, -0.6488, -0.3858,
# 0.3169, -0.0216, -0.1136, 0.1411, -0.0296, -0.2860, -0.1277,
# -0.0154, 0.2620, -0.3553, 0.0637, 0.8245, 0.6886, -0.0048,
# -0.1717, 0.2495, -0.8636, -0.0217, -0.2365, 0.1324, 0.1790,
# -0.1515, 0.3530, -0.1644, 0.0073, 0.6709, 0.1577, -0.0190,
# 0.0384, 0.4871, -0.7375, -0.1804, 0.3034, -0.3516, -0.2870,
# 0.6387, 0.0414, 0.6983, -0.3211, 0.0449, 0.2127, -0.0421,
# -0.3454, -0.8145, -0.3629, -0.0828, -0.1558, 0.4048, -0.4971,
# -0.7495, 0.0622, -0.3318, 0.3913, -0.0322, -0.0678, 0.0307,
# -0.0153, -0.1535, -0.2719, 0.0128, -0.1521, -0.2924, 0.7109,
# -0.8551, 0.0330, 0.0482, 0.2410, -0.0655, -0.2496, 0.1816,
# 0.4963, -0.7593, -0.0022, -0.1122, 0.6857, -0.5693, 0.5805,
# 0.7660, 0.4430, 0.0243, -0.0313, 0.0780, -0.2419, 0.0745,
# -0.0119, 0.5761, -0.0285, -0.1085, -0.1783, -0.0706, 0.1243,
# 0.6333, 0.0296, 0.5557, 0.2717, -0.0071, 0.0503, -0.0405,
# -0.4542, 0.8905, 0.4492, 0.8809, 0.7021, 0.8616, 0.2825,
# -0.2114, 0.3026, -0.1384, 0.1252, 0.4989, 0.2236, -0.5374,
# -0.1352, -0.0561, 0.0378, -0.5291, -0.1004, -0.3723, 0.0836,
# 0.3500, 0.0542, 0.2013]], # [[-0.0041, -0.0040, -0.1428, 0.2783, -0.1378, -0.4242, 0.1000,
# 0.1641, -0.0175, -0.0896, 0.3241, 0.3513, -0.4675, -0.1250,
# 0.0546, 0.2400, -0.0997, -0.5614, 0.2026, -0.1505, 0.0833,
# -0.3128, -0.4646, -0.0778, 0.2204, 0.5597, -0.7004, 0.0419,
# -0.3699, -0.5748, 0.7741, -0.7220, 0.0494, 0.3430, 0.1389,
# 0.0178, 0.0136, 0.0273, 0.1559, 0.4333, 0.2411, 0.0804,
# -0.0202, 0.6204, -0.4104, 0.5382, 0.1804, 0.0179, 0.1118,
# 0.3084, 0.1894, 0.0092, -0.2182, 0.0022, 0.4377, -0.0575,
# 0.0906, -0.0531, 0.0936, -0.1203, -0.0836, -0.1718, 0.2059,
# 0.7192, 0.0833, 0.3712, 0.2354, -0.1141, -0.0993, -0.0433,
# -0.1474, -0.0078, 0.0834, 0.0864, 0.9305, -0.3387, 0.0818,
# 0.3775, 0.1609, -0.0246, 0.2563, -0.1253, -0.0897, 0.0322,
# -0.0848, 0.0673, -0.1051, 0.0205, 0.0183, -0.0007, 0.1229,
# -0.3388, -0.0948, 0.0335, 0.0450, -0.2747, 0.2763, -0.2691,
# 0.6240, -0.0018, 0.2048, 0.3943, 0.2015, -0.5962, -0.0069,
# -0.3460, -0.7910, 0.3002, -0.4653, -0.0611, 0.6912, -0.8154,
# -0.0443, -0.0189, -0.1265, -0.1202, 0.0013, -0.5983, 0.0879,
# 0.0752, 0.8593, 0.0357, 0.0953, -0.0525, 0.2069, -0.6292,
# -0.0456, -0.7646, 0.0166, 0.0584, 0.0142, 0.0575, -0.1658,
# -0.4304, 0.3228, 0.4094, 0.0149, 0.1478, 0.1447, 0.4192,
# -0.0783, -0.0683, 0.0259, 0.0665, 0.6224, 0.3775, 0.0247,
# 0.1710, -0.3622, 0.0931]], # [[-0.2315, -0.2179, 0.1716, -0.6143, 0.4329, 0.2288, 0.1208,
# -0.3435, 0.6314, 0.0106, 0.1470, -0.0915, -0.3019, 0.1302,
# 0.2325, 0.1794, 0.0145, -0.6598, 0.0062, -0.0743, 0.0232,
# 0.9310, -0.8155, -0.0449, -0.5504, 0.5746, 0.3607, 0.4053,
# -0.1887, -0.5448, 0.1522, -0.0189, -0.4852, 0.6322, 0.0011,
# -0.1590, -0.0054, 0.2972, -0.0270, 0.0047, 0.2944, -0.0629,
# -0.1138, 0.1349, 0.5449, 0.8018, -0.0218, 0.0523, 0.3262,
# -0.0506, 0.2821, -0.0661, -0.7165, -0.3653, 0.3321, -0.0255,
# -0.0551, -0.1826, 0.6027, -0.1995, 0.0598, 0.0205, -0.1769,
# -0.0789, 0.4500, -0.1641, 0.5002, -0.0716, -0.3708, -0.0020,
# -0.5195, -0.0896, 0.1421, 0.1149, 0.3407, 0.2649, 0.0858,
# 0.2778, -0.3768, 0.6176, -0.2148, 0.5444, 0.3009, 0.4848,
# -0.1174, 0.0019, 0.6213, 0.2524, -0.0816, 0.4639, 0.4747,
# -0.7812, 0.2435, -0.0867, 0.1617, 0.2194, 0.0426, 0.1393,
# -0.0448, 0.0506, -0.5524, 0.0707, 0.2226, -0.0337, 0.7445,
# -0.4516, 0.1107, -0.2617, -0.1914, 0.7238, -0.2689, 0.0110,
# -0.3139, -0.0027, -0.5964, -0.9012, -0.4319, 0.0112, -0.0306,
# 0.4002, -0.1117, -0.1021, 0.1652, -0.2872, 0.3640, 0.2162,
# -0.3843, -0.0869, -0.1623, 0.0297, -0.0048, -0.0735, -0.0886,
# -0.4138, 0.2325, -0.4248, 0.3354, 0.0712, -0.4079, 0.0821,
# 0.1413, 0.2241, -0.1938, -0.0807, 0.3551, -0.0814, 0.1438,
# -0.6870, -0.3647, 0.0276]], # [[ 0.0258, 0.3281, -0.8145, -0.0476, -0.2886, -0.8013, 0.2135,
# 0.1541, 0.2069, 0.1345, -0.0171, -0.0228, -0.5237, 0.4917,
# 0.5187, 0.1402, 0.0928, -0.0373, 0.2698, 0.1259, 0.0021,
# -0.1624, -0.4100, 0.5377, -0.1013, -0.5658, -0.1015, 0.5609,
# 0.1661, 0.5731, 0.0012, -0.1766, 0.0743, -0.3630, 0.1082,
# 0.4643, 0.0175, -0.0260, 0.3810, -0.6425, -0.5515, 0.8800,
# -0.1158, -0.5741, 0.0463, 0.4033, 0.0803, 0.0403, 0.1159,
# 0.4471, 0.0294, 0.2899, 0.0248, -0.1772, 0.6600, -0.2252,
# -0.4896, -0.1285, -0.2377, -0.4179, -0.4056, -0.3224, -0.6855,
# -0.2703, 0.2971, 0.1259, -0.0456, -0.2495, 0.8141, 0.4453,
# 0.7480, -0.0578, 0.8023, -0.3586, -0.5229, 0.2299, 0.9668,
# -0.0717, 0.5355, -0.0743, 0.5246, 0.1604, -0.1464, -0.0757,
# 0.0414, -0.0861, 0.2245, 0.1247, 0.0676, -0.2053, 0.0113,
# 0.7875, -0.0308, 0.2025, 0.1289, -0.0020, -0.3099, 0.5317,
# -0.0117, 0.0928, -0.4100, -0.6184, 0.1171, 0.0216, -0.1266,
# 0.1640, 0.0821, -0.4097, -0.0691, 0.5805, 0.1692, -0.2021,
# 0.5971, 0.1172, -0.6535, -0.0579, 0.1177, 0.1123, -0.1943,
# 0.0488, -0.1305, -0.4859, -0.2758, -0.2972, -0.0605, -0.0029,
# -0.1508, 0.0375, -0.5942, -0.2139, -0.0335, -0.2320, -0.1152,
# -0.2054, -0.2643, -0.1770, 0.1245, 0.6334, -0.0363, 0.0264,
# -0.3348, -0.0434, -0.3794, -0.0913, 0.1293, -0.6537, 0.6490,
# 0.1305, -0.0631, -0.2243]]], device='cuda:0'),
# tensor([[[ -0.0775, -4.6346, 7.9077, 0.1164, 0.8626, 0.4240,
# -0.9286, -0.1612, -0.6049, 0.6771, 0.7443, 1.7457,
# -0.3930, 0.0112, -13.5393, 0.0317, 0.1236, -0.8475,
# 0.1212, -1.3623, 0.0117, -0.3297, 0.9009, 0.3840,
# -0.5885, 0.7411, -8.9613, -1.1402, 0.4511, -0.0753,
# -0.3107, 1.6518, -0.0870, -3.1360, -12.0200, -0.0464,
# 0.2756, -0.6695, 0.1604, 4.8299, 6.4623, -0.9555,
# -0.6904, 0.4469, -11.3343, -0.1669, -0.2747, 0.1590,
# 0.5829, -0.3345, 2.1731, -0.5636, 0.0207, 0.9874,
# 0.6291, -1.2261, 0.1946, 1.1287, -1.5759, -0.1875,
# 0.5550, -1.7350, -0.8235, 1.5122, 0.2019, 5.5143,
# -3.8153, 0.6771, 0.3011, -0.2994, -0.7320, -1.5857,
# -0.4785, -0.5584, -0.3226, 1.1932, -4.3901, -1.6923,
# 0.3526, -1.0625, 0.9279, -0.1843, -0.4376, 2.2389,
# -0.1558, -0.3959, -1.2987, 0.0279, -0.4938, -0.3364,
# 3.2596, -2.2647, 0.1448, 0.0726, 0.3968, -0.1885,
# -0.3960, 0.2141, 0.6785, -2.1622, -0.0043, -0.7516,
# 0.9367, -0.7092, 4.1853, 1.3348, 0.6993, 0.2043,
# -0.0916, 0.1392, -1.5672, 0.0867, -0.0346, 0.9226,
# -0.0470, -0.6870, -0.3002, -0.1131, 0.7785, 1.0582,
# 0.0914, 2.8785, 0.8164, -0.1048, 0.0573, -0.0499,
# -0.5990, 3.1714, 0.7925, 1.7461, 1.3243, 2.9236,
# 0.8966, -0.4455, 0.8763, -0.3036, 0.3302, 2.6581,
# 0.4608, -0.7280, -2.9457, -0.1973, 0.0585, -0.6555,
# -0.6621, -0.4549, 0.5812, 0.4495, 0.1350, 1.8521]], # [[ -0.0053, -0.0843, -0.1763, 0.3929, -0.1668, -0.6609,
# 0.1269, 0.2214, -0.0208, -0.3571, 0.7532, 0.7496,
# -4.9288, -0.5457, 0.3557, 0.4795, -0.2318, -0.9659,
# 0.6826, -1.6542, 0.4917, -0.3956, -1.5164, -0.2274,
# 0.6779, 1.1201, -3.1397, 0.0434, -0.4993, -0.8809,
# 6.1257, -5.6283, 0.4273, 1.5070, 0.6624, 0.1289,
# 0.2180, 0.9920, 0.1646, 0.8828, 0.5732, 0.3255,
# -0.0679, 0.9843, -1.8408, 1.0547, 0.1959, 0.0748,
# 0.1907, 0.4751, 0.3174, 0.0747, -0.6487, 0.0377,
# 0.5554, -0.4095, 0.2593, -0.0568, 0.3751, -0.3646,
# -0.2031, -0.3284, 0.4058, 1.2788, 0.1348, 1.8184,
# 0.8482, -0.7494, -0.2395, -0.4352, -0.1584, -0.0105,
# 0.2676, 0.3763, 2.1413, -1.3001, 0.3923, 1.6432,
# 0.2987, -1.2708, 5.5667, -0.1727, -0.5106, 0.5180,
# -6.7258, 0.5001, -0.3052, 0.0843, 0.0474, -0.2306,
# 0.1908, -2.2523, -1.5432, 0.2809, 0.7099, -0.4145,
# 0.7393, -0.6529, 0.7825, -0.0019, 1.3337, 2.2042,
# 9.2887, -0.8515, -0.0610, -0.6146, -1.5616, 0.3592,
# -1.3585, -0.2641, 1.4763, -3.2525, -0.5447, -0.0453,
# -1.0416, -2.4657, 0.0556, -0.7654, 0.2062, 0.0855,
# 3.0740, 0.0952, 0.4923, -0.1772, 0.9173, -4.2004,
# -0.1298, -2.4266, 0.0181, 0.5039, 0.0399, 7.9909,
# -0.5778, -2.9112, 0.4854, 1.2364, 0.0686, 0.6365,
# 0.1869, 0.6050, -0.1246, -0.1848, 0.5406, 0.2110,
# 1.2367, 1.9466, 0.0302, 0.2002, -0.5902, 0.1069]], # [[ -0.3559, -0.2859, 0.2699, -1.4359, 0.9814, 0.2811,
# 0.8539, -2.6654, 2.5455, 0.0434, 0.5947, -0.5325,
# -0.4638, 0.5487, 1.4376, 0.9863, 0.7429, -1.8308,
# 0.0402, -0.2282, 0.0366, 12.7877, -1.2491, -0.1437,
# -1.4960, 0.7364, 0.8599, 1.8343, -3.5117, -1.2758,
# 0.2930, -0.0472, -0.7527, 0.9555, 0.0446, -0.3389,
# -0.1985, 1.7953, -0.5702, 0.0141, 0.6166, -0.0924,
# -0.5182, 0.5146, 2.0801, 2.7460, -0.2606, 0.2090,
# 0.9266, -0.4758, 0.9961, -0.1723, -1.2069, -1.1735,
# 0.3683, -0.4933, -0.0604, -0.2354, 0.8239, -5.4226,
# 0.0854, 0.1185, -0.2656, -0.2689, 0.6047, -0.6246,
# 1.0131, -0.1673, -0.4990, -0.0690, -0.6092, -0.5205,
# 0.1808, 0.3061, 0.3924, 0.5868, 0.1452, 2.8930,
# -0.6085, 1.6086, -0.4763, 5.0389, 1.1569, 3.4060,
# -0.7565, 0.0247, 0.8477, 0.3714, -0.1043, 1.5607,
# 4.0700, -1.8363, 0.4370, -0.3571, 0.7268, 0.3435,
# 0.0972, 7.1477, -0.1486, 0.3342, -0.9733, 0.2311,
# 0.6104, -0.4988, 2.8838, -1.3387, 1.4291, -0.4121,
# -0.6722, 2.6834, -0.5188, 0.0428, -0.3452, -0.0131,
# -0.9004, -3.0346, -0.9254, 0.0150, -0.0386, 1.0639,
# -0.2444, -0.4562, 4.1626, -1.9304, 1.0662, 2.0683,
# -0.9553, -0.6434, -1.6777, 0.0702, -0.0113, -0.5503,
# -0.1873, -0.6916, 1.0729, -0.8234, 0.6421, 0.3022,
# -0.6065, 0.1016, 0.7792, 0.2533, -0.2670, -0.1314,
# 0.7515, -0.9859, 0.5050, -1.0552, -0.5632, 1.0697]], # [[ 0.3612, 2.4727, -4.6103, -1.6459, -1.7761, -1.4302,
# 0.2737, 0.3302, 4.0617, 0.7206, -0.0749, -0.8146,
# -1.0134, 0.8741, 1.9300, 0.5426, 0.1386, -1.3920,
# 0.4602, 1.3387, 0.0068, -0.3648, -7.6665, 0.9011,
# -0.3286, -1.5220, -0.2155, 0.7959, 4.0746, 0.9382,
# 0.0023, -0.2666, 0.4571, -1.9530, 0.3216, 2.1178,
# 0.4043, -0.0309, 2.5116, -1.2250, -0.9842, 5.0822,
# -4.1296, -5.3579, 0.9115, 0.4843, 0.1365, 0.0491,
# 0.1446, 0.6523, 0.0765, 0.3761, 0.0310, -0.4825,
# 7.1485, -0.4211, -3.7914, -0.2492, -0.3775, -0.4745,
# -1.3320, -1.8203, -1.0266, -0.4446, 2.2385, 0.6003,
# -0.1759, -1.9601, 2.3865, 1.3325, 4.8762, -0.2398,
# 7.5251, -0.4380, -2.3422, 0.6013, 13.8362, -0.4112,
# 2.3579, -0.1720, 1.0265, 0.6521, -0.7363, -0.7864,
# 0.2986, -0.1298, 0.5078, 0.1386, 1.4856, -0.3133,
# 0.9933, 1.5144, -0.0433, 1.0841, 0.3962, -0.0024,
# -0.3937, 2.2719, -0.0198, 1.6771, -1.2469, -0.8017,
# 0.1607, 0.0244, -0.1429, 0.9912, 0.1635, -1.2396,
# -0.1615, 1.0921, 0.8146, -0.3309, 0.8553, 0.4243,
# -3.5547, -0.1382, 0.1513, 0.4036, -0.2505, 0.2295,
# -0.6219, -0.7644, -0.7568, -0.4494, -0.0775, -0.0178,
# -0.2550, 0.2258, -2.7895, -0.3362, -0.2364, -0.9864,
# -1.3459, -1.8118, -0.4397, -0.8312, 0.3526, 1.3541,
# -0.0467, 1.6161, -0.4478, -0.5202, -0.4164, -0.8265,
# 0.1626, -4.2044, 3.2649, 0.2940, -0.8260, -0.4956]]], device='cuda:0'))
######## Padding functions used in pytorch. #########
1. torch.nn.utils.rnn.PackedSequence(*args)
Holds the data and list of batch_sizes of a packed sequence.
All RNN modules accept packed sequences as inputs.
2. torch.nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first=False)
Packs a Tensor containing padded sequences of variable length.
Input can be of size T*B** where T is the length of the longest sequence, B is the batch size, and the * is any number of dimensions.
If batch_first is True B*T** inputs are expected. The sequences should be sorted by length in a decreasing order.
3. torch.nn.utils.rnn.pad_packed_sequence(sequence, batch_first=False, padding_values=0.0, total_length=None)
Pads a packed batch of variable length sequences.
It is an inverse operation to pack_padded_sequence().
Batch elements will be ordered decreasingly by their length.
Note: total_length is useful to implement the pack sequence -> rnn -> unpack sequence .
Return: Tuple of tensor containing the padded sequence, and a tensor containing the list of lengths of each sequence in the batch.
4. torch.nn.utils.rnn.pad_sequence(sequence, batch_first=False, padding_value=0)
Pad a list of variable length Tensors with zero.
5. torch.nn.utils.rnn.pack_sequence(sequences)
Packs a list of variable length Tensors.
6. Tutorials on these functions.
(1). https://zhuanlan.zhihu.com/p/34418001
(2). https://zhuanlan.zhihu.com/p/28472545
总结起来就是：在利用 recurrent neural network 处理变长的句子序列时，我们可以配套的使用：
torch.nn.utils.rnn.pack_padded_sequence () 来对一个 mini-batch 中的句子进行 padding；
torch.nn.utils.rnn.pad_packed_sequence () 来避免 padding 对句子表示的影响。
===
Tutorial on word2vector using GloVe and Word2Vec的更多相关文章
- 词向量(one-hot/SVD/NNLM/Word2Vec/GloVe)
目录 词向量简介 1. 基于one-hot编码的词向量方法 2. 统计语言模型 3. 从分布式表征到SVD分解 3.1 分布式表征(Distribution) 3.2 奇异值分解(SVD) 3.3 基 ...
- word2vec 和 glove 模型的区别
2019-09-09 15:36:13 问题描述:word2vec 和 glove 这两个生成 word embedding 的算法有什么区别. 问题求解: GloVe (global vectors ...
- [白话解析] 带你一起梳理Word2vec相关概念
[白话解析] 带你一起梳理Word2vec相关概念 0x00 摘要 本文将尽量使用易懂的方式,尽可能不涉及数学公式,而是从整体的思路上来说,运用感性直觉的思考来帮大家梳理Word2vec相关概念. 0 ...
- NLP︱高级词向量表达（一）——GloVe（理论、相关测评结果、R&python实现、相关应用）
有很多改进版的word2vec,但是目前还是word2vec最流行,但是Glove也有很多在提及,笔者在自己实验的时候,发现Glove也还是有很多优点以及可以深入研究对比的地方的,所以对其进行了一定的 ...
- Getting Started with Word2Vec
Getting Started with Word2Vec 1. Source by Google Project with Code: https://code.google.com/archive ...
- 四步理解GloVe！(附代码实现)
1. 说说GloVe 正如GloVe论文的标题而言,GloVe的全称叫Global Vectors for Word Representation,它是一个基于全局词频统计(count-based & ...
- 词嵌入之GloVe
什么是GloVe GloVe(Global Vectors for Word Representation)是一个基于全局词频统计(count-based & overall statisti ...
- RNN 与 LSTM 的应用
之前已经介绍过关于 Recurrent Neural Nnetwork 与 Long Short-Trem Memory 的网络结构与参数求解算法( 递归神经网络(Recurrent Neural N ...
- 从Word Embedding到Bert模型—自然语言处理中的预训练技术发展史（转载）
转载 https://zhuanlan.zhihu.com/p/49271699 首发于深度学习前沿笔记 写文章 从Word Embedding到Bert模型—自然语言处理中的预训练技术发展史 张 ...
随机推荐
- 排名函数——ROW_NUMBER()、RANK()、DENSE_RANK()和NTILE(n)
ROW_NUMBER()函数:行号,根据作为参数传递给这个函数的ORDER BY子句的值,返回一个不断递增的整数值.如果ROW_NUMBER的ORDER BY的值和结果集中的顺序相匹配,返回值将是递增 ...
- bzoj4358 premu
题目链接 莫队算法 没有用线段树,而是看了showson的并查集%%% #include<algorithm> #include<iostream> #include<c ...
- JS 和 Jquery 的一些常用效果
https://www.cnblogs.com/beiz/tag/%E7%BD%91%E9%A1%B5%E5%B8%B8%E8%A7%81%E6%95%88%E6%9E%9C/ 北执
- Python进阶【第四篇】函数
一.变量 变量是记录一系列状态变化的量 1.变量分为可变类型与不可变类型——可变 与不可变是根据变量在内存中占据的位置 可变类型:列表list[ ].字典dicta{ } 不可变类型:字符串str.数 ...
- Kattis之旅——Inverse Factorial
题目意思就是已知n的阶乘,求n. 当输入的阶乘小于10位数的时候,我们可以用long long将字符串转化成数字,直接计算. 而当输入的阶乘很大的时候,我们就可以利用位数去大概的估计n. //Asim ...
- 笔面试复习（spring常用.jar包/事务/控制反转/bean对象管理和创建/springMVC工作原理/sql查询）
###spring常用jar包1.spring.jar是包含有完整发布模块的单个jar包.2.org.springframework.aop包含在应用中使用Spring的AOP特性时所需要的类.3.o ...
- 关于nginx配置虚拟主机
前提:我的虚拟主机的外网ip为111.231.226.228(是云服务器哈) 本地测试环境为windows7(修改本地的hosts文件) 步骤:(安装nginx可以看看我文章“linux ng ...
- lnmp重置密码
wget http://soft.vpser.NET/lnmp/ext/reset_mysql_root_password.sh;sh reset_mysql_root_password.sh
- Spring学习笔记2:Spring HelloWorld
1:IntelliJ新建Maven工程 2:pom文件加入Spring依赖 <project xmlns="http://maven.apache.org/POM/4.0.0" ...
- go实例—函数或方法的可变长参数
支持可变长参数列表的函数可以支持任意个传入参数,比如fmt.Println函数就是一个支持可变长参数列表的函数. 需要注意的是,可变长参数应该是函数定义的最右边的参数,即最后一个参数 package ...