How to Use Convolutional Neural Networks for Time Series Classification

How to Use Convolutional Neural Networks for Time Series Classification

2019-10-08 12:09:35

This blog is from: https://towardsdatascience.com/how-to-use-convolutional-neural-networks-for-time-series-classification-56b1b0a07a57

Introduction

A large amount of data is stored in the form of time series: stock indices, climate measurements, medical tests, etc. Time series classification has a wide range of applications: from identification of stock market anomalies to automated detection of heart and brain diseases.

There are many methods for time series classification. Most of them consist of two major stages: on the first stage you either use some algorithm for measuring the difference between time series that you want to classify (dynamic time warping is a well-known one) or you use whatever tools are at your disposal (simple statistics, advanced mathematical methods etc.) to represent your time series as feature vectors. In the second stage you use some algorithm to classify your data. It can be anything from k-nearest neighbors and SVMs to deep neural network models. But one thing unites these methods: they all require some kind of feature engineering as a separate stage before classification is performed.

Fortunately, there are models that not only incorporate feature engineering in one framework, but also eliminate any need to do it manually: they are able to extract features and create informative representations of time series automatically. These models are recurrent and convolutional neural networks (CNNs).

Research has shown that using CNNs for time series classification has several important advantages over other methods. They are highly noise-resistant models, and they are able to extract very informative, deep features, which are independent from time. In this article we will examine in detail how exactly the 1-D convolution works on time series. Then, I will give an overview of a more sophisticated model proposed by the researchers from Washington University in St. Louis. Finally, we will look at a simplified multi-scale CNN code example.

1-D Convolution for Time Series

Imagine a time series of length n and width k. The length is the number of timesteps, and the width is the number of variables in a multivariate time series. For example, for electroencephalography it is the number of channels (nodes on the head of a person), and for a weather time series it can be such variables as temperature, pressure, humidity etc.

The convolution kernels always have the same width as the time series, while their length can be varied. This way, the kernel moves in one direction from the beginning of a time series towards its end, performing convolution. It does not move to the left or to the right as it does when the usual 2-D convolution is applied to images.

 

1-D Convolution for Time Series. Source: [2] (modified).

The elements of the kernel get multiplied by the corresponding elements of the time series that they cover at a given point. Then the results of the multiplication are added together and a nonlinear activation function is applied to the value. The resulting value becomes an element of a new “filtered” univariate time series, and then the kernel moves forward along the time series to produce the next value. The number of new “filtered” time series is the same as the number of convolution kernels. Depending on the length of the kernel, different aspects, properties, “features” of the initial time series get captured in each of the new filtered series.

The next step is to apply global max-pooling to each of the filtered time series vectors: the largest value is taken from each vector. A new vector is formed from these values, and this vector of maximums is the final feature vector that can be used as an input to a regular fully connected layer. This whole process is illustrated in the picture above.

Let’s take it to another level

With this simple example in mind, let’s examine the model of a multi-scale convolutional neural network for time series classification [1].

The multi-scalability of this model consists in its architecture: in the first convolutional layer the convolution is performed on 3 parallel independent branches. Each branch extracts features of different nature from the data, operating at different time and frequency scales.

The framework of this network consists of 3 consecutive stages: transformation, local convolution, and full convolution.

 

Multi-Scale Convolutional Neural Network Architecture [1].

Transformation

On this stage different transformations are applied to the original time series on 3 separate branches. The first branch transformation is identity mapping, meaning that the original time series remains intact.

The second branch transformation is smoothing the original time series with a moving average with various window sizes. This way, several new time series with different degrees of smoothness are created. The idea behind this is that each new time series consolidates information from different frequencies of the original data.

Finally, the third branch transformation is down-sampling the original time series with various down-sampling coefficients. The smaller the coefficient, the more detailed the new time series is, and, therefore, it consolidates information about the time series features on a smaller time scale. Down-sampling with larger coefficients results in less detailed new time series which capture and emphasize those features of the original data that exhibit themselves on larger time scales.

Local Convolution

On this stage the 1-D convolution with different filter sizes that we discussed earlier is applied to the time series. Each convolutional layer is followed by a max-pooling layer. In the previous, simpler example global max pooling was used. Here, max pooling is not global, but still the pooling kernel size is extremely large, much larger than the sizes you are used to when working with image data. More specifically, the pooling kernel size is determined by the formula n/p, where n is the length of the time series, and p is a pooling factor, typically chosen between the values {2, 3, 5}. This stage is called local convolution because each branch is processed independently.

Full Convolution

On this stage all the outputs of local convolution stage from all 3 branches are concatenated. Then several more convolutional and max-pooling layers are added. After all the transformations and convolutions, you are left with a flat vector of deep, complex features that capture information about the original time series in a wide range of frequency and time scale domains. This vector is then used as an input to fully connected layers with Softmax function on the last layer.

Keras Example

from keras.layers import Conv1D, Dense, Dropout, Input, Concatenate, GlobalMaxPooling1D
from keras.models import Model#this base model is one branch of the main model
#it takes a time series as an input, performs 1-D convolution, and returns it as an output ready for concatenationdef get_base_model(input_len, fsize):
#the input is a time series of length n and width 19
input_seq = Input(shape=(input_len, 19))
#choose the number of convolution filters
nb_filters = 10
#1-D convolution and global max-pooling
convolved = Conv1D(nb_filters, fsize, padding="same", activation="tanh")(input_seq)
processed = GlobalMaxPooling1D()(convolved)
#dense layer with dropout regularization
compressed = Dense(50, activation="tanh")(processed)
compressed = Dropout(0.3)(compressed)
model = Model(inputs=input_seq, outputs=compressed)
return model#this is the main model
#it takes the original time series and its down-sampled versions as an input, and returns the result of classification as an outputdef main_model(inputs_lens = [512, 1024, 3480], fsizes = [8,16,24]):
#the inputs to the branches are the original time series, and its down-sampled versions
input_smallseq = Input(shape=(inputs_lens[0], 19))
input_medseq = Input(shape=(inputs_lens[1] , 19))
input_origseq = Input(shape=(inputs_lens[2], 19))#the more down-sampled the time series, the shorter the corresponding filter
base_net_small = get_base_model(inputs_lens[0], fsizes[0])
base_net_med = get_base_model(inputs_lens[1], fsizes[1])
base_net_original = get_base_model(inputs_lens[2], fsizes[2])embedding_small = base_net_small(input_smallseq)
embedding_med = base_net_med(input_medseq)
embedding_original = base_net_original(input_origseq)#concatenate all the outputs
merged = Concatenate()([embedding_small, embedding_med, embedding_original])
out = Dense(1, activation='sigmoid')(merged)model = Model(inputs=[input_smallseq, input_medseq, input_origseq], outputs=out)
return model

This model is a much simpler version of the multi-scale convolutional neural network.

It takes the original time series and 2 down-sampled versions of it (medium and small length) as an input. The first branch of the model processes the original time series of length 3480 and of width 19. The corresponding convolution filter length is 24. The second branch processes the medium-length (1024 timesteps) down-sampled version of the time series, and the filter length used here is 16. The third branch processes the shortest version (512 timesteps) of the time series, with the filter length of 8. This way every branch extracts features on different time scales.

After convolutional and global max-pooling layers, dropout regularization is added, and all the outputs are concatenated. The last fully connected layer returns the result of classification.

Conclusion

In this article I tried to explain how deep convolutional neural networks can be used to classify time series. It is worth mentioning that the proposed method is not the only one that exists. There are ways of presenting time series in the form of images (for example, using their spectrograms), to which a regular 2-D convolution can be applied.

Thank you very much for reading this article. I hope it was helpful to you, and I would really appreciate your feedback.

References:

[1] Z. Cui, W. Chen, Y. Chen, Multi-Scale Convolutional Neural Networks for Time Series Classification (2016), https://arxiv.org/abs/1603.06995.

[2] Y. Kim, Convolutional Neural Networks for Sentence Classification (2014), https://arxiv.org/abs/1408.5882.