EE5111_A0206839W
EE5111 Selected Topics in Industrial Control & Instrumentation
Assessment:
Implement a simple IoT pipeline with AWS Cloud platform and Visualise the data.
Author:
Zhan Yuancheng (A0206839W)
Github:
https://github.com/nkzyc/EE5111_A0206839W
Medium post:
https://www.cnblogs.com/nkzyc/p/11561289.html
1.Introduction. 3
1.1 IOT Background. 3
1.2 Goals of the Assessment. 5
1.3 Prerequisites. 5
2.Sending data to AWS by Official routine. 6
2.1 Create the AWS IoT Policy. 6
2.2 Create the things. 7
2.3 Email Alerts for Low Water Content Readings. 8
2.4 Simulate random water content. 10
2.5 Results. 11
3.Sending data FD001 to AWS. 11
3.1 Introduction of the database. 11
3.2 Create the AWS IoT Policy for FD001. 12
3.3 Create the things of FD001. 13
3.4 Create AWS IoT rule and AWS DynamoDB. 14
3.5 Use python codes to meet the requirements. 17
3.6 Results and Analysis. 19
4. Sending data from FD001 and FD002 to AWS. 20
4.1 Create the Policy, Thing and DynamoDB for FD002. 20
4.2 Create IoT rule for both FD001 and FD002. 22
4.3 Results and Analysis. 22
5.Visualization of data. 23
5.1 Get AWS access key. 23
5.2 Redash environment. 24
5.3 Obtain and Visualize the data. 25
5.4 Examples of Visualization by Redash. 26
6.Anylasis of Motor Vehicle Inspection data source. 28
6.1 Introduction of data source. 28
6.2 Create the Policy, Thing, Rule and DynamoDB. 29
6.3 Data processing and Results. 31
6.4 Visualization of data source. 33
7.Conclusion. 36
8.Appendix(completed python codes) 36
1.Introduction
1.1 IOT Background
IOT
The Internet of Things (IoT) is a system of interrelated computing devices, mechanical and digital machines, objects, animals or people that are provided with unique identifiers (UIDs) and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction.
The definition of the Internet of things has evolved due to the convergence of multiple technologies, real-time analytics, machine learning, commodity sensors, and embedded systems. Traditional fields of embedded systems, wireless sensor networks, control systems, automation (including home and building automation), and others all contribute to enabling the Internet of Things. In the consumer market, IoT technology is most synonymous with products pertaining to the concept of the "smart home", covering devices and appliances (such as lighting fixtures, thermostats, home security systems and cameras, and other home appliances) that support one or more common ecosystems, and can be controlled via devices associated with that ecosystem, such as smartphones and smart speakers.
There are a number of serious concerns about dangers in the growth of IoT, especially in the areas of privacy and security; and consequently industry and governmental moves to begin to address these.
AWS IoT
AWS IoT provides secure, bi-directional communication between Internet-connected devices such as sensors, actuators, embedded micro-controllers, or smart appliances and the AWS Cloud. This enables you to collect telemetry data from multiple devices, and store and analyze the data. You can also create applications that enable your users to control these devices from their phones or tablets.
AWS IoT consists of the following components:
Device gateway
Message broker
Rules engine
Security and Identity service
Registry
Group registry
Device shadow
Device Shadow service
Device Provisioning service
Custom Authentication service
Jobs service
AWS IoT integrates directly with the following AWS services:
Amazon Simple Storage Service
Amazon DynamoDB
AWS Lambda
Amazon Simple Notification Service
Amazon Simple Queue Service
And so on
1.2 Goals of the Assessment
The whole information of the goals are on the github:
https://github.com/iceberg12/NUS_guest_lecture/tree/master/EE5111
From my report, I achieve the following goals:
1.Follow the AWS instruction to understand how to populate, send data from a computer to AWS through MQTT protocol. (Optional)
2. Publish pre-defined engine data to AWS.
3. Simulate two IoT things.
4. Visualize the data for the two engines for all the sensors.
5. Use other data sources.
All of the goals I have achieved will be implemented in the following report with detailed description.
1.3 Prerequisites
I will first explain The tools and environment configuration I used in this task are following. Please refer to the corresponding official documentation for the specific installation steps.
Environment:
Windows 8.1
Python 3.5.4
AWS IOT core
Tools:
Pycharm 2017
Anaconda 3
Datadog
Redash
2.Sending data to AWS by Official routine
In the Official routine, there is a tutorial of Plant Watering with AWS IoT. This will help us to understand how to send data to AWS with our computer.
This control of Plant Watering shows us how to use a moisture sensor and AWS IoT to monitor the soil moisture level for a house plant or garden. We can create a rule in AWS IoT that sends an email to an address subscribed to an Amazon SNS topic when the moisture level falls below a threshold.
There are five steps as follow.
2.1 Create the AWS IoT Policy
In this step, create an AWS IoT policy that allows our sensor to connect and send messages to AWS IoT. We should do the following:
Sign in to the AWS Management(https://aws.amazon.com) and open the AWS IoT console; enter the IoT Core and select the policies as shown in Figure2.1
Figure2.1 create policy1
select Create Policy and provide a name PlantWateringPolicy; Enter iot:* for action and * for ARN as shown in Figure2.2;
Figure2.2 create policy2
2.2 Create the things
In this step, we will create one of the AWS IoT in things , which is used a device simulator. Devices connected to AWS IoT are represented by things in the AWS IoT registry.
Select Manage and choose Things and use Create; choose to create a single thing as shown in Figure2.3.
Figure2.3
Give the name MyRPi and select Next; On the Add a Certificate page, choose Create Certificate as shown in Figure2.4
Figure2.4
For the certificate of the thing , choose to download; Select Activate; For the policy of adding things , select PlantWateringPolicy. Then our thing has been created as shown in Figure2.5.
Figure2.5
2.3 Email Alerts for Low Water Content Readings
In this step, we set up Amazon Simple Notification Service (Amazon SNS) to automatically send an email alert to the owner of the green plant when the soil moisture content is too low, alerting the watering.
Create an AWS IoT rule to trigger email alerts through Amazon SNS. Select Actions and select Create a rule; Enter a name for this rule MyRPiLowMoistureAlertRule; For rule query statements, enter the following AWS IoT SQL statement.
SELECT* FROM '$aws/things/MyRPi/shadow/update/accepted' WHERE state.reported.moisture = 'low'
For setting one or more actions , select Add Action choose to send the message as an SNS push notification; All are as shown in Figure2.6
Figure2.6
Set up Amazon SNS to send messages to my email inbox via an Amazon SNS topic; For the topic ARN , select the ARN for the topic; For the agreement , select Email and enter my email address.
All are as shown in Figure2.7
Figure2.7
2.4 Simulate random water content
In this step, we can then press these readings to the relevant shadow in the AWS IoT. When the reading is too low, Amazon SNS automatically sends an email alert to the owner of the green plant.
Create a new file on the development machine using the code (plantwater.py) in the Github as shown in Figure2.8
Figure2.9
Use the pycharm to run the code.
2.5 Results
My NUS email has received the alerts from AWS as shown in Figure2.9, which indicate our first goal of sending data from a computer to AWS through MQTT protocol is successful.
Figure2.9
3.Sending data FD001 to AWS
3.1 Introduction of the database
The data provided is from a high fidelity system level engine simulation designed to simulate nominal and fault engine degradation over a series of flights. The flights are full flight recording sampled at 1 Hz and consists of 30 engine and flight condition parameters. Each flight contains 7 unique flight conditions for an approximately 90 min flight including ascent to cruise at 35K ft and descent back to sea level. The structure of the plane sensor is as shown in Figure3.1.
Figure3.1
FD001 and FD002 are input data for machine fleets. Sensor names can be found in CMAPSS file. The column names for the Text Input Files are: id, cycle, os1, os2, os3, sensor1, ...., sensor22.
3.2 Create the AWS IoT Policy for FD001
This step is similar to the 2.1 Create the AWS IoT Policy in the report. I will show the change between them.
Sign in to the AWS Management(https://aws.amazon.com) and open the AWS IoT console; Enter the IoT Core and select the policies.
select Create Policy and provide a name Mypo1_A0206839W; Enter iot:* for action and * for ARN as shown in Figure3.2;
Figure3.2
3.3 Create the things of FD001
This step is similar to the 2.2 Create the things in the report. I will show the change between them.
Select Manage and choose Things and use Create; choose to create a single thing;
Give the name Mything1_A0206839W and select Next; On the Add a Certificate page, choose Create Certificate as shown in Figure3.3
Figure3.3
For the certificate of the thing , choose to download of the public key , the private key, and the root CA of the AWS IoT.
Select Activate; For the policy of adding things , select Mypo1_A0206839W as shown in Figure2.4.
Figure3.4
Then Mything1_A0206839W has been created as shown in Figure3.5.
Figure3.4
3.4 Create AWS IoT rule and AWS DynamoDB
In this step, we need to use DynamoDB rules to collect information from MQTT message, and send those information to the DynamoDB database, where we can analyze the them.
Select Actions and select Create a rule; Enter a name for this rule Role1; For rule query statements, enter the following AWS IoT SQL statement as shown in Figure3.5
SELECT state.reported.* FROM '$aws/things/ Mything1_A0206839W /shadow/update/accepted'
Figure3.5
For setting one or more actions , select Add Action choose to as shown in Figure3.6
Figure3.6
On the Configure action page, choose Create a new resource and go to the Amazon DynamoDB page, choose Create table as shown in Figure3.7.
Figure3.7
On the Create DynamoDB Table page, enter a name A0206839W. Choose 'id' as the partition key and 'timestamp' as the sort key as shown in Figure3.8
Figure3.8
choose Create to create table A0206839W as shown in Figure3.9
Figure3.9
Back to the configure action; in Create a new role , enter the name role_A0206839W and use Create as shown in Figure3.10
Figure3.10
Select Create Rule to create a rule, the rule Role1 as shown in Figure3.11 .
Figure3.11
3.5 Use python codes to meet the requirements
3.5.1 Requirement
The requirements of the Publish pre-defined engine data to AWS(Step 2) is as follow:
a. Download data for engines FD001 and FD002 from Github .
b. Modify the Jupyter notebook in Step 1 to read and publish data from trainFD001.txt to your thing under AWS IoT platform at the rate of 10 seconds per row. Overwrite column 'id' of the engine as 'FD001' + id; add one more columns 'timestamp' as timestamp in UTC; also add one more column that contains your Matric number.
c. Setup an Act under AWS IoT platform to define a rule that splits the message into columns of a DynamoDB table.
d. Check under AWS DynamoDB service if your table is populated correctly.
3.5.2 Methods
I design the method to satisfy all of the requirements step by step, which will show in the following.
The completed codes are in the appendix(main1.py)
For the a.,
I has download from the Github;
For the b.,
First, I design some codes to meet the specific requirement:
Read and publish data from FD001.txt under AWS IoT platform at the rate of 10 seconds per row
time.sleep(10)
Overwrite column 'id' of the engine as 'FD001' + id
content.append(str('"' + 'FD002_' + x[0] + '",'))
Add one more columns 'timestamp' as timestamp in UTC
tim = str(datetime.datetime.utcnow());#tim= timestamp
Add one more column that contains your Matric number
maNum = 'A0206839W'
Second, I design two function to achieve specific form
# to form the title of the column of the content
def form_title ():
#change
date into joson form to send
def form_joson (content,x,title,tim,maNum):
Third, the codes of reading file FD001.txt and sending it to DynamoDB is as follow:
fp = open('train_FD002.txt', 'r')
#title which satisfy
the request by teacher
title=form_title ();
for reader in fp.readlines():
x = reader.split(" ")
#time
tim = str(datetime.datetime.utcnow());
#the content of the message
content = [];
#the joson form of data to send
send=form_joson(content, x, title, tim, maNum)
print(send)
#send to the dynamnoDB
myDeviceShadow.shadowUpdate(send, myShadowUpdateCallback,
5)
time.sleep(10)
fp.close();
For the c.,
I have finish it on the 3.4 create AWS IoT rule and AWS DynamoDB
For the d.
I will show the outcome in the 3.6 Results and Analysis
3.6 Results and Analysis
Results
The results is shown as Figure3.12.
Figure3.12
Analysis
From the results, we can conclude the
following things:
1. the DynamoDB table A0206839W has received the data sent by Mything1_A0206839W successfully,
which indicates the whole operations of AWS IoT are correct.
2. By subtracting one line, the ‘id’ is FD001_1, the ‘timestamp’ is 2019-09-20
11:38:57.733647, and the other ‘sensorx’
is the same as the ‘FD001.txt’ data,
which indicates the python codes satisfy the requirements.
4. Sending data from FD001 and
FD002 to AWS
We
are going to simulate two IoT things.
Therefore, we need to use another IoT thing to complete.
As the whole processes of another thing are
similar to the 3.sending data FD001 to
AWS, my report will omit duplicates.
4.1 Create the Policy, Thing and DynamoDB
for FD002
4.1.1 Policy
Select Create
Policy and provide a name Mypo2_A0206839W; Enter iot:* for action and * for ARN, the policy is as shown in Figure4.1
Figure4.1
4.1.2 Thing
Select Manage and choose Things and use Create; choose to create a
single thing;
Give the name Mything2_A0206839W;
Mything2_A0206839W has been created as shown in Figure4.2.
Figure4.2.
4.1.3 DynamoDB
As we need to simulate the two
"things" to run in parallel, we have to use the same DynamoDB(A0206839W) to the first thing.
4.2 Create IoT rule for both FD001
and FD002
We need to modify the rule
Role1 to meet the requirements of step3.
To be specific, we need to change the rule query statements as follow:
Change
SELECT state.reported.* FROM '$aws/things/ Mything1_A0206839W
/shadow/update/accepted'
Into
SELECT state.reported.* FROM '$aws/things/ +
/shadow/update/accepted'
In that way, the rule can choose both Mything1_A0206839W and Mything2_A0206839W
to the DynamoDB and we can simulate the two "things" to run in
parallel.
The updated Role1 is shown
in Figure4.3
.
Figure4.3
4.3 Results and Analysis
The completed codes of Mything2_A0206839W are in the appendix.
I use the two .py(main1.py
and main2.py in the appendix) run at the same time, which is aimed at stimulating
the two "things" to run in parallel to publish data.
Results
The results in DynamoDB are shown as Figure4.4
.
Figure4.3
Analysis
From the results, we can conclude the following things:
1. the DynamoDB table A0206839W has received the data sent by Mything2_A0206839W successfully, which
indicates the whole operations of Mything2_A0206839W
are correct.
2. By studying the second figure of Figure4.3, we find that the DynamoDB has received both FD001 and
FD002 data and stored them individually. This indicates my methods satisfy the
requirements of step3.
5.Visualization of data
5.1 Get AWS access key
Go
to the AWS account and click the zyc18 in the Users as shown in Figure5.1
Figure5.1
Add
the Add permissions and select AmazonDynamo Db as shown in Figure5.2
Figure5.2
Download the access keys from
it as shown in Figure5.3
Figure5.3
Then we can use those keys to permit us to
subtract data from dynamoDB table.
5.2 Redash environment
Introduction
Redash is an Open Source Company to
democratize data and make data driven decision making easy.
We
can query Amazon DynamoDB by using its natural syntax, enjoy live auto-complete
and explore Amazon DynamoDB schema easily in Redash's cloud-based query editor;
And we can visualize Amazon DynamoDB data and gather it into
thematic dashboards from multiple sources, share the story your data tells with
team or extermal partners.
Configuration
On the Redash home page(https://app.redash.io), open the New Data Source page and choose to Create a New Data Source, and select DynamoDB;
Put the Name, Access Key and Secret Key of our AWS DynamoDB as shown
in in Figure5.4.
Figure5.4
5.3 Obtain and Visualize the data
Obtain
Open
the Queries page and use SQL
language to query data from the DynamoDB
table as shown in Figure5.5
Figure5.5
Now we can obtain the data from the DynamoDB table of A0206839W
Visualize
Open the New Visualization to visualize the data as shown in Figure5.6
Figure5.6
We can use this tools to visualize data
which we want to study.
5.4 Examples of Visualization by Redash
Using this visualization, I choose several
representative examples to show the relationship between the data.
Chat line
I take
time as X Colum and os1,os2,os3 as
Y Colum, the outcome as shown in Figure5.7
Figure5.7
Chat bar
I take time
as X Colum and sensor1, sensor2, sensor3
as Y Colum, the outcome as shown in Figure5.8
Figure5.8
Chat Scatter
I take time
as X Colum and sensor16, sensor17,
sensor18 as Y Colum, the outcome as shown in Figure5.9
Boxplot
The outcome of all the columns as shown in
Figure5.10
6.Anylasis of Motor Vehicle
Inspection data source
6.1 Introduction of data source
The data source is the about annual motor vehicle inspection - passing
pate of motor vehicles on first inspection from the https://data.gov.sg.
The columns of the data source are shown as Figure5.1
Figure5.1
Some samples of the data are
shown as Figure5.2
Figure5.2
The goals of my IoT are to collect
those source from specific monitors, store
them in the AWS DynamoDB and do the
analysis through the Visualization.
6.2 Create the Policy, Thing, Rule
and DynamoDB
As the whole processes are similar to the past
operation, I will omit duplicates and point to the emphasis.
6.1.1 Policy
Select Create
Policy and provide a name Mypo2_A0206839W;
Enter iot:* for action and * for ARN, the policy is as shown in Figure6.1
Figure6.1
6.1.2 Thing
Select Manage and choose Things and use Create; choose to create a
single thing;
Give the name Mypas_rate_A0206839W;
The thing has been created as shown in Figure6.2.
Figure6.2
6.1.3 DynamoDB
On the Create DynamoDB Table
page, enter a name pass_rate. Choose
'year' as the partition key and 'timestamp'
as the sort key. The DynamoDB table as
shown in Figure6.3
Figure6.3
6.1.4 Rule
Select Actions
and select Create a rule; Enter a name Rule_pasrate for this rule Role1;
For setting
one or more actions , select Add
Action and choose the DynamoDB table
The rule has been created as shown in
Figure6.4
Figure6.4
6.3 Data processing and Results
We
need to change the .csv data into .json data. In addition, in order to show
more clearly, we need to add one more column 'timestamp' as timestamp and add
one more column that contains Matric number.
The
completed codes are in the appendix(main3.py)
Methods
Change the .csv data into .json data
with open("pas_rate.csv", "r", encoding="utf-8") as fp:
# reader是一个迭代器
reader = csv.reader(fp)
# 执行一次next,指针跳过一位,可以不获取标题
next(reader)
for x in reader:
content = [];
tim = str(datetime.datetime.utcnow());
content.append(str(title[0] + ':'))
content.append(str('"' + x[0] + '",'))
for i in range(1,8):
content.append(str(title[i] + ':'))
if i==1:
content.append(str('"' + tim + '",'))
elif i==2:
content.append(str('"' + maNum + '",'))
else:
content.append(str('"' + x[i-2] + '",'))
cc=''.join(content)[:-1]
Add one more column 'timestamp' as timestamp
tim = str(datetime.datetime.utcnow());#tim= timestamp
Add one more column that contains Matric number
maNum = 'A0206839W'
Results
The results in DynamoDB pass_rate are shown as Figure6.5
Figure6.5
Analysis
From the results, we can conclude the following things:
1. the DynamoDB table pass_rate has received the data sent by Mypas_rate_A0206839W
successfully, which indicates the whole operations of Mypas_rate_A0206839W are correct.
2. We find that the pass_rate has the following columns:
'year','timestamp','MatricNumber','type','age',‘number_reported','number_passed','passing_rate'
with correct data.
This indicates my methods satisfy the requirements of step5.
6.4 Visualization of data source
Using the same methods of 5.Visualization
of data, we can study the annual motor vehicle inspection by Redash and visualize them.
6.4.1 Obtain the data
Open the Queries page and use SQL language to query data from the DynamoDB
table pass_rate as shown in Figure6.6
Figure6.6
6.4.2 Visualize and Analyze the data
Open the New Visualization to visualize the data. We can use this function
to draw some conclusions which may
help us to determinate vehicles.
I take type
as X Colum and passing_rate as Y
Colum, the outcome as shown in Figure6.7
From the figure we can conclude that Other Vehicles has the lowest pass
rate and Taxi has the highest pass rate.
Figure6.7
I take year
as X Colum and passing_rate as Y
Colum, the outcome as shown in Figure6.8
From the figure we can conclude that pass rate become better by time, which
indicate the vehicles are more and more reliable.
Figure6.8
I take age
as X Colum and reported and passed
as Y Colum, the outcome as shown in Figure6.9
From the figure we can conclude thatmost people love to examine their
vehicles within 3 years..
7.Conclusion
I completed the following goals that the teacher requested:
1. Understand
how to populate, send data from a computer to AWS through MQTT protocol.
2.
Publish pre-defined engine data to AWS.
3. Simulate two IoT things.
4. Visualize the data.
5. Use different data sources to analyze.
In addition, I have the following benefits:
1.
Master the skills of how to quickly access knowledge from official documents
2.
Master more python and java code methods
3.
Learned a lot of new features on AWS to prepare for future use
4.
Have a new understanding of the entire control system
Finally, thanks Dr. Nguyen Hoang Tuan Minh for your knowledge and guidance.
EE5111_A0206839W的更多相关文章
随机推荐
- Android组件内核之Activity调用栈分析(一)
阿里P7Android高级架构进阶视频免费学习请点击:https://space.bilibili.com/474380680 导语 我们陈述一下Activity,Activity是整个应用用户交互的 ...
- Mybatis缓存1----系统缓存及简单配置介绍
mybatis缓存 系统缓存:常用的一级缓存和二级缓存 一级缓存 一级缓存是SqlSession级别的缓存,在操作数据库时需要构建SqlSession对象,在对象中有一个数据结构用于存储缓存数据. ...
- vue对象侦测
http://blog.csdn.net/yihanzhi/article/details/74200618 数组:this.$set(this.arr,index,value)
- python skimage库HOG特征提取原码解读
Hog特征+SVM常用来做行人检测. opencv中也有Hog特征提取的原码,但是由于原码不是用python写的,而skimage用python实现了,所以就解读的skimage的代码. 先看用ski ...
- 解决.Net MVC 中出现 非介入式客户端验证规则中的验证类型名称必须唯一。下列验证类型出现重复: required 的bug
最近在开动科技创新作品的开发,出现了一个让人很烦恼的错误,每次从浏览页跳转到编辑页时就会出现一下错误 非介入式客户端验证规则中的验证类型名称必须唯一.下列验证类型出现重复: required 上一下出 ...
- idea maven springmvc mybabit 多模块管理整合
一.安装软件jdk1.7,tomcat7.0,idea,mysql,maven 二.在idea中配置jdk 1.依次点开File -->Project Structure,点击左侧标签页,点击S ...
- 2018焦作网络赛-E- Jiu Yuan Wants to Eat
题目描述 You ye Jiu yuan is the daughter of the Great GOD Emancipator. And when she becomes an adult, s ...
- 搭建appium+maven手机自动化测试环境搭建
搭建安卓自动化测试框架记录: 需要的软件: jdk版本:jdk1.8 appium版本:v1.4.16 下载AppiumForWindows.zip (下载appium1.7不需要配置环境变量) no ...
- Qt 【关于控件样式,鼠标进入、离开、点击】
比如举以下这个例子: QPushButton * okBtn; okBtn->setstylesheet("QPushButton{border-image:url(:/image/h ...
- HBase封装easy-hbase设计实现
新增码云地址:https://gitee.com/hanmov5/mop-hbase-template 一.写在前面 业务架构用到HBase,但由于某些不可名状原因,没有用phoniex等上层工具,开 ...