How to generate the complex data regularly to Ministry of Transport of P.R.C by DB Query Analyzer

How to generate the complex data regularly to Ministry of Transport of P.R.C by DB Query Analyzer

1 About DB Query Analyzer

DB Query Analyzer is presented by Master Genfeng, Ma from Chinese Mainland. It has English version named ‘DB Query Analyzer’ and Simplified Chinese version named  .

DB Query Analyzer is one of the few excellent Client Tools in the world for its’ powerful function, friendly interface, easy operation and applicability to every production of RDBMS.

① It lets you query ODBC data sources, author SQL scripts and queries, return query results to a grid or free-form text or a file, retrieve ODBC driver information, ②execute multiple SQL scripts or stored procedures simultaneously, ③save the result to files with high efficiency which is as quick as tools provided by database system. ④ What is more, from 6.01,DB Query Analyzer provides SQL Execute Schedule function to execute SQL scripts in certain time. If you select the option ‘SQL Schedule Reconnect’ in configurations window, the Schedule SQL scripts will be executed as long as the Database Server is in use, even though the Database Server had been down. ⑤From version 6.03, without changing any Windows OS settings or configurations, DB Query Analyzer can run on any Microsoft Windows OS directly such as Windows 10,Windows 8, Windows 7, Windows Vista, Windows 2003, Windows XP, Windows 2000,Windows NT, Windows ME, Windows 9X.

In the New products & Tools reviews of programmer second issue of 2007, DB Query Analyzer had been strongly recommended.

Now the Simplified Chinese version of DB Query Analyzer is the top 50 database application software in the famous software website http://xiazai.zol.com.cn/download_order/sub_550.html .In most case it lies the top 10 and it has been downloaded more than 110,000 times.

85 technical articles about DB Query Analyzer have been published in computer journal, BAIDU Library, the CSDN resource or my three blog-websites.

2 Contents reported regularly

First in Chinese there is a time unit, which is defined as a period of ten days in a month. One month has three sections. The first period of ten days is from the 1^st dayto the 10^th day, and the second period of ten days means the 11^stday to the 20^th day. In addition, the 21^st day to the last day is the third period of a month.

If you have experience in big data processing, you will know the complex of getting all the data required in the table.

In fact I am familiar with big data processing. Every month 1.2 billion records will occur in the database according to the vehicles that pass by the high way in Guangdong province.  So I told an officer from Guangdong Transport Department in a transport conference held in Beijing last month that I would design more efficient algorithm to generate the data required in the table every period of ten days.

As Guangdong is adeveloped province in China, the income from Guangdong of a series of state-owned company always occupies one sixth of the whole state industry income.

If there are only a few records in a table, you will not meet with any difficulties in data processing whatever algorithm you choose. Considering the big data to generate the result in the table above, I have no choice but to design more efficient algorithm.

3 Algorithm design by myself and steps of big data processing

Considering the complex of the data required by Ministry of Transport of P.R.C every period of tendays, I design the following algorithm to generate the data required.

To begin with, get theinter mediate results by the five group SQL scripts in schedule with DB Query Analyzer. In addition, the final data will generate with DB Query Analyzer by accessing the six text files also in schedule.

Figure 1   Five section to get the intermiediate results

3.1 How to get the newly added toll station in this month.

The definition of newly added toll station is that occurs from the 1^st day to the end of this period of ten days this month compare with the same month in last year. As aresult, the maximum record number is all the record occurred in this month when processing the third period of ten days. Traffic volume in Guangdong is about 1.2 billion a month. So in this case, I have to process 1.2 billion records this month and 1.2 billion records the same month last year to get the set of newly added toll station.

Figure 2   Get the set of newly added toll station in schedule by DB Query Analyzer

From 6.01, DB Query Analyzer provides SQL Execute Schedule function to execute SQL scripts in certain time. If you select the option ‘SQL Schedule Reconnect’ in configurations window, the Schedule SQL scripts will be executed as long as the Database Server is in use, even though the Database Server had been down.

3.2 How to get a series of traffic data of trunk this period of ten days

Figure 3   Get a series of traffic data of trunk this period of ten days in schedule by DB Query Analyzer

Similarly, other traffic data of trunk can be gotten with corresponding SQL scripts in schedule by DB Query Analyzer as well.

3.3 All the intermediate result generated

The sixth text files are the intermediate result and five files are created by DB Query Analyzer inschedule. At last, the intermediate result is used to generate the final result.

Figure 4  The sixth text files , the intermediate result

3.4 How to configure the ODBC data source to the directory in Figure 4

DB Query Analyzer can access text/csv files with SQL script by ODBC data source. So if you can configure a ODBC data source firstly, then you can copy some text/csv files thatyou want to access to the directory. Consequently, you can access those files with SQL scripts by DB Query Analyzer.

Figure 5   Configure the ODBC data source to the directory

 

Figure 6   Configure the ODBC datasource to the directory

3.5 Access six text files to generate all the data required  

Figure 7   Access 6 text files in schedule to generate all the data

Figure 8   Open the result file with EXCEL and you can paste them to the table

 

4 The advantage of my algorithm design

Firstly, you have to choose the strategy to disassemble the complex task into a few easier sub tasks when facing big data, which prevent you from locking tables for quite long time.

Secondly, you need not copy intermediate data to other machine and create intermediate tables in database which you have to fill in quite a few tables for approval process. Also you need not to insert data into tables you will create.

Thirdly, it is very convenient to test SQL scripts, find logic errors in your SQL scripts which may result in error data, since you can open text/csv files with NotePad, Wordpad as well as DB Query Analyzer using SQL scripts.

 

5 Summary

Today I just demonstrate that DB Query Analyzer is very good tool in big data processing.What’s more, 85 technical articles about DB Query Analyzer which have been published in computer journal, BAIDU Library, the CSDN resource or my there blog-websites.

At present, DB QueryAnalyzer is widely used in our company and our colleagues are fond of it toaccess DB2 database.

 

 

DB Query Analyzer 6.04 download URL: 

http://xiazai.zol.com.cn/detail/43/420901.shtml

http://www.unitedpowersoft.com/UpFile/DBQueryAnalyzer_English_604.rar

 

DB Query Analyzer Simplified Chinese version 6.04  download URL:

http://xiazai.zol.com.cn/detail/27/264455.shtml

My blogs are http://blog.csdn.net/magenfeng

http://blog.sina.com.cn/magenfeng

http://user.qzone.qq.com/630414817

Interview with CSDN:  http://www.csdn.net/article/2014-08-09/2821124

 

85 technical articles about DB Query Analyzer have been published in computer journal, Baidu Library, the CSDN resource and my three blog-websites.

The following 7 articles have been published in computer journals.

Guangdong Highway settlement business analysis system based on Star Schema in Data Warehousing according to national standardization in Computer Era 7th issue of 2015 in Hangzhou.

DB Query Analyzer to cancel a running SQL statement in Computer Era 12th issue of 2011 in Hangzhou.

DB Query Analyzer Returns Information in More Detail While Batch SQL Statement End of Execution in Computer Programming Skill & Maintenance 24th issue of 2011 in Beijing

The Application of the Transactions Control in DB2 with DB Query Analyzer in Computer Programming Skill & Maintenance 22nd issue of 2011 in Beijing.

How does DB Query Analyzer cancel the SQL statement committed to DBMS in Computer Engineering & Software 6th issue of 2011 in Tianjin.

The Application of the Transactions Control in Oracle with DB Query Analyzer in Microcomputer Applications 11th issue of 2011 in Shanghai.

    DB Query Analyzer had been strongly recommended In the New products & Tools reviews of programmer 2nd issue of 2007.

The following 78 articles have been published in Baidu Library, the CSDN resource and my three blog-websites.

The DBMS that DB Query Analyzer Users often use cover all kinds of DBMS have been published.

How to configure ODBC DSN in Client to access remote DB2 for Windows

How to configure ODBC DSN to access local DB2 for Windows

Which SQL statement is the trump card to the senior software developer

Exepacker prevent DB Query Analyzer from being debugged have been published.

16 articles from The 1st tip of DB Query Analyze to The 16th tips of DB QueryAnalyzer have been published.

The table name must be enclosed in double quotation marks or square bracket while accessing EXCEL by DB Query Analyzer

How to download the installation package by ZOL Downer

DB Query Analyzer, the most excellent Universal database Access tools on any Microsoft Windows OS

Demonstration of DB Query Analyzer 6.03 Installation and Running on Microsoft Windows 8

How to easily create DB2 user function by DB Query Analyzer 6.03

Install and run DB Query Analyzer 6.04 on Microsoft Windows 10

A universal algorithm design of locating fixed length string

How to generate the complex data regularly to Ministry of Transport of P.R.C by DB Query Analyzer

我

马根峰本人的微信公众号：

万能数据库查询分析器

马根峰的博客：

CSDN博客：http://blog.csdn.net/magenfeng

新浪博客：  http://blog.sina.com.cn/magenfeng

官方网址：   http://www.unitedpowersoft.com/

更多精彩文章，技术分享，敬请期待