大数据(bigdata)练习题

1.在HDFS文件系统的根目录下创建递归目录“1daoyun/file”，将附件中的BigDataSkills.txt文件，上传到1daoyun/file目录中，使用相关命令查看文件系统中1daoyun/file目录的文件列表信息。

答：

[root@master MapReduce]# hadoop fs -mkdir -p /1daoyun/file

[root@master MapReduce]# hadoop fs -put BigDataSkills.txt /1daoyun/file

[root@master MapReduce]# hadoop fs -ls /1daoyun/file

Found 1 items

-rw-r--r--   3 root hdfs       1175 2018-02-12 08:01 /1daoyun/file/BigDataSkills.txt

2.在HDFS文件系统的根目录下创建递归目录“1daoyun/file”，将附件中的BigDataSkills.txt文件，上传到1daoyun/file目录中，上传过程指定BigDataSkills.txt文件在HDFS文件系统中的复制因子为2，并使用fsck工具检查存储块的副本数。

答：

[root@master MapReduce]# hadoop fs -mkdir -p /1daoyun/file

[root@master MapReduce]# hadoop fs -D dfs.replication=2 -put BigDataSkills.txt /1daoyun/file

[root@master MapReduce]# hadoop fsck /1daoyun/file/BigDataSkills.txt

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

Connecting to namenode via http://master.hadoop:50070/fsck?ugi=root&path=%2F1daoyun%2Ffile%2FBigDataSkills.txt

FSCK started by root (auth:SIMPLE) from /10.0.6.123 for path /1daoyun/file/BigDataSkills.txt at Mon Feb 12 08:11:47 UTC 2018

.

/1daoyun/file/BigDataSkills.txt:  Under replicated BP-297530755-10.0.6.123-1518056860260:blk_1073746590_5766. Target Replicas is 2 but found 1 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).

Status: HEALTHY

Total size: 1175 B

Total dirs: 0

Total files: 1

Total symlinks: 0

Total blocks (validated): 1 (avg. block size 1175 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 1 (100.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 3

Average block replication: 1.0

Corrupt blocks: 0

Missing replicas: 1 (50.0 %)

Number of data-nodes: 1

Number of racks: 1

FSCK ended at Mon Feb 12 08:11:47 UTC 2018 in 1 milliseconds

The filesystem under path '/1daoyun/file/BigDataSkills.txt' is HEALTHY

3.HDFS文件系统的根目录下存在一个/apps的文件目录，要求开启该目录的可创建快照功能，并为该目录文件创建快照，快照名称为apps_1daoyun，使用相关命令查看该快照文件的列表信息。

答：

[hdfs@master ~]# hadoop dfsadmin -allowSnapshot /apps

Allowing snaphot on /apps succeeded

[hdfs@master ~]# hadoop fs -createSnapshot /apps apps_1daoyun

Created snapshot /apps/.snapshot/apps_1daoyun

[hdfs@master ~]# hadoop fs -ls /apps/.snapshot

Found 1 items

drwxrwxrwx   - hdfs hdfs          0 2017-05-07 09:48 /apps/.snapshot/apps_1daoyun

4.为了防止操作人员误删文件，HDFS文件系统提供了回收站的功能，但过多的垃圾文件会占用大量的存储空间。要求在Linux Shell中使用“vi”命令修改相应的配置文件以及参数信息，关闭回收站功能。完成后，重启相应的服务。

答：

[root@master ~]# vi /etc/hadoop/ 2.6.1.0-129/0/hdfs-site.xml

<property>

<name>fs.trash.interval</name>

<value>0</value>

</property>

[root@master ~]# su - hdfs

Last login: Mon May  8 09:31:52 UTC 2017

[hdfs@master ~]$ /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf stop namenode

[hdfs@master ~]$ /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode

[hdfs@master ~]$ /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf stop datanode

[hdfs@master ~]$ /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode

5.使用命令查看hdfs文件系统中/tmp目录下的目录个数，文件个数和文件总大小。

答：

[root@master ~]# hadoop fs -count  /tmp

21            6               4336 /tmp

6.在集群节点中/usr/hdp/ 2.6.1.0-129/hadoop-mapreduce/目录下，存在一个案例JAR包hadoop-mapreduce-examples.jar。运行JAR包中的wordcount程序来对/1daoyun/file/BigDataSkills.txt文件进行单词计数，将运算结果输出到/1daoyun/output目录中，使用相关命令查询单词计数结果。

答：

[root@master ~]# hadoop jar /usr/hdp/ 2.6.1.0-129/hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3.2.6.1.0-129.jar wordcount /1daoyun/file/BigDataSkills.txt /1daoyun/output

[root@master ~]# hadoop fs -cat /1daoyun/output/part-r-00000

"duiya  1

hello   1

nisibusisha     1

wosha"  1

zsh     1

7.在集群节点中/usr/hdp/ 2.6.1.0-129/hadoop-mapreduce/目录下，存在一个案例JAR包hadoop-mapreduce-examples.jar。运行JAR包中的sudoku程序来计算下表中数独运算题的结果。

8
		3	6
	7			9		2
	5				7
				4	5	7
			1				3
		1					6	8
		8	5				1
	9					4
答： [root@master ~]# cat puzzle1.dta 8 ? ? ? ? ? ? ? ? ? ? 3 6 ? ? ? ? ? ? 7 ? ? 9 ? 2 ? ? ? 5 ? ? ? 7 ? ? ? ? ? ? ? 4 5 7 ? ? ? ? ? 1 ? ? ? 3 ? ? ? 1 ? ? ? ? 6 8 ? ? 8 5 ? ? ? 1 ? ? 9 ? ? ? ? 4 ? ? [root@master hadoop-mapreduce]# hadoop jar hadoop-mapreduce-examples-2.7.1.2.4.3.0-227.jar sudoku /root/puzzle1.dta WARNING: Use "yarn jar" to launch YARN applications. Solving /root/puzzle1.dta 8 1 2 7 5 3 6 4 9 9 4 3 6 8 2 1 7 5 6 7 5 4 9 1 2 8 3 1 5 4 2 3 7 8 9 6 3 6 9 8 4 5 7 2 1 2 8 7 1 6 9 5 3 4 5 2 1 9 7 4 3 6 8 4 3 8 5 2 6 9 1 7 7 9 6 3 1 8 4 5 2 Found 1 solutions

8.在集群节点中/usr/hdp/2.6.1.0-129/hadoop-mapreduce/目录下，存在一个案例JAR包hadoop-mapreduce-examples.jar。运行JAR包中的grep程序来统计文件系统中/1daoyun/file/BigDataSkills.txt文件中“Hadoop”出现的次数，统计完成后，查询统计结果信息。

答：

[root@master hadoop-mapreduce]# hadoop jar hadoop-mapreduce-examples-2.7.1.2.4.3.0-227.jar grep /1daoyun/file/BigDataSkills.txt /output hadoop

[root@master hadoop-mapreduce]# hadoop fs -cat /output/part-r-00000

2       hadoop

9.启动先电大数据平台的Hbase数据库，其中要求使用master节点的RegionServer。在Linux Shell中启动Hbase shell，查看进入HBase shell的当前系统用户。（相关数据库命令语言请全部使用小写格式）

答：

hbase(main):003:0> whoami

root (auth:SIMPLE)

groups: root

10.开启HBase的安全认证功能，在HBase Shell中设置root用户拥有表xiandian_user的读写与执行的权限，设置完成后，使用相关命令查看其权限信息。

答：

参数 Enable Authorization

参数值 native

hbase(main):002:0> grant 'root','RWX','xiandian_user'

0 row(s) in 0.4800 seconds

hbase(main):003:0> user_permission 'xiandian_user'

User                                             Namespace,Table,Family,Qualifier:Permission

root                                            default,xiandian_user,,: [Permission: actions=READ,WRITE,EXEC]

1 row(s) in 0.1180 seconds

11. 登录hbase数据库，创建一张表为member，列族为'address','info'，创建完之后，向该表插入数据，插入的数据为：

'xiandianA','info:age','24'

'xiandianA','info:birthday','1990-07-17'

'xiandianA','info:company','alibaba'

'xiandianA','address:contry','china'

'xiandianA','address:province','zhejiang'

'xiandianA','address:city','hangzhou'

插入完毕后，使用命令查询member表中xiandianA的所有info信息，最后将xiandianA的年龄改为99，并只查询info：age信息。

答：

hbase(main):001:0> create 'member','address','info'

0 row(s) in 1.5730 seconds

=> Hbase::Table - member

hbase(main):002:0> list

TABLE

emp

member

2 row(s) in 0.0240 seconds

hbase(main):007:0> put'member','xiandianA','info:age','24'

0 row(s) in 0.1000 seconds

hbase(main):008:0> put'member','xiandianA','info:birthday','1990-07-17'

0 row(s) in 0.0130 seconds

hbase(main):010:0> put'member','xiandianA','info:company','alibaba'

0 row(s) in 0.0080 seconds

hbase(main):011:0> put'member','xiandianA','address:contry','china'

0 row(s) in 0.0080 seconds

hbase(main):012:0> put'member','xiandianA','address:province','zhejiang'

0 row(s) in 0.0070 seconds

hbase(main):013:0> put'member','xiandianA','address:city','hangzhou'

0 row(s) in 0.0090 seconds

hbase(main):014:0> get 'member','xiandianA','info'

COLUMN                  CELL

info:age               timestamp=1522140592336, value=24

info:birthday          timestamp=1522140643072, value=1990-07-17

info:company           timestamp=1522140745172, value=alibaba

3 row(s) in 0.0170 seconds

hbase(main):015:0>

hbase(main):016:0* put 'member','xiandianA','info:age','99'

0 row(s) in 0.0080 seconds

hbase(main):018:0> get 'member','xiandianA','info:age'

COLUMN                  CELL

info:age               timestamp=1522141564423, value=99

1 row(s) in 0.0140 seconds

12.在关系数据库系统中，命名空间namespace是表的逻辑分组,同一组中的表有类似的用途。登录hbase数据库，新建一个命名空间叫newspace并用list查询，然后在这个命名空间中创建表member，列族为'address','info'，创建完之后，向该表插入数据，插入的数据为：

'xiandianA','info:age','24'

'xiandianA','info:birthday','1990-07-17'

'xiandianA','info:company','alibaba'

'xiandianA','address:contry','china'

'xiandianA','address:province','zhejiang'

'xiandianA','address:city','hangzhou'

插入完毕后，使用scan命令只查询表中info：age的信息，指定startrow为xiandianA。

答：

hbase(main):022:0> create_namespace 'newspace'

0 row(s) in 0.1130 seconds

hbase(main):024:0> list

TABLE

emp

member

newspace:member

3 row(s) in 0.0100 seconds

=> ["emp", "member", "newspace:member"]

hbase(main):023:0> create 'newspace:member','address','info'

0 row(s) in 1.5270 seconds

hbase(main):033:0> put 'newspace:member','xiandianA','info:age','24'

0 row(s) in 0.0620 seconds

hbase(main):037:0> put 'newspace:member','xiandianA','info:birthday','1990-07-17'

0 row(s) in 0.0110 seconds

hbase(main):038:0> put 'newspace:member','xiandianA','info:company','alibaba'

0 row(s) in 0.0130 seconds

hbase(main):039:0> put 'newspace:member','xiandianA','address:contry','china'

0 row(s) in 0.0070 seconds

hbase(main):040:0> put 'newspace:member','xiandianA','address:province','zhejiang'

0 row(s) in 0.0070 seconds

hbase(main):041:0> put 'newspace:member','xiandianA','address:city','hangzhou'

0 row(s) in 0.0070 seconds

hbase(main):044:0> scan 'newspace:member', {COLUMNS => ['info:age'],STARTROW => 'xiandianA'}

ROW                                              COLUMN+CELL

xiandianA                                       column=info:age, timestamp=1522214952401, value=24

1 row(s) in 0.0160 seconds

13.登录master节点，在本地新建一个文件叫hbasetest.txt文件，编写内容，要求新建一张表为'test', 列族为'cf'，然后向这张表批量插入数据，数据如下所示：

'row1', 'cf:a', 'value1'

'row2', 'cf:b', 'value2'

'row3', 'cf:c', 'value3'

'row4', 'cf:d', 'value4'

在插入数据完毕后用scan命令查询表内容，然后用get命令只查询row1的内容，最后退出hbase shell。使用命令运行hbasetest.txt，将hbasetest.txt的内容和执行命令后的返回结果提交。

答：

[root@exam1 ~]# cat hbasetest.txt

create 'test', 'cf'

list 'test'

put 'test', 'row1', 'cf:a', 'value1'

put 'test', 'row2', 'cf:b', 'value2'

put 'test', 'row3', 'cf:c', 'value3'

put 'test', 'row4', 'cf:d', 'value4'

scan 'test'

get 'test', 'row1'

exit

[root@exam1 ~]# hbase shell hbasetest.txt

0 row(s) in 1.5010 seconds

TABLE

test

1 row(s) in 0.0120 seconds

0 row(s) in 0.1380 seconds

0 row(s) in 0.0090 seconds

0 row(s) in 0.0050 seconds

0 row(s) in 0.0050 seconds

ROW                     COLUMN+CELL

row1                   column=cf:a, timestamp=1522314428726, value=value1

row2                   column=cf:b, timestamp=1522314428746, value=value2

row3                   column=cf:c, timestamp=1522314428752, value=value3

row4                   column=cf:d, timestamp=1522314428758, value=value4

4 row(s) in 0.0350 seconds

COLUMN                  CELL

cf:a                   timestamp=1522314428726, value=value1

1 row(s) in 0.0190 seconds

14.使用Hive工具来创建数据表xd_phy_course，并定义该表为外部表，外部存储位置为/1daoyun/data/hive，将phy_course_xd.txt导入到该表中，其中xd_phy_course表的数据结构如下表所示。导入完成后，在hive中查询数据表xd_phy_course的数据结构信息。（相关数据库命令语言请全部使用小写格式）

stname(string)	stID(int)	class(string)	opt_cour(string)
答： hive> create external table xd_phy_course (stname string,stID int,class string,opt_cour string) row format delimited fields terminated by '\t' lines terminated by '\n' location '/1daoyun/data/hive'; OK Time taken: 1.197 seconds hive> load data local inpath '/root/phy_course_xd.txt' into table xd_phy_course; Loading data to table default.xd_phy_course Table default.xd_phy_course stats: [numFiles=1, totalSize=89444] OK Time taken: 0.96 seconds hive> desc xd_phy_course2; OK stname                  string stid                    int class                   string opt_cour                string Time taken: 0.588 seconds, Fetched: 4 row(s)

15.使用Hive工具来统计phy_course_xd.txt文件中某高校报名选修各个体育科目的总人数，其中phy_course_xd.txt文件数据结构如下表所示，选修科目字段为opt_cour，将统计的结果导入到表phy_opt_count中，通过SELECT语句查询表phy_opt_count内容。（相关数据库命令语言请全部使用小写格式）

stname(string)	stID(int)	class(string)	opt_cour(string)
答： hive> create table xd_phy_course (stname string,stID int,class string,opt_cour string) row format delimited fields terminated by '\t' lines terminated by '\n'; OK Time taken: 4.067 seconds hive> load data local inpath '/root/phy_course_xd.txt' into table xd_phy_course; Loading data to table default.xd_phy_course Table default.xd_phy_course stats: [numFiles=1, totalSize=89444] OK Time taken: 1.422 seconds hive> create table phy_opt_count (opt_cour string,cour_count int) row format delimited fields terminated by '\t' lines terminated by '\n'; OK Time taken: 1.625 seconds hive> insert overwrite table phy_opt_count select xd_phy_course.opt_cour,count(distinct xd_phy_course.stID) from xd_phy_course group by xd_phy_course.opt_cour; Query ID = root_20170507125642_6af22d21-ae88-4daf-a346-4b1cbcd7d9fe Total jobs = 1 Launching Job 1 out of 1 Tez session was closed. Reopening... Session re-established. Status: Running (Executing on YARN cluster with App id application_1494149668396_0004) -------------------------------------------------------------------------------- VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED -------------------------------------------------------------------------------- Map 1 ..........   SUCCEEDED      1          1        0        0       0       0 Reducer 2 ......   SUCCEEDED      1          1        0        0       0       0 -------------------------------------------------------------------------------- VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 4.51 s -------------------------------------------------------------------------------- Loading data to table default.phy_opt_count Table default.phy_opt_count stats: [numFiles=1, numRows=10, totalSize=138, rawDataSize=128] OK Time taken: 13.634 seconds hive> select * from phy_opt_count; OK badminton       234 basketball      224 football        206 gymnastics      220 opt_cour        0 swimming        234 table tennis    277 taekwondo       222 tennis  223 volleyball      209 Time taken: 0.065 seconds, Fetched: 10 row(s)

16.使用Hive工具来统计phy_course_score_xd.txt文件中某高校各个班级体育课的平均成绩，使用round函数保留两位小数。其中phy_course_score_xd.txt文件数据结构如下表所示，班级字段为class，成绩字段为score。（相关数据库命令语言请全部使用小写格式）

stname(string)	stID(int)	class(string)	opt_cour(string)	score(float)
答： hive> create table phy_course_score_xd (stname string,stID int,class string,opt_cour string,score float) row format delimited fields terminated by '\t' lines terminated by '\n'; OK Time taken: 0.339 seconds hive> load data local inpath '/root/phy_course_score_xd.txt' into table phy_course_score_xd; Loading data to table default.phy_course_score_xd Table default.phy_course_score_xd stats: [numFiles=1, totalSize=1910] OK Time taken: 1.061 seconds hive> select class,round(avg(score)) from phy_course_score_xd group by class; Query ID = root_20170507131823_0bfb1faf-3bfb-42a5-b7eb-3a6a284081ae Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1494149668396_0005) -------------------------------------------------------------------------------- VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED -------------------------------------------------------------------------------- Map 1 ..........   SUCCEEDED      1          1        0        0       0       0 Reducer 2 ......   SUCCEEDED      1          1        0        0       0       0 -------------------------------------------------------------------------------- VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 26.68 s -------------------------------------------------------------------------------- OK Network_1401    73.0 Software_1403   72.0 class   NULL Time taken: 27.553 seconds, Fetched: 3 row(s)

17.使用Hive工具来统计phy_course_score_xd.txt文件中某高校各个班级体育课的最高成绩。其中phy_course_score_xd.txt文件数据结构如下表所示，班级字段为class，成绩字段为score。（相关数据库命令语言请全部使用小写格式）

stname(string)	stID(int)	class(string)	opt_cour(string)	score(float)
答： hive> create table phy_course_score_xd (stname string,stID int,class string,opt_cour string,score float) row format delimited fields terminated by '\t' lines terminated by '\n'; OK Time taken: 0.339 seconds hive> load data local inpath '/root/phy_course_score_xd.txt' into table phy_course_score_xd; Loading data to table default.phy_course_score_xd Table default.phy_course_score_xd stats: [numFiles=1, totalSize=1910] OK Time taken: 1.061 seconds hive> select class,max(score) from phy_course_score_xd group by class; Query ID = root_20170507131942_86a2bf55-49ac-4c2e-b18b-8f63191ce349 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1494149668396_0005) -------------------------------------------------------------------------------- VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED -------------------------------------------------------------------------------- Map 1 ..........   SUCCEEDED      1          1        0        0       0       0 Reducer 2 ......   SUCCEEDED      1          1        0        0       0       0 -------------------------------------------------------------------------------- VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 5.08 s -------------------------------------------------------------------------------- OK Network_1401    95.0 Software_1403   100.0 class   NULL Time taken: 144.035 seconds, Fetched: 3 row(s)

18.在Hive数据仓库将网络日志weblog_entries.txt中分开的request_date和request_time字段进行合并，并以一个下划线“_”进行分割，如下图所示，其中weblog_entries.txt的数据结构如下表所示。（相关数据库命令语言请全部使用小写格式）

md5(STRING)	url(STRING)	request_date (STRING)	request_time (STRING)	ip(STRING)
答： hive> create table weblog_entries (md5 string,url string,request_date string,request_time string,ip string) row format delimited fields terminated by '\t' lines terminated by '\n'; OK Time taken: 0.502 seconds hive> load data local inpath '/root/weblog_entries.txt' into table weblog_entries; Loading data to table default.weblog_entries Table default.weblog_entries stats: [numFiles=1, totalSize=251130] OK Time taken: 1.203 seconds hive> select concat_ws('_', request_date, request_time) from weblog_entries; 2012-05-10_21:29:01 2012-05-10_21:13:47 2012-05-10_21:12:37 2012-05-10_21:34:20 2012-05-10_21:27:00 2012-05-10_21:33:53 2012-05-10_21:10:19 2012-05-10_21:12:05 2012-05-10_21:25:58 2012-05-10_21:34:28 Time taken: 0.265 seconds, Fetched: 3000 row(s)

19. 使用Hive动态地关于网络日志weblog_entries.txt的查询结果创建Hive表。通过创建一张名为weblog_entries_url_length的新表来定义新的网络日志数据库的三个字段，分别是url，request_date，request_time。此外，在表中定义一个获取url字符串长度名为“url_length”的新字段，其中weblog_entries.txt的数据结构如下表所示。完成后查询weblog_entries_url_length表文件内容。（相关数据库命令语言请全部使用小写格式）

md5(STRING)	url(STRING)	request_date (STRING)	request_time (STRING)	ip(STRING)
答： hive> create table weblog_entries (md5 string,url string,request_date string,request_time string,ip string) row format delimited fields terminated by '\t' lines terminated by '\n'; OK Time taken: 0.502 seconds hive> load data local inpath '/root/weblog_entries.txt' into table weblog_entries; Loading data to table default.weblog_entries Table default.weblog_entries stats: [numFiles=1, totalSize=251130] OK Time taken: 1.203 seconds hive> create table weblog_entries_url_length as select url, request_date, request_time, length(url) as url_length from weblog_entries; Query ID = root_20170507065123_e3105d8b-84b6-417f-ab58-21ea15723e0a Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1494136863427_0002) -------------------------------------------------------------------------------- VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED -------------------------------------------------------------------------------- Map 1 ..........   SUCCEEDED      1          1        0        0       0       0 -------------------------------------------------------------------------------- VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 4.10 s -------------------------------------------------------------------------------- Moving data to: hdfs://master:8020/apps/hive/warehouse/weblog_entries_url_length Table default.weblog_entries_url_length stats: [numFiles=1, numRows=3000, totalSize=121379, rawDataSize=118379] OK Time taken: 5.874 seconds hive> select * from weblog_entries_url_length; /qnrxlxqacgiudbtfggcg.html      2012-05-10      21:29:01        26 /sbbiuot.html   2012-05-10      21:13:47        13 /ofxi.html      2012-05-10      21:12:37        10 /hjmdhaoogwqhp.html     2012-05-10      21:34:20        19 /angjbmea.html  2012-05-10      21:27:00        14 /mmdttqsnjfifkihcvqu.html       2012-05-10      21:33:53        25 /eorxuryjadhkiwsf.html  2012-05-10      21:10:19        22 /e.html 2012-05-10      21:12:05        7 /khvc.html      2012-05-10      21:25:58        10 /c.html 2012-05-10      21:34:28        7 Time taken: 0.08 seconds, Fetched: 3000 row(s)

20.在master和slaver节点安装Sqoop Clients，完成后，在master节点查看Sqoop的版本信息。

答：

[root@master ~]# sqoop version

Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

17/05/07 06:56:25 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.3.0-227

Sqoop 1.4.6.2.4.3.0-227

git commit id d296ad374bd38a1c594ef0f5a2d565d71e798aa6

Compiled by jenkins on Sat Sep 10 00:58:52 UTC 2016

21.使用Sqoop工具列出master节点中MySQL中ambari数据库中所有的数据表。

答：

[root@master ~]# sqoop list-tables --connect jdbc:mysql://localhost/ambari --username root --password bigdata

Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

17/05/07 07:07:01 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.3.0-227

17/05/07 07:07:01 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

17/05/07 07:07:02 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

ClusterHostMapping

QRTZ_BLOB_TRIGGERS

QRTZ_CALENDARS

QRTZ_CRON_TRIGGERS

QRTZ_FIRED_TRIGGERS

QRTZ_JOB_DETAILS

QRTZ_LOCKS

QRTZ_PAUSED_TRIGGER_GRPS

QRTZ_SCHEDULER_STATE

QRTZ_SIMPLE_TRIGGERS

QRTZ_SIMPROP_TRIGGERS

QRTZ_TRIGGERS

adminpermission

adminprincipal

adminprincipaltype

adminprivilege

adminresource

adminresourcetype

alert_current

alert_definition

alert_group

alert_group_target

alert_grouping

alert_history

alert_notice

alert_target

alert_target_states

ambari_sequences

artifact

blueprint

blueprint_configuration

clusterEvent

cluster_version

clusterconfig

clusterconfigmapping

clusters

clusterservices

clusterstate

confgroupclusterconfigmapping

configgroup

configgrouphostmapping

execution_command

groups

hdfsEvent

host_role_command

host_version

hostcomponentdesiredstate

hostcomponentstate

hostconfigmapping

hostgroup

hostgroup_component

hostgroup_configuration

hosts

hoststate

job

kerberos_descriptor

kerberos_principal

kerberos_principal_host

key_value_store

mapreduceEvent

members

metainfo

repo_version

request

requestoperationlevel

requestresourcefilter

requestschedule

requestschedulebatchrequest

role_success_criteria

servicecomponentdesiredstate

serviceconfig

serviceconfighosts

serviceconfigmapping

servicedesiredstate

stack

stage

task

taskAttempt

topology_host_info

topology_host_request

topology_host_task

topology_hostgroup

topology_logical_request

topology_logical_task

topology_request

upgrade

upgrade_group

upgrade_item

users

viewentity

viewinstance

viewinstancedata

viewinstanceproperty

viewmain

viewparameter

viewresource

widget

widget_layout

widget_layout_user_widget

workflow

22.在MySQL中创建名为xiandian的数据库，在xiandian数据库中创建xd_phy_course数据表，其数据表结构如表1所示。使用Hive工具来创建数据表xd_phy_course，将phy_course_xd.txt导入到该表中，其中xd_phy_course表的数据结构如表2所示。使用Sqoop工具将hive数据仓库中的xd_phy_course表导出到master节点的MySQL中xiandain数据库的xd_phy_course表。

表1

stname VARCHAR(20)

stID INT(1)

class VARCHAR(20)

opt_cour VARCHAR(20)

表2

stname(string)	stID(int)	class(string)	opt_cour(string)
答： [root@master ~]# mysql -uroot -pbigdata Welcome to the MariaDB monitor.  Commands end with ; or \g. Your MariaDB connection id is 37 Server version: 5.5.44-MariaDB MariaDB Server Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database xiandian; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> use xiandian; Database changed MariaDB [xiandian]> create table xd_phy_course(stname varchar(20),stID int(1),class varchar(20),opt_cour varchar(20)); Query OK, 0 rows affected (0.20 sec) hive> create table xd_phy_course (stname string,stID int,class string,opt_cour string) row format delimited fields terminated by '\t' lines terminated by '\n'; OK Time taken: 3.136 seconds hive> load data local inpath '/root/phy_course_xd.txt' into table xd_phy_course3; Loading data to table default.xd_phy_course3 Table default.xd_phy_course3 stats: [numFiles=1, totalSize=89444] OK Time taken: 1.129 seconds [root@master ~]# sqoop export --connect jdbc:mysql://localhost:3306/xiandian --username root --password bigdata --table xd_phy_course  --hcatalog-table xd_phy_course Warning: /usr/hdp/2.4.3.0-227/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 17/05/07 07:29:48 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.4.3.0-227 17/05/07 07:29:48 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 17/05/07 07:29:48 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 17/05/07 07:29:48 INFO tool.CodeGenTool: Beginning code generation 17/05/07 07:29:48 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1 17/05/07 07:29:48 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `xd_phy_course` AS t LIMIT 1 17/05/07 07:29:48 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.4.3.0-227/hadoop-mapreduce Note: /tmp/sqoop-root/compile/35d4b31b4d93274ba6bde54b3e56a821/xd_phy_course.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 17/05/07 07:29:50 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/35d4b31b4d93274ba6bde54b3e56a821/xd_phy_course.jar 17/05/07 07:29:50 INFO mapreduce.ExportJobBase: Beginning export of xd_phy_course SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.4.3.0-227/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

23.使用Pig工具在Local模式计算系统日志access-log.txt中的IP的点击数，要求使用GROUP BY语句按照IP进行分组，通过FOREACH 运算符，对关系的列进行迭代，统计每个分组的总行数，最后使用DUMP语句查询统计结果。

答：

grunt> copyFromLocal /root/Pig/access-log.txt /user/root/input/log1.txt

grunt> A =LOAD '/user/root/input/log1.txt' USING PigStorage (' ') AS (ip,others);

grunt> group_ip =group A by ip;

grunt> result =foreach group_ip generate group,COUNT(A);

grunt> dump result;

2018-02-13 08:13:36,520 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

2.7.3.2.6.1.0-129 0.16.0.2.6.1.0-129 root 2018-02-13 08:13:37 2018-02-13 08:13:41 GROUP_BY

Success!

Job Stats (time in seconds):

JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature Outputs

job_local963723433_0001 1 1 n/a n/a n/a n/a n/a n/a n/a n/a A,group_ip,result GROUP_BY,COMBINER file:/tmp/temp-1479363025/tmp133834330,

Input(s):

Successfully read 62991 records from: "/user/root/input/log1.txt"

Output(s):

Successfully stored 182 records in: "file:/tmp/temp-1479363025/tmp133834330"

(220.181.108.186,1)

(222.171.234.225,142)

(http://www.1daoyun.com/course/toregeister",1)

24.使用Pig工具计算天气数据集temperature.txt中年度最高气温，要求使用GROUP BY语句按照year进行分组，通过FOREACH 运算符，对关系的列进行迭代，统计每个分组的最大值，最后使用DUMP语句查询计算结果。

答：

grunt> copyFromLocal /root/Pig/temperature.txt /user/root/temp.txt

grunt> A = LOAD '/user/root/temp.txt' USING PigStorage(' ')AS (year:int,temperature:int);

grunt> B = GROUP A BY year;

grunt> C = FOREACH B GENERATE group,MAX(A.temperature);

grunt> dump C;

2018-02-13 08:18:52,107 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY

(2012,40)

(2013,36)

(2014,37)

(2015,39)

25.使用Pig工具统计数据集ip_to_country中每个国家的IP地址数。要求使用GROUP BY语句按照国家进行分组，通过FOREACH 运算符，对关系的列进行迭代，统计每个分组的IP地址数目，最后将统计结果保存到/data/pig/output目录中，并查看数据结果。

答：

grunt> copyFromLocal /root/Pig/ip_to_country.txt /user/root/ip_to_country.txt

grunt> ip_countries = LOAD '/user/root/ip_to_country.txt' AS (ip: chararray, country:chararray);

grunt> country_grpd = GROUP ip_countries BY country;

grunt> country_counts = FOREACH country_grpd GENERATE FLATTEN(group),COUNT(ip_countries) as counts;

grunt> STORE country_counts INTO '/data/pig/output';

2018-02-13 08:23:35,621 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY

Moldova, Republic of 1

Syrian Arab Republic 1

United Arab Emirates 2

Bosnia and Herzegovina 1

Iran, Islamic Republic of 2

Tanzania, United Republic of 1

26.在master节点安装Mahout Client，打开Linux Shell运行mahout命令查看Mahout自带的案例程序。

答：

[root@master ~]# mahout

MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.

Running on hadoop, using /usr/hdp/2.6.1.0-129/hadoop/bin/hadoop and HADOOP_CONF_DIR=/usr/hdp/2.6.1.0-129/hadoop/conf

MAHOUT-JOB: /usr/hdp/2.6.1.0-129/mahout/mahout-examples-0.9.0.2.6.1.0-129-job.jar

An example program must be given as the first argument.

Valid program names are:

arff.vector: : Generate Vectors from an ARFF file or directory

baumwelch: : Baum-Welch algorithm for unsupervised HMM training

buildforest: : Build the random forest classifier

canopy: : Canopy clustering

cat: : Print a file or resource as the logistic regression models would see it

cleansvd: : Cleanup and verification of SVD output

clusterdump: : Dump cluster output to text

clusterpp: : Groups Clustering Output In Clusters

cmdump: : Dump confusion matrix in HTML or text formats

concatmatrices: : Concatenates 2 matrices of same cardinality into a single matrix

cvb: : LDA via Collapsed Variation Bayes (0th deriv. approx)

cvb0_local: : LDA via Collapsed Variation Bayes, in memory locally.

describe: : Describe the fields and target variable in a data set

evaluateFactorization: : compute RMSE and MAE of a rating matrix factorization against probes

fkmeans: : Fuzzy K-means clustering

hmmpredict: : Generate random sequence of observations by given HMM

itemsimilarity: : Compute the item-item-similarities for item-based collaborative filtering

kmeans: : K-means clustering

lucene.vector: : Generate Vectors from a Lucene index

lucene2seq: : Generate Text SequenceFiles from a Lucene index

matrixdump: : Dump matrix in CSV format

matrixmult: : Take the product of two matrices

parallelALS: : ALS-WR factorization of a rating matrix

qualcluster: : Runs clustering experiments and summarizes results in a CSV

recommendfactorized: : Compute recommendations using the factorization of a rating matrix

recommenditembased: : Compute recommendations using item-based collaborative filtering

regexconverter: : Convert text files on a per line basis based on regular expressions

resplit: : Splits a set of SequenceFiles into a number of equal splits

rowid: : Map SequenceFile<Text,VectorWritable> to {SequenceFile<IntWritable,VectorWritable>, SequenceFile<IntWritable,Text>}

rowsimilarity: : Compute the pairwise similarities of the rows of a matrix

runAdaptiveLogistic: : Score new production data using a probably trained and validated AdaptivelogisticRegression model

runlogistic: : Run a logistic regression model against CSV data

seq2encoded: : Encoded Sparse Vector generation from Text sequence files

seq2sparse: : Sparse Vector generation from Text sequence files

seqdirectory: : Generate sequence files (of Text) from a directory

seqdumper: : Generic Sequence File dumper

seqmailarchives: : Creates SequenceFile from a directory containing gzipped mail archives

seqwiki: : Wikipedia xml dump to sequence file

spectralkmeans: : Spectral k-means clustering

split: : Split Input data into test and train sets

splitDataset: : split a rating dataset into training and probe parts

ssvd: : Stochastic SVD

streamingkmeans: : Streaming k-means clustering

svd: : Lanczos Singular Value Decomposition

testforest: : Test the random forest classifier

testnb: : Test the Vector-based Bayes classifier

trainAdaptiveLogistic: : Train an AdaptivelogisticRegression model

trainlogistic: : Train a logistic regression using stochastic gradient descent

trainnb: : Train the Vector-based Bayes classifier

transpose: : Take the transpose of a matrix

validateAdaptiveLogistic: : Validate an AdaptivelogisticRegression model against hold-out data set

vecdist: : Compute the distances between a set of Vectors (or Cluster or Canopy, they must fit in memory) and a list of Vectors

vectordump: : Dump vectors from a sequence file to text

viterbi: : Viterbi decoding of hidden states from given output states sequence

27.使用Mahout挖掘工具对数据集user-item-score.txt（用户-物品-得分）进行物品推荐，要求采用基于项目的协同过滤算法，欧几里得距离公式定义，并且每位用户的推荐个数为3，设置非布尔数据，最大偏好值为4，最小偏好值为1，将推荐输出结果保存到output目录中，通过-cat命令查询输出结果part-r-00000中的内容 。

答：

[hdfs@master ~]$ hadoop fs -mkdir -p /data/mahout/project

[hdfs@master ~]$ hadoop fs -put user-item-score.txt /data/mahout/project

[hdfs@master ~]$ mahout recommenditembased -i /data/mahout/project/ user-item-score.txt -o /data/mahout/project/output -n 3 -b false -s SIMILARITY_EUCLIDEAN_DISTANCE --maxPrefsPerUser 4 --minPrefsPerUser 1 --maxPrefsInItemSimilarity 4 --tempDir /data/mahout/project/temp

MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.

Running on hadoop, using /usr/hdp/2.4.3.0-227/hadoop/bin/hadoop and

17/05/15 19:37:25 INFO driver.MahoutDriver: Program took 259068 ms (Minutes: 4.3178)

[hdfs@master ~]$ hadoop fs -cat /data/mahout/project/output/part-r-00000

1       [105:3.5941463,104:3.4639049]

2       [106:3.5,105:2.714964,107:2.0]

3       [103:3.59246,102:3.458911]

4       [107:4.7381864,105:4.2794304,102:4.170158]

5       [103:3.8962872,102:3.8564017,107:3.7692602]

28.在master节点安装启动Flume组件，打开Linux Shell运行flume-ng的帮助命令，查看Flume-ng的用法信息。

答：

[root@master ~]# flume-ng help

Usage: /usr/hdp/2.6.1.0-129/flume/bin/flume-ng.distro <command> [options]...

commands:

help                  display this help text

agent                 run a Flume agent

avro-client           run an avro Flume client

password              create a password file for use in flume config

version               show Flume version info

global options:

--conf,-c <conf>      use configs in <conf> directory

--classpath,-C <cp>   append to the classpath

--dryrun,-d           do not actually start Flume, just print the command

--plugins-path <dirs> colon-separated list of plugins.d directories. See the

plugins.d section in the user guide for more details.

Default: $FLUME_HOME/plugins.d

-Dproperty=value      sets a Java system property value

-Xproperty=value      sets a Java -X option

agent options:

--conf-file,-f <file> specify a config file (required)

--name,-n <name>      the name of this agent (required)

--help,-h             display help text

avro-client options:

--rpcProps,-P <file>   RPC client properties file with server connection params

--host,-H <host>       hostname to which events will be sent

--port,-p <port>       port of the avro source

--dirname <dir>        directory to stream to avro source

--filename,-F <file>   text file to stream to avro source (default: std input)

--headerFile,-R <file> File containing event headers as key/value pairs on each new line

--help,-h              display help text

Either --rpcProps or both --host and --port must be specified.

password options:

--outfile              The file in which encoded password is stored

Note that if <conf> directory is specified, then it is always included first

in the classpath.

29. 根据提供的模板hdfs-example.conf文件，使用Flume NG工具设置master节点的系统路径/opt/xiandian/为实时上传文件至HDFS文件系统的实时路径，设置HDFS文件系统的存储路径为/data/flume/，上传后的文件名保持不变，文件类型为DataStream，然后启动flume-ng agent。

答：

[root@master ~]# flume-ng agent --conf-file hdfs-example.conf --name master -Dflume.root.logger=INFO,cnsole

Warning: No configuration directory set! Use --conf <dir> to override.

Info: Including Hadoop libraries found via (/bin/hadoop) for HDFS access

Info: Excluding /usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-api-1.7.10.jar from classpath

Info: Excluding /usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-log4j12-1.7.10.jar from classpath

Info: Excluding /usr/hdp/2.4.3.0-227/tez/lib/slf4j-api-1.7.5.jar from classpath

Info: Including HBASE libraries found via (/bin/hbase) for HBASE access

Info: Excluding /usr/hdp/2.4.3.0-227/hbase/lib/slf4j-api-1.7.7.jar from classpath

Info: Excluding /usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-api-1.7.10.jar from classpath

Info: Excluding /usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-log4j12-1.7.10.jar from classpath

Info: Excluding /usr/hdp/2.4.3.0-227/tez/lib/slf4j-api-1.7.5.jar from classpath

Info: Excluding /usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-api-1.7.10.jar from classpath

Info: Excluding /usr/hdp/2.4.3.0-227/hadoop/lib/slf4j-log4j12-1.7.10.jar from classpath

Info: Excluding /usr/hdp/2.4.3.0-227/zookeeper/lib/slf4j-api-1.6.1.jar from classpath

Info: Excluding /usr/hdp/2.4.3.0-227/zookeeper/lib/slf4j-log4j12-1.6.1.jar from classpath

Info: Including Hive libraries found via () for Hive access

[root@master ~]# cat hdfs-example.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent

master.sources = webmagic

master.sinks = k1

master.channels = c1

# Describe/configure the source

master.sources.webmagic.type = spooldir

master.sources.webmagic.fileHeader = true

master.sources.webmagic.fileHeaderKey = fileName

master.sources.webmagic.fileSuffix = .COMPLETED

master.sources.webmagic.deletePolicy = never

master.sources.webmagic.spoolDir = /opt/xiandian/

master.sources.webmagic.ignorePattern = ^$

master.sources.webmagic.consumeOrder = oldest

master.sources.webmagic.deserializer = org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder

master.sources.webmagic.batchsize = 5

master.sources.webmagic.channels = c1

# Use a channel which buffers events in memory

master.channels.c1.type = memory

# Describe the sink

master.sinks.k1.type = hdfs

master.sinks.k1.channel = c1

master.sinks.k1.hdfs.path = hdfs://master:8020/data/flume/%{dicName}

master.sinks.k1.hdfs.filePrefix = %{fileName}

master.sinks.k1.hdfs.fileType = DataStream

30.在先电大数据平台部署Spark服务组件，打开Linux Shell启动spark-shell终端，将启动的程序进程信息提交。

答：

[root@master ~]# spark-shell

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

Spark context Web UI available at http://172.24.2.110:4040

Spark context available as 'sc' (master = local[*], app id = local-1519375873795).

Spark session available as 'spark'.

Welcome to

____              __

/ __/__  ___ _____/ /__

_\ \/ _ \/ _ `/ __/  '_/

/___/ .__/\_,_/_/ /_/\_\   version 2.1.1.2.6.1.0-129

/_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77)

Type in expressions to have them evaluated.

Type :help for more information.

scala>

31.登录spark-shell，定义i值为1，sum值为0，使用while循环，求从1加到100的值，最后使用scala的标准输出函数输出sum值。

答：

scala> var i=1

i: Int = 1

scala> var sum=0

sum: Int = 0

scala> while(i<=100){

| sum+=i

| i=i+1

| }

scala> println(sum)

5050

32.登录spark-shell，定义一个list为（1,2,3,4,5,6,7,8,9），然后利用map函数，对这个list进行元素乘2的操作。

答：

scala> import scala.math._

import scala.math._

scala> val nums=List(1,2,3,4,5,6,7,8,9)

nums: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9)

scala> nums.map(x=>x*2)

res18: List[Int] = List(2, 4, 6, 8, 10, 12, 14, 16, 18)

33.登录spark-shell，定义一个list为（"Hadoop","Java","Spark"），然后利用flatmap函数将list转换为单个字母并转换为大写。

答：

scala> val data = List("Hadoop","Java","Spark")

data: List[String] = List(Hadoop, Java, Spark)

scala> println(data.flatMap(_.toUpperCase))

List(H, A, D, O, O, P, J, A, V, A, S, P, A, R, K)

34.登录大数据云主机master节点，在root目录下新建一个abc.txt，内容为：

hadoop  hive

solr    redis

kafka   hadoop

storm   flume

sqoop   docker

spark   spark

hadoop  spark

elasticsearch   hbase

hadoop  hive

spark   hive

hadoop  spark

然后登录spark-shell，首先使用命令统计abc.txt的行数，接着对abc.txt文档中的单词进行计数，并按照单词首字母的升序进行排序，最后统计结果行数。

答：

scala> val words=sc.textFile("file:///root/abc.txt").count

words: Long = 11

scala> val words=sc.textFile("file:///root/abc.txt").flatMap(_.split("\\W+")).map(x=>(x,1)).reduceByKey(_+_).sortByKey().collect

words: Array[(String, Int)] = Array((docker,1), (elasticsearch,1), (flume,1), (hadoop,5), (hbase,1), (hive,3), (kafka,1), (redis,1), (solr,1), (spark,5), (sqoop,1), (storm,1))

scala> val words=sc.textFile("file:///root/abc.txt").flatMap(_.split("\\W+")).map(x=>(x,1)).reduceByKey(_+_).count

words: Long = 12

35. 登录spark-shell，定义一个List(1,2,3,3,4,4,5,5,6,6,6,8,9)，使用spark自带函数对这个list进行去重操作。

答：

scala> val l = List(1,2,3,3,4,4,5,5,6,6,6,8,9)

l: List[Int] = List(1, 2, 3, 3, 4, 4, 5, 5, 6, 6, 6, 8, 9)

scala> l.distinct

res1: List[Int] = List(1, 2, 3, 4, 5, 6, 8, 9)