-cacheFile 分发,文件事先上传至Hdfs上,分发的是一个文件

1.找一篇文章The_Man_of_Property.txt:

He was proud of him! He could not but feel that in similar circumstances he himself would have been tempted to enlarge his replies, but his instinct told him that this taciturnity was the very thing. He sighed with relief, however, when Soames, slowly turning, and without any change of expression, descended from the box.
When it came to the turn of Bosinney’s Counsel to address the Judge, James redoubled his attention, and he searched the Court again and again to see if Bosinney were not somewhere concealed.
Young Chankery began nervously; he was placed by Bosinney’s absence in an awkward position. He therefore did his best to turn that absence to account.
He could not but fear — he said — that his client had met with an accident. He had fully expected him there to give evidence; they had sent round that morning both to Mr. Bosinney’s office and to his rooms (though he knew they were one and the same, he thought it was as well not to say so), but it was not known where he was, and this he considered to be ominous, knowing how anxious Mr. Bosinney had been to give his evidence. He had not, however, been instructed to apply for an adjournment, and in default of such instruction he conceived it his duty to go on. The plea on which he somewhat confidently relied, and which his client, had he not unfortunately been prevented in some way from attending, would have supported by his evidence, was that such an expression as a ‘free hand’ could not be limited, fettered, and rendered unmeaning, by any verbiage which might follow it. He would go further and say that the correspondence showed that whatever he might have said in his evidence, Mr. Forsyte had in fact never contemplated repudiating liability on any of the work ordered or executed by his architect. The defendant had certainly never contemplated such a contingency, or, as was demonstrated by his letters, he would never have proceeded with the work — a work of extreme delicacy, carried out with great care and efficiency, to meet and satisfy the fastidious taste of a connoisseur, a rich man, a man of property. He felt strongly on this point, and feeling strongly he used, perhaps, rather strong words when he said that this action was of a most unjustifiable, unexpected, indeed — unprecedented character. If his Lordship had had the opportunity that he himself had made it his duty to take, to go over this very fine house and see the great delicacy and beauty of the decorations executed by his client — an artist in his most honourable profession — he felt convinced that not for one moment would his Lordship tolerate this, he would use no stronger word than daring attempt to evade legitimate responsibility.
Taking the text of Soames’ letters, he lightly touched on ‘Boileau v. The Blasted Cement Company, Limited.’ “It is doubtful,” he said, “what that authority has decided; in any case I would submit that it is just as much in my favour as in my friend’s.” He then argued the ‘nice point’ closely. With all due deference he submitted that Mr. Forsyte’s expression nullified itself. His client not being a rich man, the matter was a serious one for him; he was a very talented architect, whose professional reputation was undoubtedly somewhat at stake. He concluded with a perhaps too personal appeal to the Judge, as a lover of the arts, to show himself the protector of artists, from what was occasionally — he said occasionally — the too iron hand of capital. “What,” he said, “will be the position of the artistic professions, if men of property like this Mr. Forsyte refuse, and are allowed to refuse, to carry out the obligations of the commissions which they have given.” He would now call his client, in case he should at the last moment have found himself able to be present.
The name Philip Baynes Bosinney was called three times by the Ushers, and the sound of the calling echoed with strange melancholy throughout the Court and Galleries.
The crying of this name, to which no answer was returned, had upon James a curious effect: it was like calling for your lost dog about the streets. And the creepy feeling that it gave him, of a man missing, grated on his sense of comfort and security-on his cosiness. Though he could not have said why, it made him feel uneasy.
He looked now at the clock — a quarter to three! It would be all over in a quarter of an hour. Where could the young fellow be?
It was only when Mr. Justice Bentham delivered judgment that he got over the turn he had received.
Behind the wooden erection, by which he was fenced from more ordinary mortals, the learned Judge leaned forward. The electric light, just turned on above his head, fell on his face, and mellowed it to an orange hue beneath the snowy crown of his wig; the amplitude of his robes grew before the eye; his whole figure, facing the comparative dusk of the Court, radiated like some majestic and sacred body. He cleared his throat, took a sip of water, broke the nib of a quill against the desk, and, folding his bony hands before him, began.
A divorce! Thus close, the word was paralyzing, so utterly at variance with all the principles that had hitherto guided his life. Its lack of compromise appalled him; he felt — like the captain of a ship, going to the side of his vessel, and, with his own hands throwing over the most precious of his bales. This jettisoning of his property with his own hand seemed uncanny to Soames. It would injure him in his profession: He would have to get rid of the house at Robin Hill, on which he had spent so much money, so much anticipation — and at a sacrifice. And she! She would no longer belong to him, not even in name! She would pass out of his life, and he — he should never see her again!
He traversed in the cab the length of a street without getting beyond the thought that he should never see her again!
r her hair; and at this scent the burning sickness of his jealousy seized him again.
Struggling into his fur,the watch was a three-cornered note addressed ‘Soames Forsyte,’ in‘Ierceived under the softness and immobility of this figure something desperate and resolved; something not to be turned away, something dangerous. She tore off her hat, and, putting both hands to her brow, pressed back the bronze mass of her hair.

2.找一篇白名单文章white_list如下:

word
He
under
only

3.将文章上传至hdfs文件系统内

hadoop   fs   -put   The_Man_of_Property.txt   /mapreduce
hadoop fs -put white_list /mapreduce

4.map端代码如下:

#!usr/bin/python
import sys
def read_local_file(file):
word_set = set()
file_in = open (file,'r')
for line in file_in:
word = line.strip()
word_set.add(word)
return word_set
def mapper_func(file):
word_set=read_local_file(file) for line in sys.stdin:
ss=line.strip().split()
for word in ss:
word.strip()
if word != "" and (word in word_set):
print "%s\t%s"%(word,"") if __name__ == "__main__":
func = getattr(sys.modules[__name__],sys.argv[1])
args = None
if len(sys.argv) > 1:
args = sys.argv[2:]
func(*args)

5.reduce端代码如下

#!usr/bin/python
import sys
def reducer_func():
word="None"
sum=0
for line in sys.stdin:
ss=line.split()
cur_word=ss[0]
cnt=int(ss[1])
if cur_word!=word:
if word!="None":
print "%s\t%s"%(word,sum)
word=cur_word
sum=0
else:
sum+=cnt
print "%s\t%s"%(word,sum)
if __name__ == "__main__":
func = getattr(sys.modules[__name__],sys.argv[1])
args = None
if len(sys.argv) > 1:
args=sys.argv[2:]
func(*args)

5.运行shell脚本如下:

HADOOP="/usr/local/src/hadoop-1.2.1/bin/hadoop"
HADOOP_STREAMING="/usr/local/src/hadoop-1.2.1/contrib/streaming/hadoop-streaming-1.2.1.jar"
INPUT_PATH="/mapreduce/The_Man_of_Property.txt"
OUTPUT_PATH="/mapreduce/out"
$HADOOP fs -rmr $OUTPUT_PATH
$HADOOP jar $HADOOP_STREAMING \
-input "$INPUT_PATH" \
-output "$OUTPUT_PATH" \
-mapper "python map.py mapper_func ABC" \
-reducer "python red.py reducer_func" \
-file "./map.py"\
-file "./red.py"\
-cacheFile "hdfs://master:9000/mapreduce/white_list#ABC" #注意这块ABC为代名词,代表分发文件,后续将分发到各个节点的临时目录作为参数传入map函数

6.运行shell脚本

sh  run.sh

大数据python词频统计之hdfs分发-cacheFile的更多相关文章

  1. 大数据python词频统计之hdfs分发-cacheArchive

    -cacheArchive也是从hdfs上进分发,但是分发文件是一个压缩包,压缩包内可能会包含多层目录多个文件 1.The_Man_of_Property.txt文件如下(将其上传至hdfs上) ha ...

  2. 大数据python词频统计之本地分发-file

    统计某几个词在文章出现的次数 -file参数分发,是从客户端分发到各个执行mapreduce端的机器上 1.找一篇文章The_Man_of_Property.txt如下: He was proud o ...

  3. Python 词频统计

    利用Python做一个词频统计 GitHub地址:FightingBob [Give me a star , thanks.] 词频统计 对纯英语的文本文件[Eg: 瓦尔登湖(英文版).txt]的英文 ...

  4. 大数据Python学习大纲

    最近公司在写一个课程<大数据运维实训课>,分为4个部分,linux实训课.Python开发.hadoop基础知识和项目实战.这门课程主要针对刚从学校毕业的学生去应聘时不会像一个小白菜一样被 ...

  5. python词频统计及其效能分析

    1) 博客开头给出自己的基本信息,格式建议如下: 学号2017****7128 姓名:肖文秀 词频统计及其效能分析仓库:https://gitee.com/aichenxi/word_frequenc ...

  6. 大数据学习(一)-------- HDFS

    需要精通java开发,有一定linux基础. 1.简介 大数据就是对海量数据进行数据挖掘. 已经有了很多框架方便使用,常用的有hadoop,storm,spark,flink等,辅助框架hive,ka ...

  7. 大数据学习之旅1——HDFS版本演化

    最近开始学习大数据,发现大数据有很多很多组件,我现在负责的是HDFS(Hadoop分布式储存系统)的学习,整理了一下HDFS的版本情况.因为HDFS是Hadoop的重要组成部分,所以有关HDFS的版本 ...

  8. 大数据谢列3:Hdfs的HA实现

    在之前的文章:大数据系列:一文初识Hdfs , 大数据系列2:Hdfs的读写操作 中Hdfs的组成.读写有简单的介绍. 在里面介绍Secondary NameNode和Hdfs读写的流程. 并且在文章 ...

  9. 大数据学习(02)——HDFS入门

    Hadoop模块 提到大数据,Hadoop是一个绕不开的话题,我们来看看Hadoop本身包含哪些模块. Common是基础模块,这个是必须用的.剩下常用的就是HDFS和YARN. MapReduce现 ...

随机推荐

  1. mongodb 添加字段并设置默认值

    db.doc名称.update({}, {$set: {新字段名称: 默认值}}, false, true) 如:db.fly_bill.update({}, {$set: {usableStatus ...

  2. [C++]头文件<algorithm>

    本博文仅示例一些常用的函数: sort.for_each. 1. sort /* STL - <algorithm> - sort template< class RandomIt, ...

  3. Python之线程 2 - Python实现线程

    一 python与线程 1.全局解释器锁GIL(用一下threading模块之后再来看~~) 2.python线程模块的选择 二 Threading模块 1.线程创建 2.多线程与多进程 3.多线程实 ...

  4. Spark思维导图之Spark Core

  5. border-radius图解

    自己看了理解的border-radius: 设置参数: 100*100的正方形,第一个:border-top-left-radius:100px 100px,即圆的半径为100px.圆弧与上和左bor ...

  6. spark教程

    某大神总结的spark教程, 地址 http://litaotao.github.io/introduction-to-spark?s=inner

  7. 使用Retrofit时常用到的注解

  8. 同步&异步+阻塞&非阻塞(理解)

    0 - 同步&异步 同步和异步关注的是消息通信机制. 0.1 - 同步 由“调用者”主动等待这个“调用”结果.即是,发出一个“调用”时,在没有得到结果之前,该“调用”不返回,一旦调用返回,则得 ...

  9. AIDL通信过程中设置死亡代理

    关于AIDL的使用参考学习: https://blog.csdn.net/u011240877/article/details/72765136 https://blog.csdn.net/iromk ...

  10. VMware下CentOS7设置网络以及修改系统语言

    1.在VMware里,依次点击”编辑“ - ”虚拟网络编辑器“,如下图,我选择的是NAT模式 为了能够使用静态IP,这里不要勾选”使用本地DHCP服务将IP分配给虚拟机“这个选项.然后是配置子网ip, ...