win下idea远程提交WordCount任务到HA集群

一，环境配置

1，修改win下的host文件：即C:\Windows\System32\drivers\etc\host中添加集群中机子的ip

2，win下hadoop，并为win的环境变量配置hadoop_home，添加winutils.exe放到$HADOOP_HOME/bin下

3，使用idea新建maven项目，其中pom.xml设置如下：

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>big</groupId>

    <artifactId>data</artifactId>

    <version>1.0-SNAPSHOT</version>

    <dependencies>

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-common</artifactId>

            <version>2.7.5</version>

        </dependency>

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-client</artifactId>

            <version>2.7.5</version>

        </dependency>

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-hdfs</artifactId>

            <version>2.7.5</version>

        </dependency>

       <!-- <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-hdfs-client</artifactId>

            <version>2.7.5</version>

        </dependency>-->

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>

            <version>2.7.5</version>

        </dependency>

    </dependencies>

</project>

4，拷贝ha集群中hadoop的配置文件到idea中resource中，hadoop的具体配置如下：

core-site.xml:

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

       <property>

                <name>fs.defaultFS</name>

                <value>hdfs://mycluster</value>

       </property>

       <property>

                <name>ha.zookeeper.quorum</name>

                <value>cent1:2181,cent2:2181,cent3:2181</value>

        </property>

       <!--<property>

               <name>hadoop.tmp.dir</name>

               <value>/opt/hadoop2</value>

               <description>A base for other temporary   directories.</description>

       </property>-->

</configuration>

hdfs-site.xml:

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>

             <name>dfs.nameservices</name>

             <value>mycluster</value>

    </property>

     <property>

             <name>dfs.ha.namenodes.mycluster</name>

             <value>nn1,nn2</value>

    </property>

     <property>

             <name>dfs.namenode.rpc-address.mycluster.nn1</name>

             <value>cent1:9000</value>

    </property>

     <property>

             <name>dfs.namenode.rpc-address.mycluster.nn2</name>

             <value>cent2:9000</value>

    </property>

     <property>

             <name>dfs.namenode.http-address.mycluster.nn1</name>

             <value>cent1:50070</value>

    </property>

     <property>

             <name>dfs.namenode.http-address.mycluster.nn2</name>

             <value>cent2:50070</value>

    </property>

     <property>

             <name>dfs.namenode.shared.edits.dir</name>

             <value>qjournal://cent2:8485;cent3:8485;cent4:8485/mycluster</value>

    </property>

     <property>

             <name>dfs.client.failover.proxy.provider.mycluster</name>

             <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

    </property>

     <property>

             <name>dfs.ha.fencing.methods</name>

             <value>sshfence</value>

    </property>

     <property>

             <name>dfs.ha.fencing.ssh.private-key-files</name>

             <value>/root/.ssh/id_rsa</value>

    </property>

     <property>

             <name>dfs.journalnode.edits.dir</name>

             <value>/opt/jn/data</value>

    </property>

     <property>

             <name>dfs.ha.automatic-failover.enabled</name>

             <value>true</value>

    </property>

    <property>

        <name>dfs.permissions.enabled</name>

        <value>false</value>

    </property>

</configuration>

mapred-site.xml:

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

      <name>mapreduce.framework.name</name>

          <value>yarn</value>

</property>

</configuration>

yarn-site.xml:

<?xml version="1.0"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<configuration>

<!-- Site specific YARN configuration properties -->

  <property>

               <name>yarn.nodemanager.aux-services</name>

               <value>mapreduce_shuffle</value>

        </property>

        <property>

                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

                <value>org.apache.hadoop.mapred.ShuffleHandler</value>

        </property>

        <property>

               <name>yarn.resourcemanager.hostname</name>

               <value>cent1</value>

       </property>

</configuration>

log4j.properties:

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements.  See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Define some default values that can be overridden by system properties

hadoop.root.logger=INFO,console

hadoop.log.dir=.

hadoop.log.file=hadoop.log

# Define the root logger to the system property "hadoop.root.logger".

log4j.rootLogger=${hadoop.root.logger}, EventCounter

# Logging Threshold

log4j.threshold=ALL

# Null Appender

log4j.appender.NullAppender=org.apache.log4j.varia.NullAppender

#

# Rolling File Appender - cap space usage at 5gb.

#

hadoop.log.maxfilesize=256MB

hadoop.log.maxbackupindex=20

log4j.appender.RFA=org.apache.log4j.RollingFileAppender

log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}

log4j.appender.RFA.MaxFileSize=${hadoop.log.maxfilesize}

log4j.appender.RFA.MaxBackupIndex=${hadoop.log.maxbackupindex}

log4j.appender.RFA.layout=org.apache.log4j.PatternLayout

# Pattern format: Date LogLevel LoggerName LogMessage

log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

# Debugging Pattern format

#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

#

# Daily Rolling File Appender

#

log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender

log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}

# Rollover at midnight

log4j.appender.DRFA.DatePattern=.yyyy-MM-dd

log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout

# Pattern format: Date LogLevel LoggerName LogMessage

log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

# Debugging Pattern format

#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

#

# console

# Add "console" to rootlogger above if you want to use this

#

log4j.appender.console=org.apache.log4j.ConsoleAppender

log4j.appender.console.target=System.err

log4j.appender.console.layout=org.apache.log4j.PatternLayout

log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

#

# TaskLog Appender

#

#Default values

hadoop.tasklog.taskid=null

hadoop.tasklog.iscleanup=false

hadoop.tasklog.noKeepSplits=4

hadoop.tasklog.totalLogFileSize=100

hadoop.tasklog.purgeLogSplits=true

hadoop.tasklog.logsRetainHours=12

log4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender

log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}

log4j.appender.TLA.isCleanup=${hadoop.tasklog.iscleanup}

log4j.appender.TLA.totalLogFileSize=${hadoop.tasklog.totalLogFileSize}

log4j.appender.TLA.layout=org.apache.log4j.PatternLayout

log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

#

# HDFS block state change log from block manager

#

# Uncomment the following to suppress normal block state change

# messages from BlockManager in NameNode.

#log4j.logger.BlockStateChange=WARN

#

#Security appender

#

hadoop.security.logger=INFO,NullAppender

hadoop.security.log.maxfilesize=256MB

hadoop.security.log.maxbackupindex=20

log4j.category.SecurityLogger=${hadoop.security.logger}

hadoop.security.log.file=SecurityAuth-${user.name}.audit

log4j.appender.RFAS=org.apache.log4j.RollingFileAppender

log4j.appender.RFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}

log4j.appender.RFAS.layout=org.apache.log4j.PatternLayout

log4j.appender.RFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

log4j.appender.RFAS.MaxFileSize=${hadoop.security.log.maxfilesize}

log4j.appender.RFAS.MaxBackupIndex=${hadoop.security.log.maxbackupindex}

#

# Daily Rolling Security appender

#

log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender

log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}

log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout

log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd

#

# hadoop configuration logging

#

# Uncomment the following line to turn off configuration deprecation warnings.

# log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN

#

# hdfs audit logging

#

hdfs.audit.logger=INFO,NullAppender

hdfs.audit.log.maxfilesize=256MB

hdfs.audit.log.maxbackupindex=20

log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}

log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false

log4j.appender.RFAAUDIT=org.apache.log4j.RollingFileAppender

log4j.appender.RFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log

log4j.appender.RFAAUDIT.layout=org.apache.log4j.PatternLayout

log4j.appender.RFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

log4j.appender.RFAAUDIT.MaxFileSize=${hdfs.audit.log.maxfilesize}

log4j.appender.RFAAUDIT.MaxBackupIndex=${hdfs.audit.log.maxbackupindex}

#

# mapred audit logging

#

mapred.audit.logger=INFO,NullAppender

mapred.audit.log.maxfilesize=256MB

mapred.audit.log.maxbackupindex=20

log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger}

log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false

log4j.appender.MRAUDIT=org.apache.log4j.RollingFileAppender

log4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log

log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout

log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

log4j.appender.MRAUDIT.MaxFileSize=${mapred.audit.log.maxfilesize}

log4j.appender.MRAUDIT.MaxBackupIndex=${mapred.audit.log.maxbackupindex}

# Custom Logging levels

#log4j.logger.org.apache.hadoop.mapred.JobTracker=DEBUG

#log4j.logger.org.apache.hadoop.mapred.TaskTracker=DEBUG

#log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=DEBUG

# Jets3t library

log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=ERROR

# AWS SDK & S3A FileSystem

log4j.logger.com.amazonaws=ERROR

log4j.logger.com.amazonaws.http.AmazonHttpClient=ERROR

log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=WARN

#

# Event Counter Appender

# Sends counts of logging messages at different severity levels to Hadoop Metrics.

#

log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter

#

# Job Summary Appender

#

# Use following logger to send summary to separate file defined by

# hadoop.mapreduce.jobsummary.log.file :

# hadoop.mapreduce.jobsummary.logger=INFO,JSA

#

hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}

hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log

hadoop.mapreduce.jobsummary.log.maxfilesize=256MB

hadoop.mapreduce.jobsummary.log.maxbackupindex=20

log4j.appender.JSA=org.apache.log4j.RollingFileAppender

log4j.appender.JSA.File=${hadoop.log.dir}/${hadoop.mapreduce.jobsummary.log.file}

log4j.appender.JSA.MaxFileSize=${hadoop.mapreduce.jobsummary.log.maxfilesize}

log4j.appender.JSA.MaxBackupIndex=${hadoop.mapreduce.jobsummary.log.maxbackupindex}

log4j.appender.JSA.layout=org.apache.log4j.PatternLayout

log4j.appender.JSA.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

log4j.logger.org.apache.hadoop.mapred.JobInProgress$JobSummary=${hadoop.mapreduce.jobsummary.logger}

log4j.additivity.org.apache.hadoop.mapred.JobInProgress$JobSummary=false

#

# Yarn ResourceManager Application Summary Log

#

# Set the ResourceManager summary log filename

yarn.server.resourcemanager.appsummary.log.file=rm-appsummary.log

# Set the ResourceManager summary log level and appender

yarn.server.resourcemanager.appsummary.logger=${hadoop.root.logger}

#yarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY

# To enable AppSummaryLogging for the RM,

# set yarn.server.resourcemanager.appsummary.logger to

# <LEVEL>,RMSUMMARY in hadoop-env.sh

# Appender for ResourceManager Application Summary Log

# Requires the following properties to be set

#    - hadoop.log.dir (Hadoop Log directory)

#    - yarn.server.resourcemanager.appsummary.log.file (resource manager app summary log filename)

#    - yarn.server.resourcemanager.appsummary.logger (resource manager app summary log level and appender)

log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${yarn.server.resourcemanager.appsummary.logger}

log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false

log4j.appender.RMSUMMARY=org.apache.log4j.RollingFileAppender

log4j.appender.RMSUMMARY.File=${hadoop.log.dir}/${yarn.server.resourcemanager.appsummary.log.file}

log4j.appender.RMSUMMARY.MaxFileSize=256MB

log4j.appender.RMSUMMARY.MaxBackupIndex=20

log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout

log4j.appender.RMSUMMARY.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

# HS audit log configs

#mapreduce.hs.audit.logger=INFO,HSAUDIT

#log4j.logger.org.apache.hadoop.mapreduce.v2.hs.HSAuditLogger=${mapreduce.hs.audit.logger}

#log4j.additivity.org.apache.hadoop.mapreduce.v2.hs.HSAuditLogger=false

#log4j.appender.HSAUDIT=org.apache.log4j.DailyRollingFileAppender

#log4j.appender.HSAUDIT.File=${hadoop.log.dir}/hs-audit.log

#log4j.appender.HSAUDIT.layout=org.apache.log4j.PatternLayout

#log4j.appender.HSAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

#log4j.appender.HSAUDIT.DatePattern=.yyyy-MM-dd

# Http Server Request Logs

#log4j.logger.http.requests.namenode=INFO,namenoderequestlog

#log4j.appender.namenoderequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.namenoderequestlog.Filename=${hadoop.log.dir}/jetty-namenode-yyyy_mm_dd.log

#log4j.appender.namenoderequestlog.RetainDays=3

#log4j.logger.http.requests.datanode=INFO,datanoderequestlog

#log4j.appender.datanoderequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.datanoderequestlog.Filename=${hadoop.log.dir}/jetty-datanode-yyyy_mm_dd.log

#log4j.appender.datanoderequestlog.RetainDays=3

#log4j.logger.http.requests.resourcemanager=INFO,resourcemanagerrequestlog

#log4j.appender.resourcemanagerrequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.resourcemanagerrequestlog.Filename=${hadoop.log.dir}/jetty-resourcemanager-yyyy_mm_dd.log

#log4j.appender.resourcemanagerrequestlog.RetainDays=3

#log4j.logger.http.requests.jobhistory=INFO,jobhistoryrequestlog

#log4j.appender.jobhistoryrequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.jobhistoryrequestlog.Filename=${hadoop.log.dir}/jetty-jobhistory-yyyy_mm_dd.log

#log4j.appender.jobhistoryrequestlog.RetainDays=3

#log4j.logger.http.requests.nodemanager=INFO,nodemanagerrequestlog

#log4j.appender.nodemanagerrequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.nodemanagerrequestlog.Filename=${hadoop.log.dir}/jetty-nodemanager-yyyy_mm_dd.log

#log4j.appender.nodemanagerrequestlog.RetainDays=3

二，编写WordCount程序

import java.io.IOException;

import java.net.URI;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

    public static class TokenizerMapper

            extends Mapper<Object, Text, Text, IntWritable> {

        private final static IntWritable one = new IntWritable(1);

        private Text word = new Text();

        public void map(Object key, Text value, Context context

        ) throws IOException, InterruptedException {

            StringTokenizer itr = new StringTokenizer(value.toString());

            while (itr.hasMoreTokens()) {

                word.set(itr.nextToken());

                context.write(word, one);

            }

        }

    }

    public static class IntSumReducer

            extends Reducer<Text, IntWritable, Text, IntWritable> {

        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable<IntWritable> values,

                           Context context

        ) throws IOException, InterruptedException {

            int sum = 0;

            for (IntWritable val : values) {

                sum += val.get();

            }

            result.set(sum);

            context.write(key, result);

        }

    }

    public static void main(String[] args) throws Exception {

        Configuration conf = new Configuration();

       System.setProperty("hadoop.home.dir", "E:\\softs\\majorSoft\\hadoop-2.7.5");//初始时解决winutils异常

        conf.set("mapreduce.app-submission.cross-platform", "true");//允许远程访问

        Path input = new Path(URI.create("hdfs://mycluster/testFile/wordCount"));

        Path output = new Path(URI.create("hdfs://mycluster/output"));

        Job job = Job.getInstance(conf, "word count");

        job.setJar("E:\\bigData\\hadoopDemo\\out\\artifacts\\wordCount_jar\\hadoopDemo.jar");//必须要先打包出jar包

        job.setJarByClass(WordCount.class);

        job.setMapperClass(TokenizerMapper.class);

        job.setCombinerClass(IntSumReducer.class);

        job.setReducerClass(IntSumReducer.class);

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, input);

        FileOutputFormat.setOutputPath(job, output);

        System.exit(job.waitForCompletion(true) ? 0 : 1);

    }

}

三，遇到的异常

1，RuntimeException, ClassNotFoundException: Class WordCount$Map not found . Mapper class issue

job.setJar("WordCount.jar");

2，Exception message:/bin/bash:第0行fg：无任务控制  #表示运行远程访问格式

conf.set(“mapreduce.app-submission.cross-platform”, “true”);

和设置hdfs-site.xml

<property>

        <name>dfs.permissions.enabled</name>

        <value>false</value>

    </property>

3. java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

System.setProperty("hadoop.home.dir", "E:\\softs\\majorSoft\\hadoop-2.7.5");

4，无法访问hdfs权限和识别不到集群

修改C:\Windows\System32\drivers\etc文件

win下idea远程提交WordCount任务到HA集群的更多相关文章

Idea里面远程提交spark任务到yarn集群
Idea里面远程提交spark任务到yarn集群 1.本地idea远程提交到yarn集群 2.运行过程中可能会遇到的问题 2.1首先需要把yarn-site.xml,core-site.xml,hdf ...
VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群
VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群下一篇:VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群-整合Zookeeper和Hbase 近期 ...
在linux环境下安装redis并且搭建自己的redis集群
此文档主要介绍在linux环境下安装redis并且搭建自己的redis集群搭建环境: ubuntun 16.04 + redis-3.0.6 本文章分为三个部分:redis安装.搭建redis集群 ...
VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群—整合Zookeeper和Hbase
VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群-整合Zookeeper和Hbase 这篇是接着上一篇hadoop集群搭建进行的.在hadoop-1.2.1基础之上安装zoo ...
win下写任务提交给集群
一,复制和删除hdfs中的文件 import org.apache.hadoop.fs.{FileSystem, Path} import org.apache.spark.{SparkConf, S ...
联想ThinkPad S3-S440虚拟机安装，ubuntu安装，Hadoop(2.7.1)详解及WordCount运行，spark集群搭建
下载ubuntu操作系统版本 ubuntu-14.10-desktop-amd64.iso(64位) 安装过程出现错误: This kernel requires an X86-64 CPU,but ...
Windows下ELK环境搭建（单机多节点集群部署）
1.背景日志主要包括系统日志.应用程序日志和安全日志.系统运维和开发人员可以通过日志了解服务器软硬件信息.检查配置过程中的错误及错误发生的原因.经常分析日志可以了解服务器的负荷,性能安全性,从而及时 ...
linux系统下对网站实施负载均衡+高可用集群需要考虑的几点
随着linux系统的成熟和广泛普及,linux运维技术越来越受到企业的关注和追捧.在一些中小企业,尤其是牵涉到电子商务和电子广告类的网站,通常会要求作负载均衡和高可用的Linux集群方案. 那么如何实 ...
Docker环境下搭建DNS LVS(keepAlived) OpenResty服务器简易集群
现在上网已经成为每个人必备的技能,打开浏览器,输入网址,回车,简单的几步就能浏览到漂亮的网页,那从请求发出到返回漂亮的页面是怎么做到的呢,我将从公司中一般的分层架构角度考虑搭建一个简易集群来实现.目标 ...

随机推荐

WordPress Shortcode（简码）介绍及使用详解
WordPress 从 2.5 版本开始增加了一个类似 BBCode 标签的 Shortcode API,可以使用它在日志的内容中来给日志内容添加各种功能.Shortcode 这个接口非常容易使用,并 ...
Linux下GCC相关知识点
摘要: 总结GCC的具体使用,动态库静态库的相关问题参考资料: <Linux网络编程> ISBN:9787302207177 p19 1 GCC简介 GCC是Linux下的编译工具集,是 ...
python怎么解压压缩的字符串数据
范例1: gzip import StringIO import gzip compresseddata = gzip方式压缩的字符串(html) compressedstream = StringI ...
Eclipse中快速打出 main方法的签名
有时,我们创建一个空白类,需要打出main方法 public static void main(String [] args){ } 在Eclipse先敲main字符,然后按住ALT+/,再按回车即可 ...
(3) go 指针
1.获取地址取地址 &i 2.指针 var p *int=&i ptr 是一个地址 *ptr 是可看成一个变量,该地址所在的变量,也就是 num 3.常见值类型引用类型值类型:栈 ...
windows同时使用python2和3
前言手头有些脚本,有些是在python2的环境下使用,有些是在python3的环境下使用以前我是把python3的脚本都放在虚拟机中,现在发现是真的麻烦,于是来研究一下如何使得python2和3共 ...
【SQL】将特定的元素按照自己所需的位置排序
Oracle中,平时我们排序常用“Order by 列名” 的方式来排序,但是有的时候我们希望这个列中的某些元素排在前面或者后面或者中间的某个位置. 这时我们可以使用Order by case whe ...
Arduino可穿戴开发入门教程Arduino开发环境介绍
Arduino可穿戴开发入门教程Arduino开发环境介绍 Arduino开发环境介绍 Arduino不像我们使用的PC端操作系统一样,可以直接在操作系统中安装软件为操作系统编程.Arduino的软件 ...
[CODECHEF]SEGPROD
题意:给定$a_{1\cdots n}$和$p$,多次询问$\prod\limits_{i=l}^ra_i$对$p$取模的值,强制在线,每次询问要求$O(1)$回答一个微小的黑科技... 静态区间查 ...
【主席树】【最近公共祖先】hdu6162 Ch’s gift
题意:一棵树,每个点有个权值,m次询问,每次给你一条链和两个值a,b,问你这条链上权值在[a,b]之间的权值的和是多少. std竟然是2个log的……完全没必要链剖,每个结点的主席树从其父节点转移过 ...

win下idea远程提交WordCount任务到HA集群

win下idea远程提交WordCount任务到HA集群的更多相关文章

随机推荐

热门专题