win下idea远程提交WordCount任务到HA集群

一，环境配置

1，修改win下的host文件：即C:\Windows\System32\drivers\etc\host中添加集群中机子的ip

2，win下hadoop，并为win的环境变量配置hadoop_home，添加winutils.exe放到$HADOOP_HOME/bin下

3，使用idea新建maven项目，其中pom.xml设置如下：

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>big</groupId>

    <artifactId>data</artifactId>

    <version>1.0-SNAPSHOT</version>

    <dependencies>

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-common</artifactId>

            <version>2.7.5</version>

        </dependency>

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-client</artifactId>

            <version>2.7.5</version>

        </dependency>

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-hdfs</artifactId>

            <version>2.7.5</version>

        </dependency>

       <!-- <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-hdfs-client</artifactId>

            <version>2.7.5</version>

        </dependency>-->

        <dependency>

            <groupId>org.apache.hadoop</groupId>

            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>

            <version>2.7.5</version>

        </dependency>

    </dependencies>

</project>

4，拷贝ha集群中hadoop的配置文件到idea中resource中，hadoop的具体配置如下：

core-site.xml:

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

       <property>

                <name>fs.defaultFS</name>

                <value>hdfs://mycluster</value>

       </property>

       <property>

                <name>ha.zookeeper.quorum</name>

                <value>cent1:2181,cent2:2181,cent3:2181</value>

        </property>

       <!--<property>

               <name>hadoop.tmp.dir</name>

               <value>/opt/hadoop2</value>

               <description>A base for other temporary   directories.</description>

       </property>-->

</configuration>

hdfs-site.xml:

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>

             <name>dfs.nameservices</name>

             <value>mycluster</value>

    </property>

     <property>

             <name>dfs.ha.namenodes.mycluster</name>

             <value>nn1,nn2</value>

    </property>

     <property>

             <name>dfs.namenode.rpc-address.mycluster.nn1</name>

             <value>cent1:9000</value>

    </property>

     <property>

             <name>dfs.namenode.rpc-address.mycluster.nn2</name>

             <value>cent2:9000</value>

    </property>

     <property>

             <name>dfs.namenode.http-address.mycluster.nn1</name>

             <value>cent1:50070</value>

    </property>

     <property>

             <name>dfs.namenode.http-address.mycluster.nn2</name>

             <value>cent2:50070</value>

    </property>

     <property>

             <name>dfs.namenode.shared.edits.dir</name>

             <value>qjournal://cent2:8485;cent3:8485;cent4:8485/mycluster</value>

    </property>

     <property>

             <name>dfs.client.failover.proxy.provider.mycluster</name>

             <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

    </property>

     <property>

             <name>dfs.ha.fencing.methods</name>

             <value>sshfence</value>

    </property>

     <property>

             <name>dfs.ha.fencing.ssh.private-key-files</name>

             <value>/root/.ssh/id_rsa</value>

    </property>

     <property>

             <name>dfs.journalnode.edits.dir</name>

             <value>/opt/jn/data</value>

    </property>

     <property>

             <name>dfs.ha.automatic-failover.enabled</name>

             <value>true</value>

    </property>

    <property>

        <name>dfs.permissions.enabled</name>

        <value>false</value>

    </property>

</configuration>

mapred-site.xml:

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

      <name>mapreduce.framework.name</name>

          <value>yarn</value>

</property>

</configuration>

yarn-site.xml:

<?xml version="1.0"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<configuration>

<!-- Site specific YARN configuration properties -->

  <property>

               <name>yarn.nodemanager.aux-services</name>

               <value>mapreduce_shuffle</value>

        </property>

        <property>

                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

                <value>org.apache.hadoop.mapred.ShuffleHandler</value>

        </property>

        <property>

               <name>yarn.resourcemanager.hostname</name>

               <value>cent1</value>

       </property>

</configuration>

log4j.properties:

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements.  See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Define some default values that can be overridden by system properties

hadoop.root.logger=INFO,console

hadoop.log.dir=.

hadoop.log.file=hadoop.log

# Define the root logger to the system property "hadoop.root.logger".

log4j.rootLogger=${hadoop.root.logger}, EventCounter

# Logging Threshold

log4j.threshold=ALL

# Null Appender

log4j.appender.NullAppender=org.apache.log4j.varia.NullAppender

#

# Rolling File Appender - cap space usage at 5gb.

#

hadoop.log.maxfilesize=256MB

hadoop.log.maxbackupindex=20

log4j.appender.RFA=org.apache.log4j.RollingFileAppender

log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}

log4j.appender.RFA.MaxFileSize=${hadoop.log.maxfilesize}

log4j.appender.RFA.MaxBackupIndex=${hadoop.log.maxbackupindex}

log4j.appender.RFA.layout=org.apache.log4j.PatternLayout

# Pattern format: Date LogLevel LoggerName LogMessage

log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

# Debugging Pattern format

#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

#

# Daily Rolling File Appender

#

log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender

log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}

# Rollover at midnight

log4j.appender.DRFA.DatePattern=.yyyy-MM-dd

log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout

# Pattern format: Date LogLevel LoggerName LogMessage

log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

# Debugging Pattern format

#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

#

# console

# Add "console" to rootlogger above if you want to use this

#

log4j.appender.console=org.apache.log4j.ConsoleAppender

log4j.appender.console.target=System.err

log4j.appender.console.layout=org.apache.log4j.PatternLayout

log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

#

# TaskLog Appender

#

#Default values

hadoop.tasklog.taskid=null

hadoop.tasklog.iscleanup=false

hadoop.tasklog.noKeepSplits=4

hadoop.tasklog.totalLogFileSize=100

hadoop.tasklog.purgeLogSplits=true

hadoop.tasklog.logsRetainHours=12

log4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender

log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}

log4j.appender.TLA.isCleanup=${hadoop.tasklog.iscleanup}

log4j.appender.TLA.totalLogFileSize=${hadoop.tasklog.totalLogFileSize}

log4j.appender.TLA.layout=org.apache.log4j.PatternLayout

log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

#

# HDFS block state change log from block manager

#

# Uncomment the following to suppress normal block state change

# messages from BlockManager in NameNode.

#log4j.logger.BlockStateChange=WARN

#

#Security appender

#

hadoop.security.logger=INFO,NullAppender

hadoop.security.log.maxfilesize=256MB

hadoop.security.log.maxbackupindex=20

log4j.category.SecurityLogger=${hadoop.security.logger}

hadoop.security.log.file=SecurityAuth-${user.name}.audit

log4j.appender.RFAS=org.apache.log4j.RollingFileAppender

log4j.appender.RFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}

log4j.appender.RFAS.layout=org.apache.log4j.PatternLayout

log4j.appender.RFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

log4j.appender.RFAS.MaxFileSize=${hadoop.security.log.maxfilesize}

log4j.appender.RFAS.MaxBackupIndex=${hadoop.security.log.maxbackupindex}

#

# Daily Rolling Security appender

#

log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender

log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}

log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout

log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd

#

# hadoop configuration logging

#

# Uncomment the following line to turn off configuration deprecation warnings.

# log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN

#

# hdfs audit logging

#

hdfs.audit.logger=INFO,NullAppender

hdfs.audit.log.maxfilesize=256MB

hdfs.audit.log.maxbackupindex=20

log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}

log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false

log4j.appender.RFAAUDIT=org.apache.log4j.RollingFileAppender

log4j.appender.RFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log

log4j.appender.RFAAUDIT.layout=org.apache.log4j.PatternLayout

log4j.appender.RFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

log4j.appender.RFAAUDIT.MaxFileSize=${hdfs.audit.log.maxfilesize}

log4j.appender.RFAAUDIT.MaxBackupIndex=${hdfs.audit.log.maxbackupindex}

#

# mapred audit logging

#

mapred.audit.logger=INFO,NullAppender

mapred.audit.log.maxfilesize=256MB

mapred.audit.log.maxbackupindex=20

log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger}

log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false

log4j.appender.MRAUDIT=org.apache.log4j.RollingFileAppender

log4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log

log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout

log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

log4j.appender.MRAUDIT.MaxFileSize=${mapred.audit.log.maxfilesize}

log4j.appender.MRAUDIT.MaxBackupIndex=${mapred.audit.log.maxbackupindex}

# Custom Logging levels

#log4j.logger.org.apache.hadoop.mapred.JobTracker=DEBUG

#log4j.logger.org.apache.hadoop.mapred.TaskTracker=DEBUG

#log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=DEBUG

# Jets3t library

log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=ERROR

# AWS SDK & S3A FileSystem

log4j.logger.com.amazonaws=ERROR

log4j.logger.com.amazonaws.http.AmazonHttpClient=ERROR

log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=WARN

#

# Event Counter Appender

# Sends counts of logging messages at different severity levels to Hadoop Metrics.

#

log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter

#

# Job Summary Appender

#

# Use following logger to send summary to separate file defined by

# hadoop.mapreduce.jobsummary.log.file :

# hadoop.mapreduce.jobsummary.logger=INFO,JSA

#

hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}

hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log

hadoop.mapreduce.jobsummary.log.maxfilesize=256MB

hadoop.mapreduce.jobsummary.log.maxbackupindex=20

log4j.appender.JSA=org.apache.log4j.RollingFileAppender

log4j.appender.JSA.File=${hadoop.log.dir}/${hadoop.mapreduce.jobsummary.log.file}

log4j.appender.JSA.MaxFileSize=${hadoop.mapreduce.jobsummary.log.maxfilesize}

log4j.appender.JSA.MaxBackupIndex=${hadoop.mapreduce.jobsummary.log.maxbackupindex}

log4j.appender.JSA.layout=org.apache.log4j.PatternLayout

log4j.appender.JSA.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

log4j.logger.org.apache.hadoop.mapred.JobInProgress$JobSummary=${hadoop.mapreduce.jobsummary.logger}

log4j.additivity.org.apache.hadoop.mapred.JobInProgress$JobSummary=false

#

# Yarn ResourceManager Application Summary Log

#

# Set the ResourceManager summary log filename

yarn.server.resourcemanager.appsummary.log.file=rm-appsummary.log

# Set the ResourceManager summary log level and appender

yarn.server.resourcemanager.appsummary.logger=${hadoop.root.logger}

#yarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY

# To enable AppSummaryLogging for the RM,

# set yarn.server.resourcemanager.appsummary.logger to

# <LEVEL>,RMSUMMARY in hadoop-env.sh

# Appender for ResourceManager Application Summary Log

# Requires the following properties to be set

#    - hadoop.log.dir (Hadoop Log directory)

#    - yarn.server.resourcemanager.appsummary.log.file (resource manager app summary log filename)

#    - yarn.server.resourcemanager.appsummary.logger (resource manager app summary log level and appender)

log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${yarn.server.resourcemanager.appsummary.logger}

log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false

log4j.appender.RMSUMMARY=org.apache.log4j.RollingFileAppender

log4j.appender.RMSUMMARY.File=${hadoop.log.dir}/${yarn.server.resourcemanager.appsummary.log.file}

log4j.appender.RMSUMMARY.MaxFileSize=256MB

log4j.appender.RMSUMMARY.MaxBackupIndex=20

log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout

log4j.appender.RMSUMMARY.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

# HS audit log configs

#mapreduce.hs.audit.logger=INFO,HSAUDIT

#log4j.logger.org.apache.hadoop.mapreduce.v2.hs.HSAuditLogger=${mapreduce.hs.audit.logger}

#log4j.additivity.org.apache.hadoop.mapreduce.v2.hs.HSAuditLogger=false

#log4j.appender.HSAUDIT=org.apache.log4j.DailyRollingFileAppender

#log4j.appender.HSAUDIT.File=${hadoop.log.dir}/hs-audit.log

#log4j.appender.HSAUDIT.layout=org.apache.log4j.PatternLayout

#log4j.appender.HSAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

#log4j.appender.HSAUDIT.DatePattern=.yyyy-MM-dd

# Http Server Request Logs

#log4j.logger.http.requests.namenode=INFO,namenoderequestlog

#log4j.appender.namenoderequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.namenoderequestlog.Filename=${hadoop.log.dir}/jetty-namenode-yyyy_mm_dd.log

#log4j.appender.namenoderequestlog.RetainDays=3

#log4j.logger.http.requests.datanode=INFO,datanoderequestlog

#log4j.appender.datanoderequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.datanoderequestlog.Filename=${hadoop.log.dir}/jetty-datanode-yyyy_mm_dd.log

#log4j.appender.datanoderequestlog.RetainDays=3

#log4j.logger.http.requests.resourcemanager=INFO,resourcemanagerrequestlog

#log4j.appender.resourcemanagerrequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.resourcemanagerrequestlog.Filename=${hadoop.log.dir}/jetty-resourcemanager-yyyy_mm_dd.log

#log4j.appender.resourcemanagerrequestlog.RetainDays=3

#log4j.logger.http.requests.jobhistory=INFO,jobhistoryrequestlog

#log4j.appender.jobhistoryrequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.jobhistoryrequestlog.Filename=${hadoop.log.dir}/jetty-jobhistory-yyyy_mm_dd.log

#log4j.appender.jobhistoryrequestlog.RetainDays=3

#log4j.logger.http.requests.nodemanager=INFO,nodemanagerrequestlog

#log4j.appender.nodemanagerrequestlog=org.apache.hadoop.http.HttpRequestLogAppender

#log4j.appender.nodemanagerrequestlog.Filename=${hadoop.log.dir}/jetty-nodemanager-yyyy_mm_dd.log

#log4j.appender.nodemanagerrequestlog.RetainDays=3

二，编写WordCount程序

import java.io.IOException;

import java.net.URI;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

    public static class TokenizerMapper

            extends Mapper<Object, Text, Text, IntWritable> {

        private final static IntWritable one = new IntWritable(1);

        private Text word = new Text();

        public void map(Object key, Text value, Context context

        ) throws IOException, InterruptedException {

            StringTokenizer itr = new StringTokenizer(value.toString());

            while (itr.hasMoreTokens()) {

                word.set(itr.nextToken());

                context.write(word, one);

            }

        }

    }

    public static class IntSumReducer

            extends Reducer<Text, IntWritable, Text, IntWritable> {

        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable<IntWritable> values,

                           Context context

        ) throws IOException, InterruptedException {

            int sum = 0;

            for (IntWritable val : values) {

                sum += val.get();

            }

            result.set(sum);

            context.write(key, result);

        }

    }

    public static void main(String[] args) throws Exception {

        Configuration conf = new Configuration();

       System.setProperty("hadoop.home.dir", "E:\\softs\\majorSoft\\hadoop-2.7.5");//初始时解决winutils异常

        conf.set("mapreduce.app-submission.cross-platform", "true");//允许远程访问

        Path input = new Path(URI.create("hdfs://mycluster/testFile/wordCount"));

        Path output = new Path(URI.create("hdfs://mycluster/output"));

        Job job = Job.getInstance(conf, "word count");

        job.setJar("E:\\bigData\\hadoopDemo\\out\\artifacts\\wordCount_jar\\hadoopDemo.jar");//必须要先打包出jar包

        job.setJarByClass(WordCount.class);

        job.setMapperClass(TokenizerMapper.class);

        job.setCombinerClass(IntSumReducer.class);

        job.setReducerClass(IntSumReducer.class);

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, input);

        FileOutputFormat.setOutputPath(job, output);

        System.exit(job.waitForCompletion(true) ? 0 : 1);

    }

}

三，遇到的异常

1，RuntimeException, ClassNotFoundException: Class WordCount$Map not found . Mapper class issue

job.setJar("WordCount.jar");

2，Exception message:/bin/bash:第0行fg：无任务控制  #表示运行远程访问格式

conf.set(“mapreduce.app-submission.cross-platform”, “true”);

和设置hdfs-site.xml

<property>

        <name>dfs.permissions.enabled</name>

        <value>false</value>

    </property>

3. java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

System.setProperty("hadoop.home.dir", "E:\\softs\\majorSoft\\hadoop-2.7.5");

4，无法访问hdfs权限和识别不到集群

修改C:\Windows\System32\drivers\etc文件

win下idea远程提交WordCount任务到HA集群的更多相关文章

Idea里面远程提交spark任务到yarn集群
Idea里面远程提交spark任务到yarn集群 1.本地idea远程提交到yarn集群 2.运行过程中可能会遇到的问题 2.1首先需要把yarn-site.xml,core-site.xml,hdf ...
VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群
VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群下一篇:VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群-整合Zookeeper和Hbase 近期 ...
在linux环境下安装redis并且搭建自己的redis集群
此文档主要介绍在linux环境下安装redis并且搭建自己的redis集群搭建环境: ubuntun 16.04 + redis-3.0.6 本文章分为三个部分:redis安装.搭建redis集群 ...
VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群—整合Zookeeper和Hbase
VMWare9下基于Ubuntu12.10搭建Hadoop-1.2.1集群-整合Zookeeper和Hbase 这篇是接着上一篇hadoop集群搭建进行的.在hadoop-1.2.1基础之上安装zoo ...
win下写任务提交给集群
一,复制和删除hdfs中的文件 import org.apache.hadoop.fs.{FileSystem, Path} import org.apache.spark.{SparkConf, S ...
联想ThinkPad S3-S440虚拟机安装，ubuntu安装，Hadoop(2.7.1)详解及WordCount运行，spark集群搭建
下载ubuntu操作系统版本 ubuntu-14.10-desktop-amd64.iso(64位) 安装过程出现错误: This kernel requires an X86-64 CPU,but ...
Windows下ELK环境搭建（单机多节点集群部署）
1.背景日志主要包括系统日志.应用程序日志和安全日志.系统运维和开发人员可以通过日志了解服务器软硬件信息.检查配置过程中的错误及错误发生的原因.经常分析日志可以了解服务器的负荷,性能安全性,从而及时 ...
linux系统下对网站实施负载均衡+高可用集群需要考虑的几点
随着linux系统的成熟和广泛普及,linux运维技术越来越受到企业的关注和追捧.在一些中小企业,尤其是牵涉到电子商务和电子广告类的网站,通常会要求作负载均衡和高可用的Linux集群方案. 那么如何实 ...
Docker环境下搭建DNS LVS(keepAlived) OpenResty服务器简易集群
现在上网已经成为每个人必备的技能,打开浏览器,输入网址,回车,简单的几步就能浏览到漂亮的网页,那从请求发出到返回漂亮的页面是怎么做到的呢,我将从公司中一般的分层架构角度考虑搭建一个简易集群来实现.目标 ...

随机推荐

msyql安装
1.安装msyql yum install -y ncurses-devel automake autoconf bison libtool-ltdl-devel cd /soft wget http ...
NBUT 1220 SPY
$map$,简单模拟. #include<cstdio> #include<cstring> #include<cmath> #include<algorit ...
进入CentOS7紧急模式恢复root密码
第一步.重启CentOS7,在以下界面选择要编辑的内核(一般第一个),按e进入编辑界面第二步.在编辑界面找到如下一行,将ro改为rw init=/sysroot/bin/sh.改完后<Ctrl ...
[BZOJ4537][HNOI2016]最小公倍数(分块+并查集)
4537: [Hnoi2016]最小公倍数 Time Limit: 40 Sec Memory Limit: 512 MBSubmit: 1687 Solved: 607[Submit][Stat ...
【二分】Defense Lines
[UVa1471] Defense Lines 算法入门经典第8章8-8 (P242) 题目大意:将一个序列删去一个连续子序列,问最长的严格上升子序列 (N<=200000) 试题分析:算法1: ...
【DFS】算24点
[tyvj2802/RQNOJ74]算24点描述几十年前全世界就流行一种数字游戏,至今仍有人乐此不疲．在中国我们把这种游戏称为“算24点”.您作为游戏者将得到4个1~9之间的自然数作为操作数,而您 ...
【组合计数】UVA - 11538 - Chess Queen
考虑把皇后放在同一横排或者统一纵列,答案为nm(m-1)和nm(n-1),显然. 考虑同一对角线的情况不妨设,n<=m,对角线从左到右依次为1,2,3,...,n-1,n,n,n,...,n(m ...
python基础之re,sys,suprocess模块
re 正则表达式 1.什么是正则? 正则就是用一系列具有特殊含义的字符组成的规则,该规则用来描述具有某一特征的字符串. 正则就是用来在一个大的字符串匹配出符合规则的子字符串 2.为什么用正则? 正则可 ...
用Java Swing实现Freecell(空当接龙)
目录引言 1 游戏规则 2 界面设计和大致逻辑 2.1 界面设计 2.2 大致逻辑 3 主要功能模块设计与实现 3.1 主要思路 3.2 主要工具类 3.3 异常类 3.4 游戏初始化模块 3.5 ...
找出最小元素 Exercise07_09
import java.util.Scanner; /** * @author 冰樱梦 * 时间:2018年下半年 * 题目:找出最小元素 * */ public class Exercise07_0 ...

win下idea远程提交WordCount任务到HA集群

win下idea远程提交WordCount任务到HA集群的更多相关文章

随机推荐

热门专题