搭建maven开发环境测试Hadoop组件HDFS文件系统的一些命令

1.PC已经安装Eclipse Software，测试平台windows10及Centos6.8虚拟机

2.新建maven project

3.打开pom.xml,maven工程项目的pom文件加载以下内容

 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.neusoft</groupId>
     <artifactId>bigdata001</artifactId>
     <version>0.0.1-SNAPSHOT</version>
     <name>bigdata001</name>
     <dependencies>
         <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
     <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-common</artifactId>
         <version>2.6.0</version>
     </dependency>
     <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
     <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-hdfs</artifactId>
         <version>2.6.0</version>
     </dependency>
     <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
     <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-client</artifactId>
         <version>2.6.0</version>
     </dependency>
         <dependency>
         <groupId>jdk.tools</groupId>
         <artifactId>jdk.tools</artifactId>
         <version>1.7</version>
         <scope>system</scope>
         <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
     </dependency>
     <!-- https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper -->
     <dependency>
         <groupId>org.apache.zookeeper</groupId>
         <artifactId>zookeeper</artifactId>
         <version>3.4.6</version>
     </dependency>

     </dependencies>

     <build>
         <plugins>
             <plugin>
             <groupId>org.apache.maven.plugins</groupId>
             <artifactId>maven-compiler-plugin</artifactId>
             <version>2.3.2</version>
             <configuration>
                 <encoding>UTF-8</encoding>
                 <source>1.7</source>
                 <target>1.7</target>
                 <showWarning>true</showWarning>
             </configuration>
             </plugin>
         </plugins>
     </build>
 </project>

pom.xml

4.如下图所示，下一步在src/main/java/下新建hdfs包，并新建java class：FileSystemTest.java

5.FileSystemTest.java代码内容如下：

 package Hdfs;

 import java.net.URI;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;

 public class FileSystemTest {
     public static void main(String[] args) throws Exception {
         FileSystem fileSystem = FileSystem.newInstance(new URI("hdfs://neusoft-master:9000"), new Configuration());
         FileStatus[] listStatus = fileSystem.listStatus(new Path("/"));
         for (FileStatus fileStatus : listStatus) {
             System.out.println(fileStatus);
         }
     }
 }

6.运行结果如下所示

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
FileStatus{path=hdfs://neusoft-master:9000/hbase; isDirectory=true; modification_time=1483712306703; access_time=0; owner=root; group=supergroup; permission=rwxr-xr-x; isSymlink=false}
FileStatus{path=hdfs://neusoft-master:9000/tmp; isDirectory=true; modification_time=1483709831059; access_time=0; owner=root; group=supergroup; permission=rwx------; isSymlink=false}
FileStatus{path=hdfs://neusoft-master:9000/user; isDirectory=true; modification_time=1483709981792; access_time=0; owner=root; group=supergroup; permission=rwxr-xr-x; isSymlink=false}

Console view

运行java类可以显示hadoop的HDFS文件系统下面各个目录。

7.在VM虚拟机搭建的Hadoop伪分布式环境进行测试。

8.总结

上述步骤总结了如何通过在Windows平台的Eclipse平台下通过编码实现查看，hdfs文件系统内容。

备注：编程时：用到了hdfs://neusoft-master:9000的neusoft-master需要在windows及linux平台进行设置

windows系统需要在C:\Windows\System32\drivers\etc以管理员打开并添加“192.168.191.130 NEUSOFT-MASTER”

Linux系统需要修改主机名(vi /etc/sysconfig/network修改hostname)及修改vi /etc/hosts对应的ip和host