SequenceFile实例操作

HDFS API提供了一种二进制文件支持，直接将<key,value>对序列化到文件中，该文件格式是不能直接查看的，可以通过hadoop  dfs -text命令查看，后面跟上SequenceFile的HDFS路径

通过写入SequenceFile和读入SequenceFile文件，打成jar包在Hadoop环境中运行。

 1.写入SequenceFile代码：

 package Hdfs;

 import java.io.IOException;
 import java.net.URI;

 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.io.IOUtils;
 import org.apache.hadoop.io.IntWritable;
 import org.apache.hadoop.io.SequenceFile;
 import org.apache.hadoop.io.Text;

 public class SequenceFileWriter {
     private static final String[] text={
         "床前明月光",
         "疑似地上霜",
         "举头望明月",
         "低头思故乡"
     };
     public static void main(String[] args) {
         String uri="hdfs://neusoft-master:9000/user/root/test/demo1";
         Configuration conf=new Configuration();
         SequenceFile.Writer writer=null;

         try {
             FileSystem fs= FileSystem.get(URI.create(uri), conf);
             Path path = new Path(uri);
             IntWritable key = new IntWritable();
             Text value = new Text();
             writer = SequenceFile.createWriter(fs, conf, path, key.getClass(), value.getClass());
             for (int i = 0; i < 100; i++) {
                 key.set(100-i);
                 value.set(text[i%text.length]);
                 writer.append(key, value);
             }
         } catch (IOException e) {
             e.printStackTrace();
         }finally{
             IOUtils.closeStream(writer);
         }
     }
 }

SequenceFileWriter

windows上打包成testseq.jar包，通过SecureFx上传到Linux对应目录中。

以下是linux操作步骤：

问题：如果CRT显示乱码如何解决？

在会话选项修改字符编码即可。

  2.读入SequenceFile代码：

 package Hdfs;

 import java.io.IOException;
 import java.net.URI;

 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.io.IOUtils;
 import org.apache.hadoop.io.SequenceFile;
 import org.apache.hadoop.io.Writable;
 import org.apache.hadoop.util.ReflectionUtils;

 public class SequenceFileReader {

     public static void main(String[] args) {
         String uri="hdfs://neusoft-master:9000/user/root/test/demo1";
         Configuration conf = new Configuration();
         SequenceFile.Reader reader =null;
         try {
             FileSystem fs = FileSystem.get(URI.create(uri),conf);
             Path path = new Path(uri);
             reader=new SequenceFile.Reader(fs, path,conf);
             Writable key = (Writable)ReflectionUtils.newInstance(reader.getKeyClass(), conf);
             Writable value =(Writable)ReflectionUtils.newInstance(reader.getValueClass(), conf);
             long position = reader.getPosition();
             while (reader.next(key,value)) {
                 System.out.printf("[%s]\t%s\n",key,value);
                 position=reader.getPosition();
             }
         } catch (IOException e) {
             e.printStackTrace();
         }finally{
             IOUtils.closeStream(reader);
         }

     }
 }

SequenceFileReader

windows上打包成testseq.jar包，在程序中指定主類，提交jar包時無需指定，通过SecureFx上传到Linux对应目录中。

以下是linux操作步骤：

END::SequenceFile

可通過IP：50070訪問