Analyzer原理

【常用分词器】

• SimpleAnalyzer
• StopAnalyzer
• WhitespaceAnalyzer
• StandardAnalyze

【TokenStream】

she is a student ==〉TokenStream

TokenStream有2个实现类。Tokenizer、TokenFilter

1)  Tokenizer

将数据进行分割形成一定的语汇(所谓语汇是指一个一个独立的词语。)。最终结果将形成TokenStream。

2) TokenFilter

按照规则对语汇进行过滤。如：StopFilter可以对停用词进行过滤。

3) 执行过程

【存储方式】

【应用TokenStream】

     /**
      * 使用TokenStream进行分词
      * @param str
      * @param analyzer
      */
     public static void displayTokenStream(String str, Analyzer analyzer){
         try {
             //通过Analayer获取TokenStream
             //toenStream("域名称或文件名"，输入流对象)
             TokenStream stream = analyzer.tokenStream("content", new StringReader(str));

             //向流中添加一个属性
             //容器，存储每次分词所对应的语汇内容
             CharTermAttribute charAttr = stream.addAttribute(CharTermAttribute.class);

             //通过循环语句读取语汇的内容
             while(stream.incrementToken()){
                 System.out.print("[" + charAttr + "]  ");
             }
             System.out.println();
         } catch (IOException e) {
             e.printStackTrace();
         }
     }

     private Version version = Version.LUCENE_35;

     /**
      * 测试TokenStream(英文内容)
      */
     @Test
     public void test01(){
         String str = "I'm come from Hanlin，I love Hanlin";
         System.out.println("str = " + str);
         System.out.println("====================================");

         //创建Analyzer对象
         Analyzer a1 = new SimpleAnalyzer(version);
         Analyzer a2 = new StopAnalyzer(version);
         Analyzer a3 = new WhitespaceAnalyzer(version);
         Analyzer a4 = new StandardAnalyzer(version);

         //测试TokenStream
         AnalyzerUtil.displayTokenStream(str, a1);
         AnalyzerUtil.displayTokenStream(str, a2);
         AnalyzerUtil.displayTokenStream(str, a3);
         AnalyzerUtil.displayTokenStream(str, a4);
     }

     /**
      * 测试TokenStream(中文内容)
      */
     @Test
     public void test02(){
         String str = "我来自翰林，我爱翰林";
         System.out.println("str = " + str);
         System.out.println("====================================");

         //创建Analyzer对象
         Analyzer a1 = new SimpleAnalyzer(version);
         Analyzer a2 = new StopAnalyzer(version);
         Analyzer a3 = new WhitespaceAnalyzer(version);
         Analyzer a4 = new StandardAnalyzer(version);

         //测试TokenStream
         AnalyzerUtil.displayTokenStream(str, a1);
         AnalyzerUtil.displayTokenStream(str, a2);
         AnalyzerUtil.displayTokenStream(str, a3);
         AnalyzerUtil.displayTokenStream(str, a4);
     }

TokenStream可以读取到分词内容.

【Attribute】

     /**
      * 显示语汇的基本属性
      * @param str
      * @param anlyzer
      */
     public static void displayAttributes(String str, Analyzer anlyzer){
         try {
             //获取TokenStream对象
             TokenStream stream = anlyzer.tokenStream("content", new StringReader(str));

             //PositionIncrementAttribute ：存储了语汇之间的位置增量
             //添加PositionIncrementAttribute属性
             PositionIncrementAttribute positionAttr = stream.addAttribute(PositionIncrementAttribute.class);

             //添加CharTermAttrbute
             CharTermAttribute charAttr = stream.addAttribute(CharTermAttribute.class);
             //OffsetAttribute：获取语汇的偏移数据
             OffsetAttribute offsetAttr = stream.addAttribute(OffsetAttribute.class);

             //语汇的分词方式类型（了解）
             TypeAttribute typeAttr = stream.addAttribute(TypeAttribute.class);

             //遍历每一个语汇
             while(stream.incrementToken()){
                 System.out.print(positionAttr.getPositionIncrement() + "、");
                 System.out.print("[" + charAttr + "  : " + offsetAttr.startOffset() + "~" + offsetAttr.endOffset()+ "(" + typeAttr.type()+ "）] " );
             }
             System.out.println();

         } catch (IOException e) {
             e.printStackTrace();
         }
     }

     /**
      * 测试属性的应用
      */
     @Test
     public void test03(){
         String str = "I'm come from Hanlin，I love Hanlin";
         System.out.println("str = " + str);
         System.out.println("====================================");

         //创建Analyzer对象
         Analyzer a1 = new SimpleAnalyzer(version);
         Analyzer a2 = new StopAnalyzer(version);
         Analyzer a3 = new WhitespaceAnalyzer(version);
         Analyzer a4 = new StandardAnalyzer(version);

         //测试TokenStream
         AnalyzerUtil.displayAttributes(str, a1);
         AnalyzerUtil.displayAttributes(str, a2);
         AnalyzerUtil.displayAttributes(str, a3);
         AnalyzerUtil.displayAttributes(str, a4);
     }

FlagsAttribute：标志位属性信息(了解)

PayloadAttribute：负载属性信息(了解)

说明:每一个语汇单元都存在一定的属性.通过Attribute可以获取到相关的语汇信息。