Javac词法分析

参考：《深入分析Java Web》技术内幕 许令波

词法分析过程涉及到的主要类及相关的继承关系如下：

词法分析的接口为Lexer，默认实现类为Scanner，Scanner会逐个读取Java源文件中的单个字符，然后解析出符合Java语言规范的Token序列。

举个例子，如下：

package compile;

/**
 * Created by mazhi on 2018/6/5.
 */
public class Cifa {
    int a;
    int c = a + 1;
}

其最终的语法树如下：

首先的一个语法节点为JCCompilationUnit，关于这个节点的介绍如下：

Compilation Units参考：https://docs.oracle.com/javase/specs/jls/se7/html/jls-7.html#jls-7.3

定义如下：

CompilationUnit:
    PackageDeclarationopt ImportDeclarationsopt TypeDeclarationsopt

ImportDeclarations:
    ImportDeclaration
    ImportDeclarations ImportDeclaration

TypeDeclarations:
    TypeDeclaration
    TypeDeclarations TypeDeclaration

可以看到一个CompilationUnit可以有多个TypeDeclaration声明。当前只有一个JCClassDecl节点，代码这个Cifa类。

其中涉及到了PackageDeclaration、ImportDeclaration与TypeDeclaration声明，其各个定义如下：

（1）PackageDeclaration节点定义如下：

PackageDeclaration:
    Annotationsopt package PackageName ;　　

参考：https://docs.oracle.com/javase/specs/jls/se7/html/jls-7.html#jls-7.4

（2）ImportDeclaration节点定义如下：

ImportDeclaration:
    SingleTypeImportDeclaration
    TypeImportOnDemandDeclaration
    SingleStaticImportDeclaration
    StaticImportOnDemandDeclaration　　

参考：https://docs.oracle.com/javase/specs/jls/se7/html/jls-7.html#jls-7.5

（3）TypeDeclaration节点定义如下：

ClassDeclaration:
    NormalClassDeclaration
    EnumDeclaration

NormalClassDeclaration:
    ClassModifiersopt class Identifier TypeParametersopt
                                               Superopt Interfacesopt ClassBody

参考：https://docs.oracle.com/javase/specs/jls/se7/html/jls-8.html#jls-8.1　　

查看JavaCompiler.java类中的方法parse()，代码如下：

    /**
     * Parse contents of input stream.
     *
     * @param filename The name of the file from which input stream comes.
     * @param input    The input stream to be parsed.
     */
    protected JCCompilationUnit parse(JavaFileObject filename, CharSequence content) {
        long msec = now();
        JCCompilationUnit tree = treeMaker.TopLevel(List.<JCAnnotation>nil(), null, List.<JCTree>nil());
        if (content != null) {
            if (verbose) {
                log.printVerbose("parsing.started", filename);
            }
            Parser parser = parserFactory.newParser(content, keepComments(), genEndPos, lineDebugInfo);
            tree = parser.parseCompilationUnit();
            if (verbose) {
                log.printVerbose("parsing.done", Long.toString(elapsed(msec)));
            }
        }

        tree.sourcefile = filename;

        return tree;
    }　　

这个方法有三点需要解析：

（1）参数content为源文件的字符输入流。是一个HeapCharBuffer对象，其中用char数组来保存输入的字符流内容，具体内容如下：

nextToken(0,7)=|package|
Token.PACKAGE
0 = 'p' 112
1 = 'a' 97
2 = 'c' 99
3 = 'k' 107
4 = 'a' 97
5 = 'g' 103
6 = 'e' 101

processWhitespace(7,8)=| |
7 = ' ' 32

nextToken(8,15)=|compile|
Token.IDENTIFIER
8 = 'c' 99
9 = 'o' 111
10 = 'm' 109
11 = 'p' 112
12 = 'i' 105
13 = 'l' 108
14 = 'e' 101

nextToken(15,16)=|;|
Token.SEMI
15 = ';' 59

processTerminator(16,18)=||
16 = '\r' 13
17 = '\n' 10

processTerminator(18,20)=||
18 = '\r' 13
19 = '\n' 10

processComment(20,62,JAVADOC)=|/**
 * Created by mazhi on 2018/6/5.
 */|
20 = '/' 47
21 = '*' 42
22 = '*' 42
23 = '\r' 13
24 = '\n' 10
25 = ' ' 32
26 = '*' 42
27 = ' ' 32
28 = 'C' 67
29 = 'r' 114
30 = 'e' 101
31 = 'a' 97
32 = 't' 116
33 = 'e' 101
34 = 'd' 100
35 = ' ' 32
36 = 'b' 98
37 = 'y' 121
38 = ' ' 32
39 = 'm' 109
40 = 'a' 97
41 = 'z' 122
42 = 'h' 104
43 = 'i' 105
44 = ' ' 32
45 = 'o' 111
46 = 'n' 110
47 = ' ' 32
48 = '2' 50
49 = '0' 48
50 = '1' 49
51 = '8' 56
52 = '/' 47
53 = '6' 54
54 = '/' 47
55 = '5' 53
56 = '.' 46
57 = '\r' 13
58 = '\n' 10
59 = ' ' 32
60 = '*' 42
61 = '/' 47

processTerminator(62,64)=||
62 = '\r' 13
63 = '\n' 10

nextToken(64,70)=|public|
Token.PUBLIC
64 = 'p' 112
65 = 'u' 117
66 = 'b' 98
67 = 'l' 108
68 = 'i' 105
69 = 'c' 99

processWhitespace(70,71)=| |
70 = ' ' 32

nextToken(71,76)=|class|
Token.CLASS
71 = 'c' 99
72 = 'l' 108
73 = 'a' 97
74 = 's' 115
75 = 's' 115

processWhitespace(76,77)=| |
76 = ' ' 32

nextToken(77,81)=|Cifa|
Token.IDENTIFIER
77 = 'C' 67
78 = 'i' 105
79 = 'f' 102
80 = 'a' 97

processWhitespace(81,82)=| |
81 = ' ' 32

nextToken(82,83)=|{|
Token.LBRACE
82 = '{' 123

processTerminator(83,85)=||
83 = '\r' 13
84 = '\n' 10

processWhitespace(85,89)=|    |
85 = ' ' 32
86 = ' ' 32
87 = ' ' 32
88 = ' ' 32

nextToken(89,92)=|int|
Token.INT
89 = 'i' 105
90 = 'n' 110
91 = 't' 116

processWhitespace(92,93)=| |
92 = ' ' 32

nextToken(93,94)=|a|
Token.IDENTIFIER
93 = 'a' 97

nextToken(94,95)=|;|
Token.SEMI
94 = ';' 59

processTerminator(95,97)=||
95 = '\r' 13
96 = '\n' 10

processWhitespace(97,101)=|    |
97 = ' ' 32
98 = ' ' 32
99 = ' ' 32
100 = ' ' 32

nextToken(101,104)=|int|
Token.INT
101 = 'i' 105
102 = 'n' 110
103 = 't' 116

processWhitespace(104,105)=| |
104 = ' ' 32

nextToken(105,106)=|c|
Token.IDENTIFIER
105 = 'c' 99

processWhitespace(106,107)=| |
106 = ' ' 32

nextToken(107,108)=|=|
Token.EQ
107 = '=' 61

processWhitespace(108,109)=| |
108 = ' ' 32

nextToken(109,110)=|a|
Token.IDENTIFIER
109 = 'a' 97

processWhitespace(110,111)=| |
110 = ' ' 32

nextToken(111,112)=|+|
Token.PLUS
111 = '+' 43

processWhitespace(112,113)=| |
112 = ' ' 32

nextToken(113,114)=|1|
Token.INTLITERAL
113 = '1' 49

nextToken(114,115)=|;|
Token.SEMI
114 = ';' 59

processTerminator(115,117)=||
115 = '\r' 13
116 = '\n' 10

nextToken(117,118)=|}|
Token.RBRACE
117 = '}' 125

processTerminator(118,120)=||
118 = '\r' 13
119 = '\n' 10

nextToken(120,120)=||
120 = '\u001A' 26

121 = '\u0000' 0
122 = '\u0000' 0
123 = '\u0000' 0
124 = '\u0000' 0
125 = '\u0000' 0
126 = '\u0000' 0
127 = '\u0000' 0
128 = '\u0000' 0
129 = '\u0000' 0

　　

（2）通过调用treeMaker的TopLevel()方法来创建JCCompilationUnit节点，其TopLevel()方法的具体实现如下：

 /**
     * Create given tree node at current position.
     * @param defs a list of ClassDef, Import, and Skip
     */
    public JCCompilationUnit TopLevel(List<JCAnnotation> packageAnnotations,
                                      JCExpression pid,
                                      List<JCTree> defs) {
        Assert.checkNonNull(packageAnnotations);
        for (JCTree node : defs) {
            // cond/msg
            Assert.check(node instanceof JCClassDecl
                            || node instanceof JCImport
                            || node instanceof JCSkip
                            || node instanceof JCErroneous
                            || (node instanceof JCExpressionStatement && ((JCExpressionStatement) node).expr instanceof JCErroneous),
                    node.getClass().getSimpleName());
        }
        JCCompilationUnit tree = new JCCompilationUnit(packageAnnotations, pid, defs,
                                     null, null, null, null);
        tree.pos = pos;
        return tree;
    }
　　

　　

（3）调用parserFactory类的newParser()工厂方法生成一个Parser类单实例对象，然后调用parseCompilationUnit()方法，其源代码如下：

/** CompilationUnit =
     *   [ { "@" Annotation } PACKAGE Qualident ";"]
     *   {ImportDeclaration}
     *   {TypeDeclaration}
     */
    public JCCompilationUnit parseCompilationUnit() {
        int pos = S.pos();
        JCExpression pid = null;
        String dc = S.docComment();
        JCModifiers mods = null;
        List<JCAnnotation> packageAnnotations = List.nil();
        if (S.token() == MONKEYS_AT) // @ 符号
            mods = modifiersOpt(); // 解析修饰符

        if (S.token() == PACKAGE) { // 解析package声明
            if (mods != null) {
                checkNoMods(mods.flags);
                packageAnnotations = mods.annotations;
                mods = null;
            }
            S.nextToken();
            pid = qualident();
            accept(SEMI); // 分号
        }
        ListBuffer<JCTree> defs = new ListBuffer<JCTree>();
        boolean checkForImports = true;
        while (S.token() != EOF) {
            if (S.pos() <= errorEndPos) {
                // error recovery
                // 跳过错误字符
                skip(checkForImports, false, false, false);
                if (S.token() == EOF)
                    break;
            }
            if (checkForImports && mods == null && S.token() == IMPORT) {
                defs.append(importDeclaration()); //  解析import声明
            } else { // 解析class类主体
                JCTree def = typeDeclaration(mods);
                if (keepDocComments && dc != null && docComments.get(def) == dc) {
                    // If the first type declaration has consumed the first doc
                    // comment, then don't use it for the top level comment as well.
                    // 如果在前面的类型声明中已经解析过了，那么在top level中将不再重复解析
                    dc = null;
                }
                if (def instanceof JCExpressionStatement)
                    def = ((JCExpressionStatement)def).expr;
                defs.append(def);
                if (def instanceof JCClassDecl)
                    checkForImports = false;
                mods = null;
            }
        }
        JCCompilationUnit toplevel = F.at(pos).TopLevel(packageAnnotations, pid, defs.toList());
        attach(toplevel, dc);
        if (defs.elems.isEmpty())
            storeEnd(toplevel, S.prevEndPos());
        if (keepDocComments)
            toplevel.docComments = docComments;
        if (keepLineMap)
            toplevel.lineMap = S.getLineMap();
        return toplevel;
    }