Handwritten Parsers & Lexers in Go (Gopher Academy Blog)
Handwritten Parsers & Lexers in Go
(原文地址 https://blog.gopheracademy.com/advent-2014/parsers-lexers/)
In these days of web apps and REST APIs it seems that writing parsers is a dying art. You may think parsers are a complex undertaking only reserved for programming language designers but I’d like to dispel this idea. Over the past few years I’ve written parsers for JSON, CSS3, and database query languages and the more that I write parsers the more that I love them.
The Basics
Let’s start off with the basics: what is a lexer and what is a parser? When we parse a language (or, technically, a “formal grammar”) we do it in two phases. First we break up series of characters into tokens. For a SQL-like language these tokens may be “whitespace”, “number”, “SELECT”, etc. This process is called lexing (or tokenizing or scanning).
Take this simple SQL SELECT statement as an example:
SELECT * FROM mytable
When we tokenize this string we’d see it as:
`SELECT` • `WS` • `ASTERISK` • `WS` • `FROM` • `WS` • `STRING<"mytable">`
This process, called lexical analysis, is similar to how we break up words in a sentence when we read. These tokens then get fed to a parser which performs semantic analysis.
The parser’s job is to make sense of these tokens and make sure they’re in the right order. This is similar to how we derive meaning from combining words in a sentence. Our parser will construct an abstract syntax tree (AST) from our series of tokens and the AST is what our application will use.
In our SQL SELECT example, our AST may look like:
type SelectStatement struct {
Fields []string
TableName string
}
Parser Generators
Many people use parser generators to automatically write a parser and lexer for them. There are many tools made to do this: lex, yacc, ragel. There’s even a Go implementation of yacc
built into the go
toolchain.
However, after using parser generators many times I’ve found them to be problematic. First, they involve learning a new language to declare your language format. Second, they’re difficult to debug. For example, try reading the Ruby language’s yacc file. Eek!
After watching a talk by Rob Pike on lexical scanning and reading the implementation of the go
standard library package, I realized how much easier and simpler it is to hand write your parser and lexer. Let’s walk through the process with a simple example.
Writing a Lexer in Go
Defining our tokens
Let’s start by writing a simple parser and lexer for SQL SELECT statements. First, we need to define what tokens we’ll allow in our language. We’ll only allow a small subset of the SQL language:
// Token represents a lexical token.
type Token int
const (
// Special tokens
ILLEGAL Token = iota
EOF
WS
// Literals
IDENT // fields, table_name
// Misc characters
ASTERISK // *
COMMA // ,
// Keywords
SELECT
FROM
)
We’ll use these tokens to represent series of characters. For example, WS
will represent one or more whitespace characters and IDENT
will represent an identifier such as a field name or a table name.
Defining character classes
It’s useful to define functions that will let us check the type of character. Here we’ll define two functions: one to check if a character is whitespace and one to check if the character is a letter.
func isWhitespace(ch rune) bool {
return ch == ' ' || ch == '\t' || ch == '\n'
}
func isLetter(ch rune) bool {
return (ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z')
}
It’s also useful to define an “EOF” rune so that we can treat EOF like any other character:
var eof = rune(0)
Scanning our input
Next we’ll want to define our Scanner
type. This type will wrap our input reader with a bufio.Reader
so we can peek ahead at characters. We’ll also add helper functions for reading and unreading characters from our underlying reader.
// Scanner represents a lexical scanner.
type Scanner struct {
r *bufio.Reader
}
// NewScanner returns a new instance of Scanner.
func NewScanner(r io.Reader) *Scanner {
return &Scanner{r: bufio.NewReader(r)}
}
// read reads the next rune from the bufferred reader.
// Returns the rune(0) if an error occurs (or io.EOF is returned).
func (s *Scanner) read() rune {
ch, _, err := s.r.ReadRune()
if err != nil {
return eof
}
return ch
}
// unread places the previously read rune back on the reader.
func (s *Scanner) unread() { _ = s.r.UnreadRune() }
The entry function into Scanner
will be the Scan()
method which return the next token and the literal string that it represents:
// Scan returns the next token and literal value.
func (s *Scanner) Scan() (tok Token, lit string) {
// Read the next rune.
ch := s.read()
// If we see whitespace then consume all contiguous whitespace.
// If we see a letter then consume as an ident or reserved word.
if isWhitespace(ch) {
s.unread()
return s.scanWhitespace()
} else if isLetter(ch) {
s.unread()
return s.scanIdent()
}
// Otherwise read the individual character.
switch ch {
case eof:
return EOF, ""
case '*':
return ASTERISK, string(ch)
case ',':
return COMMA, string(ch)
}
return ILLEGAL, string(ch)
}
This entry function starts by reading the first character. If the character is whitespace then it is consumed with all contiguous whitespace characters. If it’s a letter then it’s treated as the start of an identifier or keyword. Otherwise we’ll check to see if it’s one of our single character tokens.
Scanning contiguous characters
When we want to consume multiple characters in a row we can do this in a simple loop. Here in scanWhitespace()
we’ll consume whitespace characters until we hit a non-whitespace character:
// scanWhitespace consumes the current rune and all contiguous whitespace.
func (s *Scanner) scanWhitespace() (tok Token, lit string) {
// Create a buffer and read the current character into it.
var buf bytes.Buffer
buf.WriteRune(s.read())
// Read every subsequent whitespace character into the buffer.
// Non-whitespace characters and EOF will cause the loop to exit.
for {
if ch := s.read(); ch == eof {
break
} else if !isWhitespace(ch) {
s.unread()
break
} else {
buf.WriteRune(ch)
}
}
return WS, buf.String()
}
The same logic can be applied to scanning our identifiers. Here in scanIdent()
we’ll read all letters and underscores until we hit a different character:
// scanIdent consumes the current rune and all contiguous ident runes.
func (s *Scanner) scanIdent() (tok Token, lit string) {
// Create a buffer and read the current character into it.
var buf bytes.Buffer
buf.WriteRune(s.read())
// Read every subsequent ident character into the buffer.
// Non-ident characters and EOF will cause the loop to exit.
for {
if ch := s.read(); ch == eof {
break
} else if !isLetter(ch) && !isDigit(ch) && ch != '_' {
s.unread()
break
} else {
_, _ = buf.WriteRune(ch)
}
}
// If the string matches a keyword then return that keyword.
switch strings.ToUpper(buf.String()) {
case "SELECT":
return SELECT, buf.String()
case "FROM":
return FROM, buf.String()
}
// Otherwise return as a regular identifier.
return IDENT, buf.String()
}
This function also checks at the end if the literal string is a reserved word. If so then a specialized token is returned.
Writing a Parser in Go
Setting up the parser
Once we have our lexer ready, parsing a SQL statement becomes easier. First let’s define our Parser
:
// Parser represents a parser.
type Parser struct {
s *Scanner
buf struct {
tok Token // last read token
lit string // last read literal
n int // buffer size (max=1)
}
}
// NewParser returns a new instance of Parser.
func NewParser(r io.Reader) *Parser {
return &Parser{s: NewScanner(r)}
}
Our parser simply wraps our scanner but also adds a buffer for the last read token. We’ll define helper functions for scanning and unscanning so we can use this buffer:
// scan returns the next token from the underlying scanner.
// If a token has been unscanned then read that instead.
func (p *Parser) scan() (tok Token, lit string) {
// If we have a token on the buffer, then return it.
if p.buf.n != 0 {
p.buf.n = 0
return p.buf.tok, p.buf.lit
}
// Otherwise read the next token from the scanner.
tok, lit = p.s.Scan()
// Save it to the buffer in case we unscan later.
p.buf.tok, p.buf.lit = tok, lit
return
}
// unscan pushes the previously read token back onto the buffer.
func (p *Parser) unscan() { p.buf.n = 1 }
Our parser also doesn’t care about whitespace at this point so we’ll define a helper function to find the next non-whitespace token:
// scanIgnoreWhitespace scans the next non-whitespace token.
func (p *Parser) scanIgnoreWhitespace() (tok Token, lit string) {
tok, lit = p.scan()
if tok == WS {
tok, lit = p.scan()
}
return
}
Parsing the input
Our parser’s entry function will be the Parse()
method. This function will parse the next SELECT statement from the reader. If we had multiple statements in our reader then we could call this function repeatedly.
func (p *Parser) Parse() (*SelectStatement, error)
Let’s break this function down into small parts. First we’ll define the AST structure we want to return from our function:
stmt := &SelectStatement{}
Then we’ll make sure there’s a SELECT
token. If we don’t see the token we expect then we’ll return an error to report the string we found instead.
if tok, lit := p.scanIgnoreWhitespace(); tok != SELECT {
return nil, fmt.Errorf("found %q, expected SELECT", lit)
}
Next we want to parse a comma-delimited list of fields. In our parser we’re just considering identifiers and an asterisk as possible fields:
for {
// Read a field.
tok, lit := p.scanIgnoreWhitespace()
if tok != IDENT && tok != ASTERISK {
return nil, fmt.Errorf("found %q, expected field", lit)
}
stmt.Fields = append(stmt.Fields, lit)
// If the next token is not a comma then break the loop.
if tok, _ := p.scanIgnoreWhitespace(); tok != COMMA {
p.unscan()
break
}
}
After our field list we want to see a FROM
keyword:
// Next we should see the "FROM" keyword.
if tok, lit := p.scanIgnoreWhitespace(); tok != FROM {
return nil, fmt.Errorf("found %q, expected FROM", lit)
}
Then we want to see the name of the table we’re selecting from. This should be an identifier token:
tok, lit := p.scanIgnoreWhitespace()
if tok != IDENT {
return nil, fmt.Errorf("found %q, expected table name", lit)
}
stmt.TableName = lit
If we’ve gotten this far then we’ve successfully parsed a simple SQL SELECT statement so we can return our AST structure:
return stmt, nil
Congrats! You’ve just built a working parser!
Diving in deeper
You can find the full source of this example (with tests) at:
This parser example was heavily influenced by the InfluxQL parser. If you’re interested in diving deeper and understanding multiple statement parsing, expression parsing, or operator precedence then I encourage you to check out the repository:
If you have any questions or just love chatting about parsers, please find me on Twitter at @benbjohnson.
Handwritten Parsers & Lexers in Go (Gopher Academy Blog)的更多相关文章
- Handwritten Parsers & Lexers in Go (翻译)
用go实现Parsers & Lexers 在当今网络应用和REST API的时代,编写解析器似乎是一种垂死的艺术.你可能会认为编写解析器是一个复杂的工作,只保留给编程语言设计师,但我想消除这 ...
- Writing a simple Lexer in PHP/C++/Java
catalog . Comparison of parser generators . Writing a simple lexer in PHP . phc . JLexPHP: A PHP Lex ...
- Python的开源人脸识别库:离线识别率高达99.38%
Python的开源人脸识别库:离线识别率高达99.38% github源码:https://github.com/ageitgey/face_recognition#face-recognitio ...
- Golang Channel用法简编
转自:http://tonybai.com/2014/09/29/a-channel-compendium-for-golang/ 在进入正式内容前,我这里先顺便转发一则消息,那就是Golang 1. ...
- Conditions in bash scripting (if statements)
Shell中判断语句if中-z至-d的意思 - sunny_2015 - 博客园 https://www.cnblogs.com/coffy/p/5748292.html Conditions in ...
- Python的开源人脸识别库:离线识别率高达99.38%(附源码)
Python的开源人脸识别库:离线识别率高达99.38%(附源码) 转https://cloud.tencent.com/developer/article/1359073 11.11 智慧上云 ...
- C#解析Markdown文档,实现替换图片链接操作
前言 又是好久没写博客了 其实也不是没写,是最近在「做一个博客」,从2月21日开始,大概一个多星期的时间,疯狂刷进度,边写代码边写了一整系列的博客开发笔记,目前为止已经写了16篇了,然后上3月之后工作 ...
- 【三】用Markdown写blog的常用操作
本系列有五篇:分别是 [一]Ubuntu14.04+Jekyll+Github Pages搭建静态博客:主要是安装方面 [二]jekyll 的使用 :主要是jekyll的配置 [三]Markdown+ ...
- 使用神经网络识别手写数字Using neural nets to recognize handwritten digits
The human visual system is one of the wonders of the world. Consider the following sequence of handw ...
随机推荐
- MFC中小笔记(二)
6.有三个API函数可以运行可执行文件WinExec.ShellExecute和CreateProcess. 关于这三者的概述总结,有好几篇,自己选择. 1.CreateProcess因为使用复杂, ...
- ##4.Glance 镜像服务-- openstack pike
##4.Glance 镜像服务 openstack pike 安装 目录汇总 http://www.cnblogs.com/elvi/p/7613861.html ##.Glance 镜像服务.txt ...
- VMware12提示 已将该虚拟机配置为使用 64 位客户机操作系统。但是,无法执行 64 位操作。
VMware12提示 已将该虚拟机配置为使用 64 位客户机操作系统.但是,无法执行 64 位操作. 此主机支持 Intel VT-x,但 Intel VT-x 处于禁用状态 解决办法: 下载LeoM ...
- JavaScript tips ——搞定闰年
前言 处理时间时,常常要考虑用户的输入是否合法,其中一个很典型的场景就是平闰年的判断,网上其实有很多类似的算法,但是其实不必那么麻烦,下面我讲讲的我的思路. 规则 公元年数可被4整除为闰年,但是整百( ...
- webrtc视频数据解码处理流程
- IQKeyboardManager 状态栏(status bar)问题
因为懒,所以具体什么样子,参考下面的链接 具体的问题情况参考:StatusBar background problem #1158 我解决的思路很简单,就是在监听键盘消失的时候,去设置 statys ...
- 第五章 MVC之 FileResult 和 JS请求二进制文件
一.FileResult 1.简介 表示一个用于将二进制文件内容发送到响应的基类.它有三个子类: FileContentResultFilePathResultFileStreamResult 推荐阅 ...
- 【DevOps】团队敏捷开发系列--开山篇
随着软件发布迭代的频率越来越高,传统的「瀑布型」(开发-测试-发布)模式已经不能满足快速交付的需求.2009 年左右 DevOps 应运而生,开发运维一体化,通过自动化工具与流程让整个软件开发构建.测 ...
- ITM事件直接接收并解析
之前在实施一个监控项目时.客户由于买了IBM的小机.当前就赠送了TIVOLI的系统监控软件一套,客户也在他们的生产环境中部署了ITM的监控.由于没有购买IBM的netcool,无法集中管理告警事件,请 ...
- 项目实战13—企业级虚拟化Virtualization-KVM技术
项目实战系列,总架构图 http://www.cnblogs.com/along21/p/8000812.html KVM的介绍.准备工作和qemu-kvm 命令详解 1.介绍 (1)介绍 KVM:就 ...