使用flatbuffers

问题

张三是个java程序员,他写产生数据的程序.李四是个python程序员,他要用python处理张三产生的数据.最直观常用的方法就是张三用java把产生的数据保存成csv或者xml文件,然后李四用python读取csv或xml文件.这没有问题.但现在有一种性能更好的方法,flatbuffers.

作用

可以把flatbuffers理解成一个可执行文件flatc.这个可执行文件可以把表示数据格式的fbs文件转化成c++,java,python...代码.张三擅长使用java,他就用flatc把fbs文件转化成java代码.用转化好的java代码把数据存成flatbuffers的格式.李四用flatc把同样的fbs文件转化成自己熟悉的python代码,李四可以用这份python代码把flatbuffers格式文件中的数据读出来.这就实现了不同语言间传数据.当然这只是一个简单的使用场景.flatbuffers的作用远不只是这些.

使用例子

先写一个表示数据格式的文件school.fbs

namespace school;

table School {
    name:string;
    students:[Student];
}

table Student {
    number:short;
    name:string;
}
root_type School;

具体的语法规则看参考资料,只简单的解释下.上面代码定义了一个根结点为School的数据.里面的student有2个成员变量number,name. School也有2个成员变量,一个是name代码school的名字.一个是Student的数组.

然后执行下面命令:

flatc.exe -p school.fbs

会生成一堆python代码.再用生成的python创建一个flatbuffers文件,代码如下：

# coding:utf-8
from __future__ import print_function
from __future__ import division

import flatbuffers

from school import School
from school import Student

builder = flatbuffers.Builder(1024)

stu1 = builder.CreateString('zhang3')
stu2 = builder.CreateString('li4')

Student.StudentStart(builder)
Student.StudentAddName(builder, stu1)
Student.StudentAddNumber(builder, 3)
zhang3 = Student.StudentEnd(builder)

Student.StudentStart(builder)
Student.StudentAddName(builder, stu2)
Student.StudentAddNumber(builder, 4)
li4 = Student.StudentEnd(builder)

School.SchoolStartStudentsVector(builder, 2)
builder.PrependUOffsetTRelative(li4)
builder.PrependUOffsetTRelative(zhang3)
students = builder.EndVector(2)

School.SchoolStart(builder)
School.SchoolAddStudents(builder, students)
school_result = School.SchoolEnd(builder)

builder.Finish(school_result)

buf = builder.Output()

new_school = School.School.GetRootAsSchool(buf, 0)
print(new_school.Students(0).Name())
print(new_school.Students(0).Number())
print(new_school.Students(1).Name())
print(new_school.Students(1).Number())

with open("result.bin", mode="wb") as f:
    f.write(buf)

这段代码的作用是新建2个student，一个zhang3,一个li4,然后把student数据保存到School里,最后生成文件result.bin. result.bin就是flatbuffers格式的文件,本质下就是二进制文件.

再执行下面命令:

flatc.exe -j school.fbs

生成一堆java代码.可以用java代码把result.bin文件中的数据读出来.代码如下:

package com.company;
import school.School;

import java.io.*;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

public class Main {
    private void process() {
        Path path = Paths.get("D:\\yourpath\\result.bin");
        try {
            byte[] data =  Files.readAllBytes(path);
            ByteBuffer buf = java.nio.ByteBuffer.wrap(data);
            School school = School.getRootAsSchool(buf);
            System.out.println(school.students(0).name());
            System.out.println(school.students(0).number());
            System.out.println(school.students(1).name());
            System.out.println(school.students(1).number());
        } catch (IOException e) {
            System.out.println("IOException");
        }

    }

    public static void main(String[] args) {
        Main m = new Main();
        m.process();
    }
}

另一个名字

google在tensorflow lite中把result.bin这类二进制文件起一个高大上的名字——tflite

参考资料

1. FlatBuffers Documentation
2. flatbuffers源代码