build doris 0.11.5 on centos 7/ubuntu

doris has envolved many thirdparty components since v0.9. so the build progress has changed a lot since v0.9, in this article, I will demostrate the process about building doris v0.11.5 on centos 7. In the building process, I have encountered some issues, and thanks to the help of May who a key engineer of doris team, I solved them.

1. install dependencies which is required for build doris on local machine

sudo yum install gcc-c++ libstdc++-static ant cmake byacc flex automake libtool binutils-devel bison ncurses-devel  && \
sudo updatedb  && \
sudo yum install redhat-lsb

2. download source from github

wget https://codeload.github.com/baidu-doris/incubator-doris/tar.gz/DORIS-0.11.5-alpha
tar -xzvf incubator-doris-DORIS-0.11.-alpha-src.tar.gz
cd incubator-doris-DORIS-0.11.-alpha-src

3. build third party components

the thirdparty components download is very slow, I have upload the thirdparty tarball onto pan.baidu.com, we can download it there to improve the process.

链接: https://pan.baidu.com/s/1kxW07GTh0Jx66f1AYfgPsw 提取码: b7ki, to use the tarball, just unzip it and move the contents into incubator-doris-DORIS-0.11.5-alpha-src/thirdparty/src folder.

cd thirdparty
sh build-thirdparty.sh

the thirdparty components reqiure GCC 5+ to build!  which has been changed since v0.8.2.1 to be compatible with ubuntu 18.04 to build doris, because the llvm component failed to build on ununtu 18.04, and the root cause is the glibc version on ununtu is higher than centos, so the doris team has made this change on centos( reqiure GCC 5+ to build the souce)

if you have installed gcc 5+ on centos, then the thirdparty components would build success. for install gcc 5+ on centos 7, please refere my previous post   in the section of 3. 安装GCC v5.2+

but there's another issue about build the source which I would refer below!!!

4. Build doris product source

4.1 build be source on centos 7

cd incubator-doris-DORIS-0.11.-alpha-src && sh build.sh  --be

In my build progess, I have encounted issues as follows:

/root/workspace/palo-0.11./be/src/olap/key_coder.cpp: In member function ‘void doris::KeyCoderResolver::add_mapping()’:
/root/workspace/palo-0.11./be/src/olap/key_coder.cpp::: error: ‘class std::unordered_map<doris::FieldType, doris::KeyCoder*>’ has no member named ‘emplace’
         _coder_map.emplace(field_type, new KeyCoder(KeyCoderTraits<field_type>()));
                    ^
make[]: *** [src/olap/CMakeFiles/Olap.dir/key_coder.cpp.o] Error
make[]: *** [src/olap/CMakeFiles/Olap.dir/all] Error
make[]: *** Waiting for unfinished jobs....
[ %] Building CXX object src/olap/rowset/CMakeFiles/Rowset.dir/rowset_id_generator.cpp.o
[ %] Building CXX object src/exec/CMakeFiles/Exec.dir/merge_join_node.cpp.o

the reason for this failure is: doris require GCC 7+ to build the product source even if the thirdparty components has build success!!! So the solution is to upgrade local GCC to version 7+, installing GCC 7+, please refer to my post centos7 gcc升级

since the thirdparty is build using GCC lower than 7, so you need to rebuild the thirdparth components,or else the product code would failed.

BTW, if you have seem the following error message during building product code:

[  %] Generating /workspace/palo-0.11./gensrc/build/geo/wkt_yacc.y.cpp, /root/workspace/palo-0.11./gensrc/build/geo/wkt_yacc.y.hpp
/root/workspace/palo-0.11./be/src/geo/wkt_yacc.y:56.20-: error: syntax error, unexpected {...}
make[]: *** [/root/workspace/palo-0.11./gensrc/build/geo/wkt_yacc.y.cpp] Error
make[]: *** [src/geo/CMakeFiles/Geo.dir/all] Error 

it most likely the version issue caused by yacc,  the doris has changed the dependency of yacc into byacc, the soluton is to  install byacc as mentioned in section 1, then this issue would be resolved. this dependency is introduced since v0.11.

after upgrade GCC to version above 7.0+, you build for thirdparty component would encouter error like

[ %] Generating ../include/mysqld_error.h, ../sql/share/english/errmsg.sys
./comp_err: error while loading shared libraries: libatomic.so.: cannot open shared object file: No such file or directory
make[]: *** [include/mysqld_error.h] Error
make[]: *** [extra/CMakeFiles/GenError.dir/all] Error
make[]: *** [libmysql/CMakeFiles/mysqlclient.dir/rule] Error
make: *** [mysqlclient] Error 

to resolve such error,

sudo cp /usr/local/lib64/libatomic.so.1.2. /usr/lib64
sudo cp /usr/local/lib64/libatomic.so.1.2. /usr/lib
sudo ldconfig

4.2 build source code on ununtu

if you are builing the source code on ununtu, since the default gcc version is v7.0+, so the gcc version is compatible for building doris, but there's another question about it:

[  %] Building CXX object src/env/CMakeFiles/Env.dir/env_posix.cpp.o
In file included from workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/env/env_posix.cpp:::
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/env/env_posix.cpp: In function ‘doris::Status doris::do_readv_at(int, const string&, uint64_t, const doris::Slice*, size_t’:
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/env/env_posix.cpp::: error: ‘preadv’ was not declared in this scope
         RETRY_ON_EINTR(r, preadv(fd, iov + completed_iov, iov_count, cur_offset));
                           ^
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/gutil/macros.h::: note: in definition of macro ‘RETRY_ON_EINTR’
   (err) = (expr); \
            ^~~~
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/env/env_posix.cpp::: note: suggested alternative: ‘pread’
         RETRY_ON_EINTR(r, preadv(fd, iov + completed_iov, iov_count, cur_offset));
                           ^
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/gutil/macros.h::: note: in definition of macro ‘RETRY_ON_EINTR’
   (err) = (expr); \
            ^~~~
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/env/env_posix.cpp: In function ‘doris::Status doris::do_writev_at(int, const string&, uint64_t, const doris::Slice*, size_t, size_t*)’:
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/env/env_posix.cpp::: error: ‘pwritev’ was not declared in this scope
         RETRY_ON_EINTR(w, pwritev(fd, iov + completed_iov, iov_count, cur_offset));
                           ^
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/gutil/macros.h::: note: in definition of macro ‘RETRY_ON_EINTR’
   (err) = (expr); \
            ^~~~
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/env/env_posix.cpp::: note: suggested alternative: ‘pwrite’
         RETRY_ON_EINTR(w, pwritev(fd, iov + completed_iov, iov_count, cur_offset));
                           ^
workspace/open-source/incubator-doris-DORIS-0.11.-alpha/be/src/gutil/macros.h::: note: in definition of macro ‘RETRY_ON_EINTR’
   (err) = (expr); \
            ^~~~
src/env/CMakeFiles/Env.dir/build.make:: recipe for target 'src/env/CMakeFiles/Env.dir/env_posix.cpp.o' failed
make[]: *** [src/env/CMakeFiles/Env.dir/env_posix.cpp.o] Error
CMakeFiles/Makefile2:: recipe for target 'src/env/CMakeFiles/Env.dir/all' failed
make[]: *** [src/env/CMakeFiles/Env.dir/all] Error
make[]: *** Waiting for unfinished jobs....
[  %] Building CXX object src/exprs/CMakeFiles/Exprs.dir/conditional_functions.cpp.o
[  %] Building CXX object src/exprs/CMakeFiles/Exprs.dir/conditional_functions_ir.cpp.o

this is caused by missing reference for  preadv,  the solution is to include header file for peadv, detail action as follows:

a) open file   incubator-doris-DORIS-0.11.5-alpha/be/src/env/env_posix.cpp

b)  include <sys/uio.h> at the header reference section, then rebuild the source ,everything would go smoothly on ubuntu.

For such issue, I have commite a pull request to the doris team, it must has been merged into the master branch.