Source:http://n8.io/converting-a-c-library-to-gyp/

Converting a C library to gyp

Published: 2012.10.02

gyp, short for "Generate Your Projects", is a build system written by Google that was specifically designed to build their Chromium web browser.gyp makes it easy to develop such a large project because the developers can create and test each piece individually by defining self-contained "targets". Then at Release-time, gyp makes it easy to stitch the "targets" together into a single statically linked executable. Ryan Dahl once called it "the module-system of C".

Node.js recently adopted gyp as its build system because V8 was already using it (through Chromium), and because of its explicit Windows support. node-gyp was created to help the community create node.js native addons by writing simple gyp files. This article is targeted towards Node.js native module authors, so hopefully you are somewhat familiar will all of this already.

gyp-ify all the things!

One thing that I want to see more of in the node community from native module authors is the additional work done to convert the dependencies of the native module to use gyp to build, and bundle them with the native module. This allows the library to be statically linked to your native module, rather than dynamically linked. There are pros and cons to both kinds of linking, but node.js has gone with static linking, mostly because of portability and to ease Windows support.

Now converting the build system of some odd C library that you haven't written can be a daunting task, so that's what this article hopes to address. I'll explain my method of converting a C library originally using autotools to build using gyp, and hopefully you will do the same (or improve on my method!) in order to make your native module more portable. Your users will thank you for the easier installation process (i.e. not having toapt-get install some external libraries).

libmp3lame, I choose you!

Our test subject for this article will be libmp3lame, a library that does MP3 encoding and decoding. This is a nice, medium complexity library that will be a good demonstration of my method.

Baseline gyp file

We're gonna start with a baseline gyp file to use. Let's call it libmp3lame.gyp in this case:

{
'target_defaults': {
'default_configuration': 'Debug',
'configurations': {
'Debug': {
'defines': [ 'DEBUG', '_DEBUG' ],
'msvs_settings': {
'VCCLCompilerTool': {
'RuntimeLibrary': 1, # static debug
},
},
},
'Release': {
'defines': [ 'NDEBUG' ],
'msvs_settings': {
'VCCLCompilerTool': {
'RuntimeLibrary': 0, # static release
},
},
}
},
'msvs_settings': {
'VCLinkerTool': {
'GenerateDebugInformation': 'true',
},
},
}, 'targets': [
]
}

The msvs_settings configuration stuff at the top ensures that we end up with statically built libraries and executables on Windows. This boilerplate isn't strictly necessary, but it matches up with some defaults set in node.js' common.gypi file, and I like to stay consistent. The emptytargets array will end up with the definitions to build libmp3lame, as well as a simple test program.

Adding the platform-specific files

Most C libraries have a ./configure step, and usually that step will create some kind of "config.h" file of some sort, which has appropriate defines for your current platform. So our strategy for converting to gyp is to pre-generate these "config.h" files on all your supported platforms and architectures, and save them for use when building with gyp.

So for example, the first platform + architecture combo I will get working is Mac OS X 64-bit. Therefore, the resulting "config.h" file will be placed atconfig/mac/x64/config.h. In the end, we want to support if possible this baseline of platforms and architectures:

OS Arch config.h path
Linux ARM config/linux/arm
Linux 32-bit config/linux/ia32
Linux 64-bit config/linux/x64
Mac OS X 32-bit config/mac/ia32
Mac OS X 64-bit config/mac/x64
Solaris 32-bit config/solaris/ia32
Solaris 64-bit config/solaris/x64
Windows 32-bit config/win/ia32
Windows 64-bit config/win/x64

Some of the combinations might not be supported by the library you are working with, but (except for linux-arm) this list matches up with the releases of the official node.js binary tarballs, and what node supports, you should try to support too!

Running ./configure

With C libraries configured using autotools, I've been using a standard set of switches, and adding library-specific ones as necessary to try to get as close of a static build as gyp is going to make. The basic set of flags I use are:

$ ./configure --enable-static --disable-shared --with-pic

These flags are basically saying, build the static version of the library (the .a/.lib files), and don't bother with the shared version (the.so/.dylib/.dll files). Also, build "position independent code".

In some cases, that will be all that you need. But for libmp3lame, I'm also gonna need to specify a few additional flags to disable some unnecessary stuff like the lame program frontend.

$ ./configure --enable-static --disable-shared --with-pic \
--disable-rpath --disable-frontend --disable-gtktest

After running configure on my 64-bit Mac OS X machine, I'm going to copy the resulting config.h file to config/mac/x64/config.h. The"config.h" file could be called something else for your library, like "expat_config.h" for libexpat for example. In some cases there could be more than 1 ".h" file generated, like libffi had 3 per platform+arch combo. So you kinda just need to take a look around, visually inspect the configureoutput, and see what gets generated.

The first change we're going to make to the gyp file is to make sure that these platform-specific header files are included:

diff --git a/libmp3lame.gyp b/libmp3lame.gyp
index d30085c..cdc5b9b 100644
--- a/libmp3lame.gyp
+++ b/libmp3lame.gyp
@@ -1,4 +1,6 @@
{
+ 'variables': { 'target_arch%': 'ia32' },
+
'target_defaults': {
'default_configuration': 'Debug',
'configurations': {
@@ -24,6 +26,10 @@
'GenerateDebugInformation': 'true',
},
},
+ 'include_dirs': [
+ # platform and arch-specific headers
+ 'config/<(OS)/<(target_arch)'
+ ],
}, 'targets': [

So you essentially need to repeat these "configure & copy to config" steps for all of the platforms + arch combinations listed in the table above. I usually do things a little out of order though, and move on to making the library build (the next section) before worrying about the other platforms...

Defining the targets

So usually the next step when building these libraries is to run the make command. Sometimes you will need something additonal like overwriting a specific variable, but in this case we'll just invoke it straight up.

$ make

The key to defining proper targets is, again, visually inspecting the make output and seeing exactly which files are compiled, and defines are added, and which flags are used. You can also get part of the picture by looking at the Makefile.am files in the library.

So based off the make output in the case of libmp3lame, there's going to be 3 total "targets" that will be defined:

  • mpglib - libmp3lame's decoder (an old version of libmpg123 apparently)
  • liblamevectorroutines - some internal routines used by libmp3lame
  • libmp3lame the API frontend for libmp3lame

Each of these "targets" are like "modules" in node.js, they're their own independent functional component that make up part of the library.

Ok, so let's define those targets. This is a lot of new gyp code here, so be sure to study it until it makes sense to you:

diff --git a/libmp3lame.gyp b/libmp3lame.gyp
index cdc5b9b..f30319b 100644
--- a/libmp3lame.gyp
+++ b/libmp3lame.gyp
@@ -27,11 +27,88 @@
},
},
'include_dirs': [
+ 'include',
+ 'mpglib',
+ 'libmp3lame',
+ 'libmp3lame/vector',
# platform and arch-specific headers
'config/<(OS)/<(target_arch)'
],
+ 'defines': [
+ 'PIC',
+ 'HAVE_CONFIG_H'
+ ],
}, 'targets': [
+ # mpglib
+ {
+ 'target_name': 'mpgdecoder',
+ 'product_prefix': 'lib',
+ 'type': 'static_library',
+ 'sources': [
+ 'mpglib/common.c',
+ 'mpglib/dct64_i386.c',
+ 'mpglib/decode_i386.c',
+ 'mpglib/interface.c',
+ 'mpglib/layer1.c',
+ 'mpglib/layer2.c',
+ 'mpglib/layer3.c',
+ 'mpglib/tabinit.c',
+ ],
+ },
+
+ # liblamevectorroutines
+ {
+ 'target_name': 'lamevectorroutines',
+ 'product_prefix': 'lib',
+ 'type': 'static_library',
+ 'sources': [
+ 'libmp3lame/vector/xmm_quantize_sub.c'
+ ],
+ },
+
+ # libmp3lame
+ {
+ 'target_name': 'mp3lame',
+ 'product_prefix': 'lib',
+ 'type': 'static_library',
+ 'sources': [
+ 'libmp3lame/VbrTag.c',
+ 'libmp3lame/bitstream.c',
+ 'libmp3lame/encoder.c',
+ 'libmp3lame/fft.c',
+ 'libmp3lame/gain_analysis.c',
+ 'libmp3lame/id3tag.c',
+ 'libmp3lame/lame.c',
+ 'libmp3lame/newmdct.c',
+ 'libmp3lame/presets.c',
+ 'libmp3lame/psymodel.c',
+ 'libmp3lame/quantize.c',
+ 'libmp3lame/quantize_pvt.c',
+ 'libmp3lame/reservoir.c',
+ 'libmp3lame/set_get.c',
+ 'libmp3lame/tables.c',
+ 'libmp3lame/takehiro.c',
+ 'libmp3lame/util.c',
+ 'libmp3lame/vbrquantize.c',
+ 'libmp3lame/version.c',
+ 'libmp3lame/mpglib_interface.c',
+ ],
+ 'dependencies': [
+ 'lamevectorroutines',
+ 'mpgdecoder',
+ ],
+ 'direct_dependent_settings': {
+ 'include_dirs': [
+ 'include',
+ 'mpglib',
+ 'libmp3lame',
+ 'libmp3lame/vector',
+ # platform and arch-specific headers
+ 'config/<(OS)/<(target_arch)'
+ ],
+ },
+ },
]
}

One important thing to note is how we make the "mp3lame" target depend on "lamevectorroutines" and "mpgdecoder". This allows us to only have to specify "mp3lame" as a dependency in the node module's binding.gyp file. Also note how the direct_dependent_settings are defined within that target as well, for the same reason.

Try it out

At this point, we should be able to build libmp3lame, at least on 64-bit OS X. I like to add a very quick little test program of some sort to the gyp file, to make sure that things get built and statically linked like we expect, and that it works at all.

We have to add an executable target to the gyp file:

diff --git a/libmp3lame.gyp b/libmp3lame.gyp
index f30319b..de2984d 100644
--- a/libmp3lame.gyp
+++ b/libmp3lame.gyp
@@ -110,5 +110,13 @@
],
},
},
+
+ # test program that prints the version number
+ {
+ 'target_name': 'test',
+ 'type': 'executable',
+ 'dependencies': [ 'mp3lame' ],
+ 'sources': [ 'test.c' ]
+ },
]
}
diff --git a/test.c b/test.c
new file mode 100644
index 0000000..1fb3dbf
--- /dev/null
+++ b/test.c
@@ -0,0 +1,7 @@
+#include <stdio.h>
+#include "lame.h"
+
+int main () {
+ printf("get_lame_version(): %s\n", get_lame_version());
+ return 0;
+}

This super basic test program simply prints out the result of the get_lame_version() function. Let's download gyp, configure our project, and run this test program:

$ svn co http://gyp.googlecode.com/svn/trunk gyp  # download gyp
$ ./gyp/gyp -f make --depth=. libmp3lame.gyp # run gyp for the `make` target
$ make V=1 # verbose build
$ ./out/Debug/test # run the `test` program
get_lame_version(): 3.99.5

Success!

Windows support

So after generating all the config files for the other platforms (mac, linux, solaris), the last one is Windows (win). Windows usually takes some special dedication to get working. At the very least, a Google search asking "how to build library xxxx on Windows" should yield something.

If there are any major differences, you can introduce a conditions block in your gyp file targeting Windows like so:

    'conditions': [
['OS=="win"', {
# windows-specific gyp stuff goes here...
}],
],

The config files

In the case of libmp3lame, the developers have included a hand-crafted configMS.h file, which is the config.h file for Windows, both 32-bit and 64-bit. So all we have to do this time is copy this file to config/win/ia32 and config/win/x64. Piece of cake!

In some cases though you might have to actually run the ./configure script using version of bash compiled for Windows. This was the case forlibffi.

The source files and defines

It's ideal to actually run the build process for your library, ideally using MSVC, doing the method that the library authors suggest. That way you can inspect that build output and be able to determine any differences for the build on Windows.

For libmp3lame, there's a Makefile.MSVC file that is compatible with the nmake command on Windows. If we run it and inspect the output, we can see that there are a few different defines used on Windows, but all the same sources files.

Let's adjust the gyp file to reflect that:

diff --git a/libmp3lame.gyp b/libmp3lame.gyp
index de2984d..bb4b010 100644
--- a/libmp3lame.gyp
+++ b/libmp3lame.gyp
@@ -38,6 +38,16 @@
'PIC',
'HAVE_CONFIG_H'
],
+ 'conditions': [
+ ['OS=="win"', {
+ 'defines': [
+ 'TAKEHIRO_IEEE754_HACK',
+ 'FLOAT8=float',
+ 'REAL_IS_FLOAT=1',
+ 'BS_FORMAT=BINARY',
+ ]
+ }]
+ ],
}, 'targets': [

Sometimes additional sources files are used (or not used) so in that case you would add a sources block within that conditional.

At this point, Windows should work as well:

C:\Users\Nathan\Desktop\lame> svn co http://gyp.googlecode.com/svn/trunk gyp
C:\Users\Nathan\Desktop\lame> gyp\gyp.bat -f msvs -G msvs_version=2010 --depth=. libmp3lame.gyp
C:\Users\Nathan\Desktop\lame> msbuild libmp3lame.sln
C:\Users\Nathan\Desktop\lame> Debug\test.exe
get_lame_version(): 3.99.5

Excellent!

The finished gyp file

So we're done! At this point, we have our completed gyp file:

{
'variables': { 'target_arch%': 'ia32' }, 'target_defaults': {
'default_configuration': 'Debug',
'configurations': {
'Debug': {
'defines': [ 'DEBUG', '_DEBUG' ],
'msvs_settings': {
'VCCLCompilerTool': {
'RuntimeLibrary': 1, # static debug
},
},
},
'Release': {
'defines': [ 'NDEBUG' ],
'msvs_settings': {
'VCCLCompilerTool': {
'RuntimeLibrary': 0, # static release
},
},
}
},
'msvs_settings': {
'VCLinkerTool': {
'GenerateDebugInformation': 'true',
},
},
'include_dirs': [
'include',
'mpglib',
'libmp3lame',
'libmp3lame/vector',
# platform and arch-specific headers
'config/<(OS)/<(target_arch)'
],
'defines': [
'PIC',
'HAVE_CONFIG_H'
],
'conditions': [
['OS=="win"', {
'defines': [
'TAKEHIRO_IEEE754_HACK',
'FLOAT8=float',
'REAL_IS_FLOAT=1',
'BS_FORMAT=BINARY',
]
}]
],
}, 'targets': [
# mpglib
{
'target_name': 'mpgdecoder',
'product_prefix': 'lib',
'type': 'static_library',
'sources': [
'mpglib/common.c',
'mpglib/dct64_i386.c',
'mpglib/decode_i386.c',
'mpglib/interface.c',
'mpglib/layer1.c',
'mpglib/layer2.c',
'mpglib/layer3.c',
'mpglib/tabinit.c',
],
}, # liblamevectorroutines
{
'target_name': 'lamevectorroutines',
'product_prefix': 'lib',
'type': 'static_library',
'sources': [
'libmp3lame/vector/xmm_quantize_sub.c'
],
}, # libmp3lame
{
'target_name': 'mp3lame',
'product_prefix': 'lib',
'type': 'static_library',
'sources': [
'libmp3lame/VbrTag.c',
'libmp3lame/bitstream.c',
'libmp3lame/encoder.c',
'libmp3lame/fft.c',
'libmp3lame/gain_analysis.c',
'libmp3lame/id3tag.c',
'libmp3lame/lame.c',
'libmp3lame/newmdct.c',
'libmp3lame/presets.c',
'libmp3lame/psymodel.c',
'libmp3lame/quantize.c',
'libmp3lame/quantize_pvt.c',
'libmp3lame/reservoir.c',
'libmp3lame/set_get.c',
'libmp3lame/tables.c',
'libmp3lame/takehiro.c',
'libmp3lame/util.c',
'libmp3lame/vbrquantize.c',
'libmp3lame/version.c',
'libmp3lame/mpglib_interface.c',
],
'dependencies': [
'lamevectorroutines',
'mpgdecoder',
],
'direct_dependent_settings': {
'include_dirs': [
'include',
'mpglib',
'libmp3lame',
'libmp3lame/vector',
# platform and arch-specific headers
'config/<(OS)/<(target_arch)'
],
},
}, # test program that prints the version number
{
'target_name': 'test',
'type': 'executable',
'dependencies': [ 'mp3lame' ],
'sources': [ 'test.c' ]
},
]
}

Are we done yet?

No still not done... now we have to add it as a dependency to our node.js native addon.

Following the model of node.js core, I'm gonna place libmp3lame with our gyp stuff in deps/lame of the node-lame repo. After that I can add it as a dependency to the main binding.gyp file like so:

{
'targets': [
{
'target_name': 'bindings',
'sources': [
'src/bindings.cc',
'src/node_lame.cc'
],
'dependencies': [
'deps/lame/libmp3lame.gyp:mp3lame'
]
}
]
}

The binding.gyp file remains nice and simple. NOW we're done. Grab a pint!

[转]Converting a C library to gyp的更多相关文章

  1. Writing Reentrant and Thread-Safe Code(译:编写可重入和线程安全的代码)

    Writing Reentrant and Thread-Safe Code 编写可重入和线程安全的代码 (http://www.ualberta.ca/dept/chemeng/AIX-43/sha ...

  2. The Java library for converting Wikipedia wikitext notation to HTML

    https://code.google.com/p/gwtwiki/ The Java Wikipedia API (Bliki engine) is a parser library for con ...

  3. Gyp语法规则参考 & 工具的使用

    转自:http://www.cnblogs.com/nanvann/p/3913880.html 翻译自 https://code.google.com/p/gyp/wiki/GypLanguageS ...

  4. Calibre - book library management application

    http://calibre-ebook.com/ Library Management E-book conversion Syncing to e-book reader devices Down ...

  5. JSBinding / Plugins & Build Mozjswrap Library

    There are 2 libraries in Plugins: mozjs-31. This is SpiderMonkey library, built from https://github. ...

  6. .net Framework Class Library(FCL)

    from:http://msdn.microsoft.com/en-us/library/ms229335.aspx 我们平时在VS.net里引用的那些类库就是从这里来的 The .NET Frame ...

  7. Multi-Targeting and Porting a .NET Library to .NET Core 2.0

    Creating a new .NET Standard Project The first step for moving this library is to create a new .NET ...

  8. Error:Error converting bytecode to dex: Cause: com.android.dex.DexException: Multiple dex files define Lcom/lidroid/xutils/task/TaskHandler;

    Error:Error converting bytecode to dex: Cause: com.android.dex.DexException: Multiple dex files defi ...

  9. c# Use NAudio Library to Convert MP3 audio into WAV audio(将Mp3格式转换成Wav格式)

    Have you been in need of converting mp3 audios to wav audios?  If so, the skill in this article prov ...

随机推荐

  1. hdoj 1226 超级password 【隐图BFS】

    称号:hdoj 1226 超级password 分析:这题属于隐式图搜索,状态不是非常明显,须要自己建立. 事实上搜索说白了就是暴力. 这个题目就是,首先对给出的能够组成的全部的数依次枚举.长度从小到 ...

  2. MySQL存储引擎差异化实验

    本篇把MySQL最常用的存储引擎给大家做一个介绍,然后通过插入.修改和并发实验来了解和验证一下它们之间的一些差异. 一.MySQL存储引擎简介 存储引擎在MySQL结构里占据核心的位置,是上层抽象接口 ...

  3. ZOJ 2675 Little Mammoth(计算几何)

    圆形与矩形截面的面积 三角仍然可以做到这一点 代码: #include<stdio.h> #include<string.h> #include<stdlib.h> ...

  4. POJ 2777 Count Color(线段树+位运算)

    题目链接:http://poj.org/problem?id=2777 Description Chosen Problem Solving and Program design as an opti ...

  5. ASP.NET 5 Hello World

    ASP.NET 5系列教程 (二):Hello World   本篇文章内容比较基础,主要是向大家展示如何创建一个 ASP.NET 5 工程,主要包含内容如下: 创建ASP.NET 5 工程 添加 T ...

  6. Visual Studio Contact

    Visual Studio Contact(); 直播笔记   昨天微软干了几件了不起的事:.NET开发环境将开源.跨平台支持(Mac OS X和Linux).多设备支持(WP.Android和iOS ...

  7. 探索Android该Parcel机制上

    一.先从Serialize说起 我们都知道JAVA中的Serialize机制.译成串行化.序列化……,其作用是能将数据对象存入字节流其中,在须要时又一次生成对象.主要应用是利用外部存储设备保存对象状态 ...

  8. word-wrap同word-break的区别

    兼容 IE 和 FF 的换行 CSS 推荐样式 最好的方式是 以下是引用片段: word-wrap:break-word; overflow:hidden; 而不是 以下是引用片段: word-wra ...

  9. 读书笔记—CLR via C#章节1-2

    这本书这几年零零散散读过两三遍了,作为经典书籍,应该重复读反复读,既然我现在开始写博了,我也准备把以前觉得经典的好书重读细读一遍,并且将笔记整理到博客中,好记性不如烂笔头,同时也在写的过程中也可以加深 ...

  10. C注意,使用的语言字符串

    转载请注明出处! 在C语言没有具体的字符串数据类型,字符串的字符串常量和字符数组的形式. 实际上该字符串是零个或更多字符的字符串.并在整个位模式0NUL字节结束.因此,字符串所包括的字符内部不能出现N ...