We are happy to finally announce the first release of mlrMBO on cran after a quite long development time. For the theoretical background and a nearly complete overview of mlrMBOs capabilities you can check our paper onmlrMBO that we presubmitted to arxiv.

The key features of mlrMBO are:

  • Global optimization of expensive Black-Box functions.
  • Mulit-Criteria Optimization.
  • Parallelization through multi-point proposals.
  • Support for optimization over categorical variables using random forests as a surrogate.

For examples covering different scenarios we have Vignettes that are also available as an online documentation. For mlr users mlrMBO is especially interesting for hyperparameter optimization.

mlrMBO for mlr hyperparameter tuning was already used in an earlier blog post. Nonetheless we want to provide a small toy example to demonstrate the work flow of mlrMBO in this post.

Example

First, we define an objective function that we are going to minimize:

set.seed(1)
library(mlrMBO)
fun = makeSingleObjectiveFunction(
name = "SineMixture",
fn = function(x) sin(x[1])*cos(x[2])/2 + 0.04 * sum(x^2),
par.set = makeNumericParamSet(id = "x", len = 2, lower = -5, upper = 5)
)

To define the objective function we use makeSingleObjectiveFunction from the neat package smoof, which gives us the benefit amongst others to be able to directly visualize the function. If you happen to be in need of functions to optimize and benchmark your optimization algorithm I recommend you to have a look at the package!

library(plot3D)
plot3D(fun, contour = TRUE, lightning = TRUE)

Let’s start with the configuration of the optimization:

# In this simple example we construct the control object with the defaults:
ctrl = makeMBOControl()
# For this numeric optimization we are going to use the Expected Improvement as infill criterion:
ctrl = setMBOControlInfill(ctrl, crit = crit.ei)
# We will allow for exactly 25 evaluations of the objective function:
ctrl = setMBOControlTermination(ctrl, max.evals = 25L)

The optimization has to so start with an initial design. mlrMBO can automatically create one but here we are going to use a randomly sampled LHS design of our own:

library(ggplot2)
des = generateDesign(n = 8L, par.set = getParamSet(fun), fun = lhs::randomLHS)
autoplot(fun, render.levels = TRUE) + geom_point(data = des)
## Warning: Ignoring unknown aesthetics: fill

The points demonstrate how the initial design already covers the search space but is missing the area of the global minimum. Before we can start the Bayesian optimization we have to set the surrogate learner to Kriging. Therefore we use an mlr regression learner. In fact, with mlrMBO you can use any regression learner integrated inmlr as a surrogate allowing for many special optimization applications.

sur.lrn = makeLearner("regr.km", predict.type = "se", config = list(show.learner.output = FALSE))

Note: mlrMBO can automatically determine a good surrogate learner based on the search space defined for the objective function. For a purely numeric domain it would have chosen Kriging as well with some slight modifications to make it a bit more stable against numerical problems that can occur during optimization.

Finally, we can start the optimization run:

res = mbo(fun = fun, design = des, learner = sur.lrn, control = ctrl, show.info = TRUE)
## Computing y column(s) for design. Not provided.
## [mbo] 0: x=-0.0101,-4.52 : y = 0.817 : 0.0 secs : initdesign
## [mbo] 0: x=-4.52,-2.48 : y = 0.677 : 0.0 secs : initdesign
## [mbo] 0: x=-2.78,-3.27 : y = 0.913 : 0.0 secs : initdesign
## [mbo] 0: x=4.92,1.09 : y = 0.787 : 0.0 secs : initdesign
## [mbo] 0: x=2.77,2.93 : y = 0.469 : 0.0 secs : initdesign
## [mbo] 0: x=0.815,-0.647 : y = 0.333 : 0.0 secs : initdesign
## [mbo] 0: x=-2.34,4.5 : y = 1.11 : 0.0 secs : initdesign
## [mbo] 0: x=1.58,1.87 : y = 0.0939 : 0.0 secs : initdesign
## [mbo] 1: x=1.48,5 : y = 1.23 : 0.0 secs : infill_ei
## [mbo] 2: x=-3.77,2.2 : y = 0.589 : 0.0 secs : infill_ei
## [mbo] 3: x=0.429,1.49 : y = 0.113 : 0.0 secs : infill_ei
## [mbo] 4: x=0.776,1.98 : y = 0.0413 : 0.0 secs : infill_ei
## [mbo] 5: x=0.126,1.93 : y = 0.127 : 0.0 secs : infill_ei
## [mbo] 6: x=1.01,2.15 : y = -0.00662 : 0.0 secs : infill_ei
## [mbo] 7: x=0.963,2.36 : y = -0.0317 : 0.0 secs : infill_ei
## [mbo] 8: x=0.922,0.539 : y = 0.388 : 0.0 secs : infill_ei
## [mbo] 9: x=-2.7,-0.524 : y = 0.119 : 0.0 secs : infill_ei
## [mbo] 10: x=-5,-0.253 : y = 1.47 : 0.0 secs : infill_ei
## [mbo] 11: x=-1.46,-0.613 : y = -0.306 : 0.0 secs : infill_ei
## [mbo] 12: x=-1.39,-1.1 : y = -0.098 : 0.0 secs : infill_ei
## [mbo] 13: x=-1.29,-0.228 : y = -0.4 : 0.0 secs : infill_ei
## [mbo] 14: x=-1.57,0.256 : y = -0.382 : 0.0 secs : infill_ei
## [mbo] 15: x=-1.43,-0.0423 : y = -0.413 : 0.0 secs : infill_ei
## [mbo] 16: x=-1.27,0.0745 : y = -0.412 : 0.0 secs : infill_ei
## [mbo] 17: x=5,-3.84 : y = 1.96 : 0.0 secs : infill_ei
res$x
## $x
## [1] -1.42836803 -0.04234841
res$y
## [1] -0.4128122

We can see that we have found the global optimum of y=−0.414964y=−0.414964 at x=(−1.35265,0)x=(−1.35265,0) quite sufficiently. Let’s have a look at the points mlrMBO evaluated. Therefore we can use the OptPath which stores all information about all evaluations during the optimization run:

opdf = as.data.frame(res$opt.path)
autoplot(fun, render.levels = TRUE, render.contours = FALSE) + geom_text(data = opdf, aes(label = dob))

It is interesting to see, that for this run the algorithm first went to the local minimum on the top right in the 6th and 7th iteration but later, thanks to the explorative character of the Expected Improvement, found the real global minimum.

Comparison

That is all good, but how do other optimization strategies perform?

Grid Search

Grid search is seldom a good idea. But especially for hyperparameter tuning it is still used. Probably because it kind of gives you the feeling that you know what is going on and have not left out any important area of the search space. In reality the grid is usually so sparse that it leaves important areas untouched as you can see in this example:

grid.des = generateGridDesign(par.set = getParamSet(fun), resolution = 5)
grid.des$y = apply(grid.des, 1, fun)
grid.des[which.min(grid.des$y),]
##      x1 x2           y
## 12 -2.5 0 -0.04923607
autoplot(fun, render.levels = TRUE, render.contours = FALSE) + geom_point(data = grid.des)

It is no surprise, that the grid search could not cover the search space well enough and we only reach a bad result.

What about a simple random search?

random.des = generateRandomDesign(par.set = getParamSet(fun), n = 25L)
random.des$y = apply(random.des, 1, fun)
random.des[which.min(random.des$y),]
##           x1         x2          y
## 20 -1.784371 -0.9802194 -0.1063019
autoplot(fun, render.levels = TRUE, render.contours = FALSE) + geom_point(data = random.des)

With the random search you could always be lucky but in average the optimum is not reached if smarter optimization strategies work well.

A fair comarison

… for stochastic optimization algorithms can only be achieved by repeating the runs. mlrMBO is stochastic as the initial design is generated randomly and the fit of the Kriging surrogate is also not deterministic. Furthermore we should include other optimization strategies like a genetic algorithm and direct competitors like rBayesOpt. An extensive benchmark is available in our mlrMBO paper. The examples here are just meant to demonstrate the package.

Engage

If you want to contribute to mlrMBO we ware always open to suggestions and pull requests on github. You are also invited to fork the repository and build and extend your own optimizer based on our toolbox.

转自:https://mlr-org.github.io/First_release_of_mlrMBO_the_toolbox_for_Bayesian_Black_Box_Optimization/

First release of mlrMBO - the toolbox for (Bayesian) Black-Box Optimization的更多相关文章

  1. matlab toolboxes 大全

    MATLAB Toolboxes top (Top) Audio - Astronomy - BiomedicalInformatics - Chemometrics  - Chaos - Chemi ...

  2. plain framework 商业版 开发总结2 项目管理器

    任何事情都有三个阶段,分析.制作.质检的过程.在程序中就分为设计.编码.调试(测试)三个阶段,其中设计最为重要,设计的不好会导致编码和调试重复,甚至最后又回到了设计的过程.为了不会重复返工,所以设计的 ...

  3. 【转】Application.mk 文件语法规范

    原文网址:http://blog.sina.com.cn/s/blog_4c451e0e0100s6q4.html Application.mk file syntax specification A ...

  4. Android NDK开发指南---Application.mk文件和android.mk文件

    https://android.googlesource.com/platform/development/+/donut-release/ndk/docs/OVERVIEW.TXT https:// ...

  5. mysql 源码编绎修改 FLAGS,调试MYSQL

    http://dev.mysql.com/doc/refman/5.7/en/source-configuration-options.html#option_cmake_cmake_c_flags ...

  6. Windows 10 安装 Docker for Windows

    Docker for Windows是Docker社区版(CE)应用程序. Docker for Windows安装包包括在Windows系统上运行Docker所需的一切. 本主题介绍了预安装注意事项 ...

  7. Docker for Windows 使用 VMware WorkStation

    一.前言 Docker for Windows 不同于 Docker Toolbox.Docker for Windows 对系统的要求至少为Windows 10专业版,因为它需要Hyper-V的支持 ...

  8. [zz] MATLAB工具箱介绍

    http://blog.sina.com.cn/s/blog_57235cc701012kfb.html Toolbox工具箱 序号 工具箱 备注   数学.统计与优化   1 Symbolic Ma ...

  9. Sphinx 2.2.11-release reference manual

    1. Introduction 1.1. About 1.2. Sphinx features 1.3. Where to get Sphinx 1.4. License 1.5. Credits 1 ...

随机推荐

  1. Python爬虫 URLError异常处理

    1.URLError 首先解释下URLError可能产生的原因: 网络无连接,即本机无法上网 连接不到特定的服务器 服务器不存在 在代码中,我们需要用try-except语句来包围并捕获相应的异常.下 ...

  2. ios UICollectionView简单说明

    原谅我记不住写下来好了 UICollectionViewFlowLayout 流式自动布局 继承于UICollectionViewLayout 初始化:[[UICollectionViewFlowLa ...

  3. input标签禁止输入,隐藏光标

    <INPUT class=""  value="" name= "" readonly onfocus="this.blur ...

  4. TDD 中关于mock一些理解

    最近在写代码的UT时case注意到: 在写某个类的test suit时,如果这个类既有组合(Composition),又有聚合关系(Aggregation). 组合关系(Composition):部分 ...

  5. Android性能优化——之防止内存泄露

    又是好久没有写博客了,一直都比较忙,最近终于有时间沉淀和整理一下最近学到和解决的一些问题. 最近进行技术支持的时候,遇到了几个崩溃的问题,都是OOM异常,一般OOM异常给人的感觉应该是加载大图片造成的 ...

  6. html常用的知识点以及混合框架

    html中:   <hr/> 在页面中创建水平线 例如:   <p> p标签是定义段落   > alt 作为可预备可替换信息,在无法加载图片时显示文字信息   定义htm ...

  7. jquery中is()函数

    is(expr)函数判断当前Jquery对象所匹配的元素是否存在.只要其中一种符合,就返回 true,否则返回 false. 如果 expr是个字符串,既视为Jquery的选择器,用于表示选择的元素. ...

  8. ZJOI2017 Round#2 滚粗记

    在杭州的火车站的KFC餐厅里,独自一人,闲来无事,便写写这篇博客.刚刚的一个礼拜,经历了余姚的省选和杭州的数学集训,感觉有些浪,学校里现在还在上新课,我已经落下一个礼拜的文化课了,回去估计补死:最重要 ...

  9. glassfish PWC6351: In TLD scanning 系统找不到指定的文件问题解决

    [2017-04-25T21:26:09.391+0800] [glassfish 4.1] [WARNING] [] [org.apache.jasper.runtime.TldScanner] [ ...

  10. JavaScript中return的用法和this的用法详解

    JavaScript中return的用法详解 最近,跟身边学前端的朋友了解,有很多人对this和函数中的return的用法和意思理解的比较模糊,这里写一篇博客跟大家一起探讨一下return和this的 ...