【原创】大数据基础之Ambari(2)通过Ambari部署ElasticSearch(ELK)
ambari2.7.3(hdp3.1) 安装 elasticsearch6.3.2
ambari的hdp中原生不支持elasticsearch安装,下面介绍如何通过mpack方式使ambari支持elasticsearch安装:
一 安装Service
1 下载
Mpack include version 6.3.2 of ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat
# wget https://community.hortonworks.com/storage/attachments/87416-elasticsearch-mpack-2600-9.tar.gz
2 安装mpack
# ambari-server install-mpack --mpack=/path/to/87416-elasticsearch-mpack-2600-9.tar.gz --verbose
3 重启ambari-server
# ambari-server restart
访问ambari页面发现services中并没有elasticsearch,检查mpack.json发现问题:
/var/lib/ambari-server/resources/mpacks/elasticsearch-ambari.mpack-6.3.2/mpack.json
{
"service_name" : "ELASTICSEARCH",
"service_version" : "6.3.2",
"applicable_stacks" : [
{
"stack_name" : "HDP",
"stack_version" : "2.3"
},
{
"stack_name" : "HDP",
"stack_version" : "2.4"
},
{
"stack_name" : "HDP",
"stack_version" : "2.5"
},
{
"stack_name" : "HDP",
"stack_version" : "2.6"
}
]
}
applicable_stacks表明service只应用到hdp2.3-2.6版本,没有3.1,手工加上;
mpack详细结构详见:https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs
要注意直接修改mpack.json然后重启ambari-server不管用,需要重新install mpack:
1 卸载
ambari-server uninstall-mpack --mpack-name=elasticsearch-ambari.mpack
2 修改mpack.json并重新打包
elasticsearch_mpack-2.6.0.0-9.tar.gz
3 安装mpack(同上)
4 重启ambari-server(同上)
这时就可以在页面上看到ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat
二 部署ElasticSearch
下面开始部署ElasticSearch,发现报错:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>
BeforeAnyHook().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook
setup_users()
File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users
groups = params.user_to_groups_dict[user],
KeyError: u'elasticsearch'
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-303.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-303.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']
跟进报错的代码:
/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py
35 should_create_users_and_groups = False
36 if params.host_sys_prepped:
37 should_create_users_and_groups = not params.sysprep_skip_create_users_and_groups
38 else:
39 should_create_users_and_groups = not params.ignore_groupsusers_create
40
41 if should_create_users_and_groups:
42 for group in params.group_list:
43 Group(group,
44 )
45
46 for user in params.user_list:
47 User(user,
48 uid = get_uid(user) if params.override_uid == "true" else None,
49 gid = params.user_to_gid_dict[user],
50 groups = params.user_to_groups_dict[user],
51 fetch_nonlocal_groups = params.fetch_nonlocal_groups,
52 )
这时需要将配置ignore_groupsusers_create打开,查看和修改配置有两种方式:
1 命令行
# cd /var/lib/ambari-server/resources/scripts
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create
"ignore_groupsusers_create": "false",
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true
详细参数详见
# python configs.py --help
2 rest api
curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env"
从返回的items中找到tag,然后再请求(增加tag参数):
curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env&tag=$tag"
返回结果中包含所有的property
"properties" : {
"agent_mounts_ignore_list" : "",
"alerts_repeat_tolerance" : "1",
"enable_external_ranger" : "false",
"fetch_nonlocal_groups" : "true",
"hide_yarn_memory_widget" : "false",
"ignore_bad_mounts" : "false",
"ignore_groupsusers_create" : "false",
修改需要先将上面的配置保存到一个文件,比如/tmp/configurations.tmp,然后修改,然后再调用接口:
curl --user admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d @/tmp/configurations.tmp http://ambari.server:8080/api/v1/clusters/$cluster_name
然后再部署即可;
三 启动ElasticSearch
启动elasticsearch,报错:
1)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'java_home' was not found in configurations dictionary!
需要返回ambari的elasticsearch config页面,将java_home修改为实际的java home路径:
# JAVA_HOME must be provided here for OS that use systemd service launch
JAVA_HOME={{java64_home}}
2)
File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2", line 85, in top-level template code
gateway.expected_data_nodes: {{ expected_data_nodes }}
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'hostname' was not found in configurations dictionary!
查看报错的文件
/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2
/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.master.yaml.j2
发现报错的这行,是因为hostname没有赋值导致,直接去掉这行
node.name: {{ hostname }}
在ambari-server端对应的模板文件为:
# grep hostname /var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch*
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.master.yaml.j2:node.name: {{ hostname }}
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.master.yaml.j2.5.6.2: name: {{hostname}}
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2:node.name: {{ hostname }}
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2.5.6.2: name: {{hostname}}
然后可以启动成功:
参考:
https://community.hortonworks.com/articles/215078/how-to-install-elk-stack-632-in-ambari.html
https://community.hortonworks.com/questions/212871/add-custom-service-via-ambari-error.html
https://community.hortonworks.com/articles/63557/how-to-edit-the-cluster-envxml-entries-using-ambar.html
【原创】大数据基础之Ambari(2)通过Ambari部署ElasticSearch(ELK)的更多相关文章
- 大数据基础环境--jdk1.8环境安装部署
1.环境说明 1.1.机器配置说明 本次集群环境为三台linux系统机器,具体信息如下: 主机名称 IP地址 操作系统 hadoop1 10.0.0.20 CentOS Linux release 7 ...
- 【原创】大数据基础之Zookeeper(2)源代码解析
核心枚举 public enum ServerState { LOOKING, FOLLOWING, LEADING, OBSERVING; } zookeeper服务器状态:刚启动LOOKING,f ...
- 【原创】大数据基础之Ambari(1)简介、编译安装、使用
官方:http://ambari.apache.org/ The Apache Ambari project is aimed at making Hadoop management simpler ...
- 【原创】大数据基础之Impala(1)简介、安装、使用
impala2.12 官方:http://impala.apache.org/ 一 简介 Apache Impala is the open source, native analytic datab ...
- 【原创】大数据基础之词频统计Word Count
对文件进行词频统计,是一个大数据领域的hello word级别的应用,来看下实现有多简单: 1 Linux单机处理 egrep -o "\b[[:alpha:]]+\b" test ...
- 【原创】大数据基础之Benchmark(2)TPC-DS
tpc 官方:http://www.tpc.org/ 一 简介 The TPC is a non-profit corporation founded to define transaction pr ...
- 大数据基础知识问答----hadoop篇
handoop相关知识点 1.Hadoop是什么? Hadoop是一个由Apache基金会所开发的分布式系统基础架构.用户可以在不了解分布式底层细节的情况下,开发分布式程序.充分利用集群的威力进行高速 ...
- 大数据基础知识:分布式计算、服务器集群[zz]
大数据中的数据量非常巨大,达到了PB级别.而且这庞大的数据之中,不仅仅包括结构化数据(如数字.符号等数据),还包括非结构化数据(如文本.图像.声音.视频等数据).这使得大数据的存储,管理和处理很难利用 ...
- 大数据基础知识问答----spark篇,大数据生态圈
Spark相关知识点 1.Spark基础知识 1.Spark是什么? UCBerkeley AMPlab所开源的类HadoopMapReduce的通用的并行计算框架 dfsSpark基于mapredu ...
随机推荐
- 磁盘IO的性能指标 阻塞与非阻塞、同步与异步 I/O模型
磁盘IO的性能指标 - 蝈蝈俊 - 博客园https://www.cnblogs.com/ghj1976/p/5611648.html 阻塞与非阻塞.同步与异步 I/O模型 - 蝈蝈俊.net - C ...
- java.lang.NullPointerException 错误原因
[http-nio-8081-exec-1] ERROR o.a.c.c.C.[.[localhost].[/].[dispatcherServlet] - Servlet.service() for ...
- Package与Activity简介
Package Package 包.只是在我们的app中这个Package是唯一的,就像你身份证号码一样.在我们做app自动化时,我们就需要知道他的Package,我们知道了Package那么也就知道 ...
- MIUI9 解锁并刷入TWRP后,删除解锁密码
如果因为某种原因导致解锁密码失效(比如刷了其他ROM),还原备份回来之后,解锁密码失效了. 那么可以进入TWRP,然后通过 adb shell 进入\data\system\文件夹 用rm命令删除g ...
- git安装和GitHub使用
一.git安装 1.git下载 https://github.com/git-for-windows/git/releases/tag/v2.17.1.windows.2 在如上地址下载git,并安装 ...
- 三、Java多人博客系统-技术架构
多人博客系统1.0版本,架构和技术还是很简单和很传统的. 1.技术 前端:jsp.html.css.javascript.jquery.easyui.echarts 后端:spring mvc.Hib ...
- Python future使用
Python的每个新版本都会增加一些新的功能,或者对原来的功能作一些改动.有些改动是不兼容旧版本的,也就是在当前版本运行正常的代码,到下一个版本运行就可能不正常了. 从Python 2.7到Pytho ...
- [BZOJ 4516] [SDOI 2016] 生成魔咒
Description 魔咒串由许多魔咒字符组成,魔咒字符可以用数字表示.例如可以将魔咒字符 1.2 拼凑起来形成一个魔咒串 [1,2]. 一个魔咒串 S 的非空字串被称为魔咒串 S 的生成魔咒. 例 ...
- 【ARC101F】Robots and Exits 树状数组
题目大意 有 \(n\) 个机器人和 \(m\) 个出口. 这 \(n\) 个机器人的初始位置是 \(a_1,a_2,\ldots,a_n\),这 \(m\) 个出口的位置是 \(b_1,b_2,\l ...
- Entity Framework入门教程(17)---记录和拦截数据库命令
记录和拦截数据库命令 这一节介绍EF6怎么记录和拦截发送给数据库的查询和操作命令. 1.记录EF发送给数据库命令(DbContext.Database.Log) 以前给了查看EF发送给数据库的命令我们 ...