ambari2.7.3(hdp3.1) 安装 elasticsearch6.3.2

ambari的hdp中原生不支持elasticsearch安装,下面介绍如何通过mpack方式使ambari支持elasticsearch安装:

一 安装Service

1 下载

Mpack include version 6.3.2 of ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat

# wget https://community.hortonworks.com/storage/attachments/87416-elasticsearch-mpack-2600-9.tar.gz

2 安装mpack

# ambari-server install-mpack --mpack=/path/to/87416-elasticsearch-mpack-2600-9.tar.gz --verbose

3 重启ambari-server

# ambari-server restart

访问ambari页面发现services中并没有elasticsearch,检查mpack.json发现问题:

/var/lib/ambari-server/resources/mpacks/elasticsearch-ambari.mpack-6.3.2/mpack.json

{

"service_name" : "ELASTICSEARCH",

"service_version" : "6.3.2",

"applicable_stacks" : [

{

"stack_name" : "HDP",

"stack_version" : "2.3"

},

{

"stack_name" : "HDP",

"stack_version" : "2.4"

},

{

"stack_name" : "HDP",

"stack_version" : "2.5"

},

{

"stack_name" : "HDP",

"stack_version" : "2.6"

}

]

}

applicable_stacks表明service只应用到hdp2.3-2.6版本,没有3.1,手工加上;

mpack详细结构详见:https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs

要注意直接修改mpack.json然后重启ambari-server不管用,需要重新install mpack:

1 卸载

ambari-server uninstall-mpack --mpack-name=elasticsearch-ambari.mpack

2 修改mpack.json并重新打包

elasticsearch_mpack-2.6.0.0-9.tar.gz

3 安装mpack(同上)

4 重启ambari-server(同上)

这时就可以在页面上看到ElasticSearch, Logstash, Kibana, FileBeat, and MetricBeat

二 部署ElasticSearch

下面开始部署ElasticSearch,发现报错:

Traceback (most recent call last):

File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>

BeforeAnyHook().execute()

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute

method(env)

File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook

setup_users()

File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users

groups = params.user_to_groups_dict[user],

KeyError: u'elasticsearch'

Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-303.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-303.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']

跟进报错的代码:

/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py

     35   should_create_users_and_groups = False
36 if params.host_sys_prepped:
37 should_create_users_and_groups = not params.sysprep_skip_create_users_and_groups
38 else:
39 should_create_users_and_groups = not params.ignore_groupsusers_create
40
41 if should_create_users_and_groups:
42 for group in params.group_list:
43 Group(group,
44 )
45
46 for user in params.user_list:
47 User(user,
48 uid = get_uid(user) if params.override_uid == "true" else None,
49 gid = params.user_to_gid_dict[user],
50 groups = params.user_to_groups_dict[user],
51 fetch_nonlocal_groups = params.fetch_nonlocal_groups,
52 )

这时需要将配置ignore_groupsusers_create打开,查看和修改配置有两种方式:

1 命令行

# cd /var/lib/ambari-server/resources/scripts
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create
"ignore_groupsusers_create": "false",
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true

详细参数详见

# python configs.py --help

2 rest api

curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env"

从返回的items中找到tag,然后再请求(增加tag参数):

curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env&tag=$tag"

返回结果中包含所有的property

"properties" : {
"agent_mounts_ignore_list" : "",
"alerts_repeat_tolerance" : "1",
"enable_external_ranger" : "false",
"fetch_nonlocal_groups" : "true",
"hide_yarn_memory_widget" : "false",
"ignore_bad_mounts" : "false",
"ignore_groupsusers_create" : "false",

修改需要先将上面的配置保存到一个文件,比如/tmp/configurations.tmp,然后修改,然后再调用接口:

curl --user admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d @/tmp/configurations.tmp http://ambari.server:8080/api/v1/clusters/$cluster_name

然后再部署即可;

三 启动ElasticSearch

启动elasticsearch,报错:

1)

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'java_home' was not found in configurations dictionary!

需要返回ambari的elasticsearch config页面,将java_home修改为实际的java home路径:

# JAVA_HOME must be provided here for OS that use systemd service launch
JAVA_HOME={{java64_home}}

2)

File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2", line 85, in top-level template code
gateway.expected_data_nodes: {{ expected_data_nodes }}
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'hostname' was not found in configurations dictionary!

查看报错的文件

/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2
/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.master.yaml.j2

发现报错的这行,是因为hostname没有赋值导致,直接去掉这行

node.name: {{ hostname }}

在ambari-server端对应的模板文件为:

# grep hostname /var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch*
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.master.yaml.j2:node.name: {{ hostname }}
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.master.yaml.j2.5.6.2: name: {{hostname}}
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2:node.name: {{ hostname }}
/var/lib/ambari-server/resources/common-services/ELASTICSEARCH/6.3.2/package/templates/elasticsearch.slave.yaml.j2.5.6.2: name: {{hostname}}

然后可以启动成功:

参考:

https://community.hortonworks.com/articles/215078/how-to-install-elk-stack-632-in-ambari.html
https://community.hortonworks.com/questions/212871/add-custom-service-via-ambari-error.html
https://community.hortonworks.com/articles/63557/how-to-edit-the-cluster-envxml-entries-using-ambar.html

【原创】大数据基础之Ambari(2)通过Ambari部署ElasticSearch(ELK)的更多相关文章

  1. 大数据基础环境--jdk1.8环境安装部署

    1.环境说明 1.1.机器配置说明 本次集群环境为三台linux系统机器,具体信息如下: 主机名称 IP地址 操作系统 hadoop1 10.0.0.20 CentOS Linux release 7 ...

  2. 【原创】大数据基础之Zookeeper(2)源代码解析

    核心枚举 public enum ServerState { LOOKING, FOLLOWING, LEADING, OBSERVING; } zookeeper服务器状态:刚启动LOOKING,f ...

  3. 【原创】大数据基础之Ambari(1)简介、编译安装、使用

    官方:http://ambari.apache.org/ The Apache Ambari project is aimed at making Hadoop management simpler ...

  4. 【原创】大数据基础之Impala(1)简介、安装、使用

    impala2.12 官方:http://impala.apache.org/ 一 简介 Apache Impala is the open source, native analytic datab ...

  5. 【原创】大数据基础之词频统计Word Count

    对文件进行词频统计,是一个大数据领域的hello word级别的应用,来看下实现有多简单: 1 Linux单机处理 egrep -o "\b[[:alpha:]]+\b" test ...

  6. 【原创】大数据基础之Benchmark(2)TPC-DS

    tpc 官方:http://www.tpc.org/ 一 简介 The TPC is a non-profit corporation founded to define transaction pr ...

  7. 大数据基础知识问答----hadoop篇

    handoop相关知识点 1.Hadoop是什么? Hadoop是一个由Apache基金会所开发的分布式系统基础架构.用户可以在不了解分布式底层细节的情况下,开发分布式程序.充分利用集群的威力进行高速 ...

  8. 大数据基础知识:分布式计算、服务器集群[zz]

    大数据中的数据量非常巨大,达到了PB级别.而且这庞大的数据之中,不仅仅包括结构化数据(如数字.符号等数据),还包括非结构化数据(如文本.图像.声音.视频等数据).这使得大数据的存储,管理和处理很难利用 ...

  9. 大数据基础知识问答----spark篇,大数据生态圈

    Spark相关知识点 1.Spark基础知识 1.Spark是什么? UCBerkeley AMPlab所开源的类HadoopMapReduce的通用的并行计算框架 dfsSpark基于mapredu ...

随机推荐

  1. 磁盘IO的性能指标 阻塞与非阻塞、同步与异步 I/O模型

    磁盘IO的性能指标 - 蝈蝈俊 - 博客园https://www.cnblogs.com/ghj1976/p/5611648.html 阻塞与非阻塞.同步与异步 I/O模型 - 蝈蝈俊.net - C ...

  2. java.lang.NullPointerException 错误原因

    [http-nio-8081-exec-1] ERROR o.a.c.c.C.[.[localhost].[/].[dispatcherServlet] - Servlet.service() for ...

  3. Package与Activity简介

    Package Package 包.只是在我们的app中这个Package是唯一的,就像你身份证号码一样.在我们做app自动化时,我们就需要知道他的Package,我们知道了Package那么也就知道 ...

  4. MIUI9 解锁并刷入TWRP后,删除解锁密码

    如果因为某种原因导致解锁密码失效(比如刷了其他ROM),还原备份回来之后,解锁密码失效了. 那么可以进入TWRP,然后通过  adb shell 进入\data\system\文件夹 用rm命令删除g ...

  5. git安装和GitHub使用

    一.git安装 1.git下载 https://github.com/git-for-windows/git/releases/tag/v2.17.1.windows.2 在如上地址下载git,并安装 ...

  6. 三、Java多人博客系统-技术架构

    多人博客系统1.0版本,架构和技术还是很简单和很传统的. 1.技术 前端:jsp.html.css.javascript.jquery.easyui.echarts 后端:spring mvc.Hib ...

  7. Python future使用

    Python的每个新版本都会增加一些新的功能,或者对原来的功能作一些改动.有些改动是不兼容旧版本的,也就是在当前版本运行正常的代码,到下一个版本运行就可能不正常了. 从Python 2.7到Pytho ...

  8. [BZOJ 4516] [SDOI 2016] 生成魔咒

    Description 魔咒串由许多魔咒字符组成,魔咒字符可以用数字表示.例如可以将魔咒字符 1.2 拼凑起来形成一个魔咒串 [1,2]. 一个魔咒串 S 的非空字串被称为魔咒串 S 的生成魔咒. 例 ...

  9. 【ARC101F】Robots and Exits 树状数组

    题目大意 有 \(n\) 个机器人和 \(m\) 个出口. 这 \(n\) 个机器人的初始位置是 \(a_1,a_2,\ldots,a_n\),这 \(m\) 个出口的位置是 \(b_1,b_2,\l ...

  10. Entity Framework入门教程(17)---记录和拦截数据库命令

    记录和拦截数据库命令 这一节介绍EF6怎么记录和拦截发送给数据库的查询和操作命令. 1.记录EF发送给数据库命令(DbContext.Database.Log) 以前给了查看EF发送给数据库的命令我们 ...