宜人贷蜂巢ELK Stack之elasticsearch权限探索

前言

上文《宜人贷蜂巢API网关技术解密之Netty使用实践》提到了，API网关“承外对内”，将外部请求，转发到内部各个抓取服务。在网关中，不仅可以做鉴权、加解密、路由、限流功能；如果想了解各服务接口的调用次数、调用时间、成功次数、失败次数等接口相关的统计，都可以网关中埋点，采集相关信息，通过logstash放入elasticsearch（简称es），在kibana中或通过es的http请求获得各服务接口的统计信息。

ELK框架

前段时间有一个需求，我们的elasticsearch需要作为一个服务提供出来。考虑到elasticsearch内保存着敏感数据，需要对elasticsearch添加权限管理功能。搜寻elasticsearch权限管理，大概有三套方案：

• ELK Stack官方安全插件X-Pack Securit
• es安全插件Search Guard
• 自实现权限管理

考虑到X-Pack Security是收费软件，自开发一个权限管理模块成本比较高，最后选用了Search Guard的社区版本，其支持的功能已经满足当前需求。

Search Guard安装过程

• Stop Elasticsearch

ps -ef | grep elasticsearch | grep -v grep | awk {'print $2'} | xargs kill -9

• Install Search Guard

/opt/soft/elasticsearch-5.5.1-guard/bin/ elasticsearch-plugin install -b com.floragunn:search-guard-5:5.5.1-14

如果线上服务器不能访问外网，可以事先下载对应版本的 Search Guard，再以本地模式安装。

/opt/soft/elasticsearch-5.5.1-guard/bin/ elasticsearch-plugin install -b file:///opt/soft/search-guard-5-5.5.1-15.zip

• 产生证书及配置es

如果在测试环境安装es，可以使用软件自带的脚本，生成CA及相应证书，并自动配置了es

sh /opt/soft/elasticsearch-5.5.1-guard/plugins/search-guard-5/tools/install_demo_configuration.sh

但以上方式不适用于线上环境，我们的es希望在线上环境能提供权限访问控制，所以采用了search-guard-ssl生成线上证书

• 首先，下载了search-guard-ssl项目

git clone https://github.com/floragunncom/search-guard-ssl.git

生成证书，可以参考官方文档详细说明

• 建议修改证书信息及密码

example-pki-scripts/etc/root-ca.conf example-pki-scripts/etc/signing-ca.conf

• 最后，执行证书生成脚本sh ./example.sh

• 将相应的证书拷贝到es的config文件夹下

• 修改es配置

cluster.name: honeycomb-es-guard-# node  to
node.name: node-
path.data: /data01/elasticsearch-guard
path.logs: /opt/logs/elasticsearch-guard

network.host: 0.0.0.0

http.cors.enabled: true
http.cors.allow-origin: "*"
thread_pool.bulk.queue_size: 

searchguard.ssl.transport.keystore_filepath: node--keystore.jks
searchguard.ssl.transport.keystore_password: node_es
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: r_ca_honeycomb
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.enable_openssl_if_available: true
searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: node--keystore.jks
searchguard.ssl.http.keystore_password: node_es
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: r_ca_honeycomb

searchguard.authcz.admin_dn:
  - CN=*,OU=client,O=client,L=test, C=de
  - CN=kirk,OU=client,O=client,L=test,C=DE
searchguard.ssl.http.clientauth_mode: NONE

• Restart Elasticsearch.

  su elk
  /opt/soft/elasticsearch--guard/bin/elasticsearch -d

• Initialise the Search Guard index by running sgadmin

<span style=-guard/plugins/search-guard-/tools/sgadmin.sh -cd /opt/soft/elasticsearch--guard/plugins/search-guard-/sgconfig/ -cn honeycomb-es-guard- -ks /opt/soft/elasticsearch--guard/config/kirk-keystore.jks -kspass ki_ca_1 -ts /opt/soft/elasticsearch--guard/config/truststore.jks -tspass r_ca_honeycomb –nhnv</span>

• 测试结果

curl -k -u admin:admin 'https://localhost:9200/_searchguard/authinfo?pretty'

es装上Search Guard后的ELK框架

kibana修改步骤

安装插件

/opt/soft/kibana--linux-x86_64-guard/bin/kibana-plugin install file:///opt/soft/searchguard-kibana-5.5.1-4.zip

kibana配置

server.port: 5601

server.host: "0.0.0.0"

elasticsearch.url: "https://localhost:9200"

kibana.index: ".kibana"

elasticsearch.username: "kibanaserver"

elasticsearch.password: "kibanaserver"

elasticsearch.ssl.verificationMode: none

• start kibana

/opt/soft/kibana-5.5.1-linux-x86_64-guard/bin/kibana &

• 访问页面

logstash修改步骤

logstash在按照官方文档配置，且咨询Search Guard官方后，并没有安装成功；现列出操作过程及问题现象，如有哪位读者了解，还望不吝赐教

• Configure logstash to use HTTPS instead of HTTP. 例如，hosts => "https://..."
• If you want to verify the server’s certificate (optional, but recommended), you need to provide the path to the keystore containing your Root CA as well. logstash不需要验证es的证书，所以设置sslcertificateverification => false 如果进行证书验证，需配置CA路径及密码
• Configure the logstash username and password.

最终配置

output {
  elasticsearch {
        hosts => ["https://xx.xx.xx.32:9200"]
        index => "%{log_project}-%{+YYYY-MM-dd}"

        ssl => true
        ssl_certificate_verification => false
        #truststore => "/opt/soft/elasticsearch-5.5.1-guard/config/truststore.jks"
        #truststore_password => r_ca_honeycomb

        user => logstash
        password => logstash
  }
}

• start logstash

/opt/soft/logstash-2.4.1-guard/bin/logstash -f /opt/soft/logstash-2.4.1-guard/conf/honeycomb-logstash.conf

问题

1、Search Guard 建议logstash不做证书验证且不需要提供truststore和truststore_password，但按操作后，进程启动成功，但仍有fail提示

** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true {:level=>:warn}
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:class=>"Manticore::ClientProtocolException", :level=>:error}
Pipeline main started

2、logstash在启动后，发送数据给es时，始终报错

Attempted to send a bulk request to Elasticsearch configured at '["https://xx.xx.xx.32:9200"]',
 but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?
{:error_message=>"PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested
target", :error_class=>"Manticore::ClientProtocolException", :backtrace=>["/opt/soft/logstash-
-guard/vendor/bundle/jruby/-java/lib/manticore/response.rb::in `initialize'",
"org/jruby/RubyProc.java:281:in `call'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in `call'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call_once'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform_request'",
"org/jruby/RubyProc.java:281:in `call'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in `perform_request'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in `bulk'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'",
"org/jruby/ext/thread/Mutex.java:149:in `synchronize'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in `multi_receive'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'",
"org/jruby/RubyHash.java:1342:in `each'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:in `worker_loop'",
"/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}

看日志似乎是目的不可达，但在logstash机器上，通过命令测试，是可以访问的 curl -k -u logstash:logstash 'https://xx.xx.xx.32:9200/_searchguard/authinfo?pretty'

另外es有打印错误日志

[--24T10::,][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:) ~[?:1.8.0_131]
        at io.netty.handler.ssl.SslHandler$SslEngineType$.unwrap(SslHandler.java:) ~[netty-handler-.Final.jar:.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:) ~[netty-handler-.Final.jar:.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:) ~[netty-handler-.Final.jar:.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:) ~[netty-codec-.Final.jar:.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:) ~[netty-codec-.Final.jar:.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:) ~[netty-codec-.Final.jar:.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:) [netty-transport-.Final.jar:.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$.run(SingleThreadEventExecutor.java:) [netty-common-.Final.jar:.Final]
        at java.lang.Thread.run(Thread.java:) [?:1.8.0_131]
[--24T10::,][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:) ~[?:1.8.0_131]
        at io.netty.handler.ssl.SslHandler$SslEngineType$.unwrap(SslHandler.java:) ~[netty-handler-.Final.jar:.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:) ~[netty-handler-.Final.jar:.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:) ~[netty-handler-.Final.jar:.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:) ~[netty-codec-.Final.jar:.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:) ~[netty-codec-.Final.jar:.Final]

3、上文es的错误日志，显示es不知道证书（应该是客户端证书），但是已经配置了不做客户端认证，这又是为什么？

es部分配置

searchguard.ssl.http.clientauth_mode: NONE

总结

虽然logstash与es通信暂时没有成功，但是在尝试开通es权限的过程中，也积累了些经验和成果。

• 使用类似文中curl的方式（http请求），直接使用es服务。
• 添加了Search Guard的es如何对外提供域名访问，实现后端多点服务/路由。

nginx可以处理来自外部的https请求，但目前不支持https代理，即nginx不能访问后端的https服务。如果仍然使用Search Guard，可以考虑使用四层负载均衡器lvs或者硬件负载均衡器，将请求透传给后端服务器；另外知乎文章上有推荐使用Squid。

参考

https://floragunn.com/

https://github.com/floragunncom/search-guard-docs

https://github.com/floragunncom/search-guard-docs/blob/master/installation.md

https://github.com/floragunncom/search-guard-docs/blob/master/tlsgeneratedemo_certificates.md

https://github.com/floragunncom/search-guard-ssl/tree/5.5.0

https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md

https://github.com/floragunncom/search-guard-docs/blob/master/logstash.md

https://forum.nginx.org/read.php?2,15124,15256

https://www.zhihu.com/question/19871146

作者：蜂巢团队

来源：宜信技术学院