es sql是一个X-pack组件 ,允许对es执行类似sql的查询,可以将Elasticsearch SQL理解为一个编译器,既能理解es,又能理解sql。可以通过利用es,实施大规模实时读取和处理数据。

sql和es的映射关系

SQL Elasticsearch
columns field
raw document
table index
catalog or database cluster实例
cluster cluster

先插入一些数据:

PUT /my_index/doc/_bulk
{"index":{"_id":""}}
{"name":"lily","birthday":"2000-01-01","gender":"female"}
{"index":{"_id":""}}
{"name":"kangkang","birthday":"1998-04-01","gender":"male"}
{"index":{"_id":""}}
{"name":"jane","birthday":"1995-02-07","gender":"female"}

SQL REST API

POST /_xpack/sql?format=txt
{
"query":"select * from my_index where birthday<'1999-01-01' limit 2"
}
#  format类型有:json,yaml,smile,cbor,txt,csv,tsv
返回结果:
birthday | gender | name
------------------------+---------------+---------------
1998-04-01T00:00:00.000Z|male |kangkang
1995-02-07T00:00:00.000Z|female |jane
POST  /_xpack/sql?
{
"query":"select * from my_index order by birthday desc",
"fetch_size":1 # fetch_size 每页返回多少个结果
}
--------->
{
"columns": [
{
"name": "birthday",
"type": "date"
},
{
"name": "gender",
"type": "text"
},
{
"name": "name",
"type": "text"
}
],
"rows": [
[
"2000-01-01T00:00:00.000Z",
"female",
"lily"
]
],
"cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3WEZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmUyeFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHRnV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGN1pGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlMmhaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA="
}
# 该column对象只是第一页的一部分,当cursor结果中没有返回时,说明到达最后一页。
# 可以通过发回cursor字段继续下一页。在文本格式的情况下,光标作为Cursorhttp标头返回。
POST /_xpack/sql?format=json
{
"cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3WEZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmUyeFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHRnV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGN1pGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlMmhaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA="
}
#结果--------->
{
"rows": [
[
"1998-04-01T00:00:00.000Z",
"male",
"kangkang"
]
],
"cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3WEZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmUyeFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHRnV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGN1pGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlMmhaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA="
}
## -------------再次发回cursor: POST /_xpack/sql?format=json
{
"cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3WEZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmUyeFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHRnV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGN1pGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlMmhaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA="
}
#结果----------------》
{
"rows": []
}
#接收到最后一页时,清空es状态,没有cursor #要提前清理状态,可以使用 clear cursor
POST _xpack/sql/close
{
  "cursor": "k4bwAgFz5AFEbkYxWlhKNVZHaGxia1psZEdOb0JRQUFBQUFBQUY3NkZrbGtNa3R5V2s1VVZFTnRORmd3Y21Gd2VHeERMVkVBQUFBQUFBQmVfaFpKWkRKTGNscE9WRlJEYlRSWU1ISmhjSGhzUXkxUkFBQUFBQUFBWHZzV1NXUXlTM0phVGxSVVEyMDBXREJ5WVhCNGJFTXRVUUFBQUFBQUFGNzhGa2xrTWt0eVdrNVVWRU50TkZnd2NtRndlR3hETFZFQUFBQUFBQUJlX1JaSlpESkxjbHBPVkZSRGJUUllNSEpoY0hoc1F5MVL/////DwMBZghiaXJ0aGRheQEAAWYGZ2VuZGVyAAABZgRuYW1lAAA="
}
#结果——----------------->
{ "succeeded": true }

通过filter参数可以指定es的Query DSL来过滤

POST _xpack/sql?format=txt
{
"query":"select * from my_index order by birthday desc",
"filter":{
"term": {
"name": "kangkang"
}
},
"fetch_size":1
}
# 除了query和cursor字段外 请求还可以包括fetch_size和time_zone
# fetch_size 每页返回多少个结果
# time_zone 日期函数和日期解析的时区,默认为utc

SQL Translate API

sql translate api接受json文档中的sql并将其转换为es查询。

POST _xpack/sql/translate
{
"query":"select * from my_index order by birthday",
"fetch_size":3
}
#结果----------------->
{
"size": 3,
"_source": {
"includes": [
"gender",
"name"
],
"excludes": []
},
"docvalue_fields": [
"birthday"
],
"sort": [
{
"birthday": {
"order": "asc"
}
}
]
}

SQL CLI

可以用命令行形式,在x-pack的bin目录执行查询语句:

# ./elasticsearch-sql-cli
sql> select * from my_index where birthday<'1999-01-01';
birthday | gender | name
------------------------+---------------+---------------
1998-04-01T00:00:00.000Z|male |kangkang
1995-02-07T00:00:00.000Z|female |jane

SQL JDBC

将jdbc调用转化为es sql

SQL 语句

  • describe table

    # DESC table
    # DESCRIBE table
    POST _xpack/sql?format=txt
    {
    "query":"describe my_index"
    }
    ---------->
    column | type
    ---------------+---------------
    birthday |TIMESTAMP
    gender |VARCHAR
    gender.keyword |VARCHAR
    name |VARCHAR
    name.keyword |VARCHAR
  • select

    # SELECT select_expr [, ...]
    [ FROM table_name ]
    [ WHERE condition ]
    [ GROUP BY grouping_element [, ...] ]
    [ HAVING condition]
    [ ORDER BY expression [ ASC | DESC ] [, ...] ]
    [ LIMIT [ count ] ]
  • show columns
    #SHOW COLUMNS [ FROM | IN ] ? table
    
    POST _xpack/sql?format=txt
    {
    "query":"show columns in my_index"
    }

    column | type
    ---------------+---------------
    birthday |TIMESTAMP
    gender |VARCHAR
    gender.keyword |VARCHAR
    name |VARCHAR
    name.keyword |VARCHAR

  • show functions
    #SHOW FUNCTIONS [ LIKE? pattern? ]?  
    
     POST _xpack/sql?format=txt
    {
      "query":"show functions like 'sum%'"
    } name | type
    ---------------+---------------
    SUM |AGGREGATE
    SUM_OF_SQUARES |AGGREGATE 
  • show tables
    # SHOW TABLES [ LIKE? pattern? ]?
    
    POST _xpack/sql?format=txt
    {
    "query":"show tables like 'my_index'"
    }
    #------------------------>
    name | type
    ---------------+---------------
    my_index |BASE TABLE

functions and operators

  • 比较运算符:   = , < , <= , > , >=,  不等于 <>  !=  <=> , between,is null/is not null
  • 逻辑运算符: AND ,OR ,NOT
  • 数字运算符:  +  -  *  / %
    POST _xpack/sql?format=txt
    {
    "query":"select 1+1 as x"
    }
    ---------->
    x
    ---------------
    2
  • 数学函数:  abs(绝对值), crbt(立方根),round(四舍五入)....
    POST _xpack/sql?format=txt
    {
    "query":"select abs(age) from test_index "
    }
    --------->
    ABS(age)
    ---------------
    27
  • 时间和日期函数: year, month, week, doy, dow, hour ,minute_of_day, minute,second,extract
    POST _xpack/sql?format=txt
    {
    "query":"select year(cast('2018-07-12' as timestamp )) as year"
    } #从日期中提取年份
    ------->
    year
    ---------------
    2018
  • 聚合:  avg , count , count(distinct) , max , min , sum 
    POST _xpack/sql/?format=txt
    {
    "query":"select avg(age) as avg from test_index"
    } POST _xpack/sql?format=txt
    {
    "query":"select count(distinct age) as count from test_index"
    } #不同值的个数

Elasticsearch SQL的更多相关文章

  1. 使用JDBC连接ElasticSearch6.3(ElasticSearch SQL JDBC)

    使用JDBC连接ElasticSearch6.3(ElasticSearch SQL JDBC) https://blog.csdn.net/scgaliguodong123_/article/det ...

  2. Elasticsearch SQL用法详解

    Elasticsearch SQL用法详解  mp.weixin.qq.com 本文详细介绍了不同版本中Elasticsearch SQL的使用方法,总结了实际中常用的方法和操作,并给出了几个具体例子 ...

  3. elasticsearch sql插件 2.4及以下版本配置

    github地址:https://github.com/NLPchina/elasticsearch-sql/ 方式一:github elasticsearch-sql上提供的安装方法cmd进入到本地 ...

  4. 手写一个简单的ElasticSearch SQL转换器(一)

    一.前言 之前有个需求,是使ElasticSearch支持使用SQL进行简单查询,较新版本的ES已经支持该特性(不过貌似还是实验性质的?) ,而且git上也有elasticsearch-sql 插件, ...

  5. elasticsearch sql插件配置(5.0及以上版本)

    github官方参考地址:https://github.com/NLPchina/elasticsearch-sql/ 采用 git + node 的方式,所以安装前需要先安装好node,node n ...

  6. Elasticsearch:Elasticsearch SQL介绍及实例(二)

    转载自:https://blog.csdn.net/UbuntuTouch/article/details/105699014

  7. Elasticsearch:Elasticsearch SQL介绍及实例 (一)

    转载自:https://blog.csdn.net/UbuntuTouch/article/details/105658911

  8. 搜索引擎ElasticSearch系列(四): ElasticSearch2.4.4 sql插件安装

    一:ElasticSearch sql插件简介 With this plugin you can query elasticsearch using familiar SQL syntax. You ...

  9. elasticsearch与ms sql server数据同步

    MS SQL Server Download Elasticsearch Install Elasticsearch Follow instructions on https://www.elasti ...

随机推荐

  1. CSS的再深入3(更新中···)

    在前面,我们学习了标准文档流,但在实际制作的过程中,用标准文档流书写显然是不现实的,因此,我们来了解几种脱离标准文档流的方法: 1.float 浮动 float:left/right:(左浮/右浮) ...

  2. memalign的作用【转】

    本文转载自:https://blog.csdn.net/lvwx369/article/details/41726415 转自:http://hi.baidu.com/narshben/item/ca ...

  3. 比较好的一些 ConcurrentHashMap讲解博客

    jdk8 https://blog.csdn.net/jianghuxiaojin/article/details/52006118#commentBox jdk7.8 https://crossov ...

  4. NLog——ElasticSearch——Kibana

    Nlog.elasticsearch.Kibana以及logstash在项目中的应用(一) Nlog.elasticsearch.Kibana以及logstash在项目中的应用(二) ASP.NET ...

  5. Kubernetes简介

    Kubernetes is an open-source platform designed to automate deploying, scaling, and operating applica ...

  6. 基于OAuth2.0的第三方认证

    浅显易懂的解释 来源 yahoo OAuth认证 原理 理解OAuth 2.0:原理.分类 一张图搞定OAuth2.0:是什么,怎么用 应用自身,完成用户认证: 缺点: 1.不同的访问Web应用提供不 ...

  7. Oracle imp导入数据

    拿到别人给的dmp文件如何导入自己的库中呢?上码: 代码: imp dms/123321@orcl file=F:\TP_ZG_20171208.DMP feedback=10000 buffer=1 ...

  8. 虚拟现实外包公司—北京动点飞扬软件承接VR/AR软件、游戏外包

    欢迎通过以下方式联系北京动点飞扬软件外包业务咨询QQ:372900288 (全天在线)   咨询邮箱:SLteam@vip.qq.com   同时我们还承接 HTML5外包. kinect外包.Uni ...

  9. input标签让光标不出现

    <input  class="red" readonly unselectable="on" > input点击变为搜索框,用form包住绑定搜索事 ...

  10. netstat 在windows下和Linux下查看网络连接和端口占用

    假设忽然起个服务,告诉我8080端口被占用了,OK,我要去看一下是什么服务正在占用着,能不能杀 先假设我是在Windows下: 第一列: Proto 协议 第二列: 本地地址[ip+端口] 第三列:远 ...