elasticsearch相关概念及常用操作汇总

背景

　　我本来是想把我的写的es的平时总结dsl发出来的,但是我发现只搞那个意义大不.干脆多写点吧.

索引的结构化和非结构

　　我们经常用数据库,当然会经常用到索引.

　　然后从索引的维度去分析,系统分为结构化索引和非结构化索引.

　　结构化索引,就是可以预设索引,比如id,可以预见的. 维护方便,性能高.

　　非结构化索引,就是数据进来之前,我可能都不知道要进来什么. 然后数据进来后,我要进行分析. 比如,挨个扫描节点,看是否包含,比较墨迹. 或者将非结构化的数据的一部分信息提取出来,使其变得有结构,感觉很像抽象,有了结构,那就好搞很多.

elasticSearch简述

elasticSearch又称es,是一种分布式的搜索引擎工具,之前我们比较熟悉的应该是luence,然后再往前推进,就是solr;

三者的关系

luence是一个信息检索工具包,类似SDK,提供了全面的api,也可以做高级查询.但是作为一个搜索引擎来讲的话,并不太完整,因为缺乏了一些获取数据,解析,分词等功能.而且,用起来较为复杂.所以,我们可能更需要完整而且更简洁的搜索引用服务. 简而言之,很全面,很基础,用起来麻烦.

solr是基于luence,是对lucen的进一步封装,补全.提供了对应的api供我们调用,很容易就能实现检索服务,而且更为完整,可以根据配置文件解析数据等.但是因为是更高一层的封装,所以luence很多新特性不能及时透传,也就是luence能查,但是solr查不了.简而言之,solr是独立的,面向企业级的,搜索引擎.

es也是基于luence,并提供了更高层次的封装的搜索引擎.特点对索引进行分片,将一个大的索引拆分多个,分布到不同节点上,降低服务器压力,构成分布式搜索,从大大提升搜索效率.是一款近乎实时分析的搜索引擎,非常强力.而且采用了restful风格,更易于上手.简而言之,分片机制,分布式搜索,实时,很强力.

实际开发中的dsl语句

记得装了es之后,再装个kibana,然后再kibana上跑这些程序.

#处理下hyd_test的全文导入
DELETE hyd_test

GET /hyd_test/_doc/_search

#批量更新
  POST /hyd_test/_doc/_bulk
  {"update":{"_id":1}}
  {"doc":{"reference_law":"《中华人民共和国民事诉讼法》第二百一十条 、第二百三十三条 、第二百零七条 、第一百七十条","trial_court":"最高人民法院","assistan t_judge":"齐召财","clerk":"齐召财","lawyer":"","law_firm":"","legal_basis":"","created_time":"2020-09-02 15:32:00","created_by":"admi  n","updated_time":"2020-09-02 15:34:12","updated_by":"","is_delete":0},"doc_as_upsert":true}

#批量写入
POST test_demo/_doc/_bulk
{"create":{"_id":1}}
{"case_name":"你好你好 ,嗨美女","count":18}
{"create":{"_id":2}}
{"case_name":"好的好的,大美女","count":19}

#查询结构
GET /test_demo/_mapping

#查询分词_analyze
POST /test_demo/_analyze
{
  "analyzer": "ik_max_word",
  "text": "你好你好 ,嗨美女"
}

#match查询
GET /test_demo/_doc/_search
{
  "query": {
    "match": {
      "case_name": {
        "query": "是个美女"
      }
    }
  }
}

#精确匹配term
GET /test_demo/_doc/_search
{
  "query": {
    "term": {
      "case_name": {
        "value": "是个美女"
      }
    }
  }
}

#and 查询
GET hyd_test/_search
{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "trial_year": "2018"
          }
        },
        {
          "match": {
            "case_cause_id": "100"
          }
        }
      ]
    }
  }
}
#统计
GET hyd_test/_search
{
  "size": 0,
  "query": {
    "match": {
      "case_cause_name": "相邻通行纠纷"
    }
  },
  "aggs": {

    "document_type_name":{
       "terms": {
        "field": "document_type_name",
        "size": 10
      }
    }
  }
}

#清空所有数据
POST authoritative_case/_delete_by_query
{
  "query":{
    "match_all":{

  }
  }
}

#高亮. 注意, 高亮必须match也要带上同样的字段.
GET hyd_test/_search
{

  "query": {
    "match": {
      "case_cause_name": "相邻通行纠纷"
    }
  },
  "highlight": {
    "fields": {
      "case_cause_name":{

      }
    }
  }

}

#高亮,加颜色,并且不要求 number_of_fragments 分段. 默认为5 ,就是划分为5断.
GET hyd_test/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "一审"
          }
        },
        {
          "match": {
            "id": {
              "query": 1
            }
          }
        }
      ]
    }
  },
  "highlight": {
    "pre_tags": [
      "<span style='color:red'>"
    ],
    "post_tags": [
      "</span>"
    ],
    "fragment_size": 1000,
    "number_of_fragments": 0,
    "require_field_match": false,
    "fields": {
      "*": {}
    }
  }
}

#id为1的全文高亮
GET hyd_test/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "民事"
          }
        },
        {
          "query_string": {
            "query": "理由"
          }
        },
        {
          "match": {
            "id": "1"
          }
        }
      ]
    }
  },
  "highlight": {
    "require_field_match": "false",
    "fields": {
      "*": {}
    }
  }
}

#查询嵌套对象: path里面是对象visible_user query最里面是实际字段visible_user.visible_user_id
GET gzmx_clue/_search
{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "nested": {
            "path": "visible_user",
            "query": {
              "terms": {
                "visible_user.visible_user_id": [
                  185
                ]
              }
            }
          }
        }
      ]
    }
  }
}

#复杂查询 先取交集,再取并集. should(must,must) .
GET gzmx_clue/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "bool": {
            "must": {
              "match": {
                "procuratorate_code": "2"
              }
            },
            "must_not": {
              "match": {
                "source": "9"
              }
            }
          }
        },
        {
          "bool": {
            "must": [
              {
                "match": {
                  "source": "9"
                }
              },
              {
                "nested": {
                  "path": "visible_user",
                  "query": {
                    "terms": {
                      "visible_user.visible_user_id": [
                        186
                      ]
                    }
                  }
                }
              },
              {
                "match": {
                  "procuratorate_code": "2"
                }
              }
            ]
          }
        }
      ]
    }
  }
}

#must和shoule交集使用会导致should不起效. 所以, 尽量把should放在must里面.
{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "terms": {
            "clue_feature": [
              "0"
            ],
            "boost": 1
          }
        },
        {
          "bool": {
            "should": [
              {
                "bool": {
                  "must": [
                    {
                      "match": {
                        "procuratorate_code": {
                          "query": 2,
                          "operator": "OR",
                          "prefix_length": 0,
                          "max_expansions": 50,
                          "fuzzy_transpositions": true,
                          "lenient": false,
                          "zero_terms_query": "NONE",
                          "auto_generate_synonyms_phrase_query": true,
                          "boost": 1
                        }
                      }
                    }
                  ],
                  "must_not": [
                    {
                      "match": {
                        "source": {
                          "query": 9,
                          "operator": "OR",
                          "prefix_length": 0,
                          "max_expansions": 50,
                          "fuzzy_transpositions": true,
                          "lenient": false,
                          "zero_terms_query": "NONE",
                          "auto_generate_synonyms_phrase_query": true,
                          "boost": 1
                        }
                      }
                    }
                  ],
                  "adjust_pure_negative": true,
                  "boost": 1
                }
              },
              {
                "bool": {
                  "must": [
                    {
                      "nested": {
                        "query": {
                          "terms": {
                            "visible_user.visible_user_id": [
                              432
                            ],
                            "boost": 1
                          }
                        },
                        "path": "visible_user",
                        "ignore_unmapped": false,
                        "score_mode": "none",
                        "boost": 1
                      }
                    },
                    {
                      "match": {
                        "procuratorate_code": {
                          "query": 2,
                          "operator": "OR",
                          "prefix_length": 0,
                          "max_expansions": 50,
                          "fuzzy_transpositions": true,
                          "lenient": false,
                          "zero_terms_query": "NONE",
                          "auto_generate_synonyms_phrase_query": true,
                          "boost": 1
                        }
                      }
                    },
                    {
                      "match": {
                        "source": {
                          "query": 9,
                          "operator": "OR",
                          "prefix_length": 0,
                          "max_expansions": 50,
                          "fuzzy_transpositions": true,
                          "lenient": false,
                          "zero_terms_query": "NONE",
                          "auto_generate_synonyms_phrase_query": true,
                          "boost": 1
                        }
                      }
                    }
                  ],
                  "adjust_pure_negative": true,
                  "boost": 1
                }
              }
            ],
            "adjust_pure_negative": true,
            "boost": 1
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  },
  "sort": [
    {
      "created_time": {
        "order": "desc"
      }
    },
    {
      "id": {
        "order": "desc"
      }
    }
  ],
  "track_total_hits": 2147483647
}
#查询所有数量
GET gzmx_clue/_search
{
    "track_total_hits":true,
    "query":{
        "match_all":{}
    }
}