ElasticSearch7.3学习(二十四)----相关度评分机制详解

1、算法介绍

relevance score(相关性分数) 算法，简单来说，就是计算出，一个索引中的文本，与搜索文本，他们之间的关联匹配程度。Elasticsearch使用的是 term frequency/inverse document frequency算法，简称为TF/IDF算法。TF词频(Term Frequency)，IDF逆向文件频率(Inverse Document Frequency)

1.1 Term frequency

搜索文本中的各个词条在field文本中出现了多少次，出现次数越多，就越相关。

数学公司并不重要，看下面例子就清楚了

搜索请求：阿莫西林
doc1：阿莫西林胶囊是什么。。。阿莫西林胶囊能做什么。。。。阿莫西林胶囊结构
doc2：本药店有阿莫西林胶囊、红霉素胶囊、青霉素胶囊。。。

很容易发现对于阿莫西林关键词来说在doc1中出现的次数大于doc2的，所以doc1的优先级高于doc2

1.2 Inverse document frequency

搜索文本中的各个词条在整个索引的所有文档中出现了多少次，出现的次数越多，就越不相关.

首先看下面内容

搜索请求：阿莫西林胶囊
doc1：A市健康大药房简介。本药店有阿莫西林胶囊、红霉素胶囊、青霉素胶囊。。。
doc2：B市民生大药房简介。本药店有阿莫西林胶囊、红霉素胶囊、青霉素胶囊。。。
doc3：C市未来大药房简介。本药店有阿莫西林胶囊、红霉素胶囊、青霉素胶囊。。。

可以看到，对于关键词阿莫西林来说，所有的doc里面都包含这个关键词，那说明这个关键词不是那么重要，说明这个关键词所占的权重很低。再看下面内容

搜索请求：A市 阿莫西林胶囊
doc1：A市健康大药房简介。本药店有阿莫西林胶囊、红霉素胶囊、青霉素胶囊。。。
doc2：B市民生大药房简介。本药店有阿莫西林胶囊、红霉素胶囊、青霉素胶囊。。。
doc3：C市未来大药房简介。本药店有阿莫西林胶囊、红霉素胶囊、青霉素胶囊。。。

再加上A市这个关键词，这样的话只有doc1里面才存在，这样的话权重才高，所以可以得出结论：整个索引库中出现的词的频率越小，那么相关度权重越高。

1.3 Field-length norm

除了上面两个因素影响相关度评分的计算之外，还有一个就是字段长度也会影响评分的计算。具体来说就是，field的长度越长，相关度越弱

搜索请求：A市 阿莫西林胶囊
doc1：{"title":"A市健康大药房简介。","content":"本药店有、红霉素胶囊、青霉素胶囊。。。（一万字）"}
doc2：{"title":"B市民生大药房简介。","content":"本药店有阿莫西林胶囊、红霉素胶囊、青霉素胶囊。。。（一万字）"}

两个文档均只有一个字段被命中。为啥doc1>doc2，因为title字段的长度小于content的字段，几个字就命中相比于一万字才命中，当然几个字就命中的排在前面

2、 _score是如何被计算出来的

步骤如下：

1. 对用户输入的关键词分词
2. 每个分词分别计算对每个匹配文档的TF和IDF值
3. 综合每个分词的TF/IDF值，利用公式计算每个文档总分
4. 最后按照score降序返回

可以举个例子来看一下。这里使用explain关键字来解释排序的过程。

首先创建索引

PUT /book/
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart"
      },
      "description": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart"
      },
      "studymodel": {
        "type": "keyword"
      },
      "price": {
        "type": "double"
      },
      "timestamp": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
      },
      "pic": {
        "type": "text",
        "index": false
      }
    }
  }
}

接着添加测试数据

PUT /book/_doc/1
{
  "name": "Bootstrap开发",
  "description": "Bootstrap是一个非常流行的开发框架。此开发框架可以帮助不擅长css页面开发的程序人员轻松的实现一个css，不受浏览器限制的精美界面css效果。",
  "studymodel": "201002",
  "price": 38.6,
  "timestamp": "2019-08-25 19:11:35",
  "pic": "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",
  "tags": [
    "bootstrap",
    "dev"
  ]
}

PUT /book/_doc/2
{
  "name": "java编程思想",
  "description": "java语言是世界第一编程语言，在软件开发领域使用人数最多。",
  "studymodel": "201001",
  "price": 68.6,
  "timestamp": "2019-08-25 19:11:35",
  "pic": "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",
  "tags": [
    "java",
    "dev"
  ]
}

PUT /book/_doc/3
{
  "name": "spring开发基础",
  "description": "spring 在java领域非常流行，java程序员都在用。",
  "studymodel": "201001",
  "price": 88.6,
  "timestamp": "2019-08-24 19:11:35",
  "pic": "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",
  "tags": [
    "spring",
    "java"
  ]
}

然后在使用如下命令查看_score的计算

GET /book/_search?explain=true
{
  "query": {
    "match": {
      "description": "java程序员"
    }
  }
}

返回的内容太多，这里只展示第一条的数据的内容

查看代码

      {
        "_shard" : "[book][0]",
        "_node" : "Alyo4fMoSyCi6eo7A6t_XA",
        "_index" : "book",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 1.9788694,
        "_source" : {
          "name" : "spring开发基础",
          "description" : "spring 在java领域非常流行，java程序员都在用。",
          "studymodel" : "201001",
          "price" : 88.6,
          "timestamp" : "2019-08-24 19:11:35",
          "pic" : "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",
          "tags" : [
            "spring",
            "java"
          ]
        },
        "_explanation" : {
          "value" : 1.9788694,
          "description" : "sum of:",
          "details" : [
            {
              "value" : 0.7502767,
              "description" : "weight(description:java in 0) [PerFieldSimilarity], result of:",
              "details" : [
                {
                  "value" : 0.7502767,
                  "description" : "score(freq=2.0), product of:",
                  "details" : [
                    {
                      "value" : 2.2,
                      "description" : "boost",
                      "details" : [ ]
                    },
                    {
                      "value" : 0.47000363,
                      "description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                      "details" : [
                        {
                          "value" : 2,
                          "description" : "n, number of documents containing term",
                          "details" : [ ]
                        },
                        {
                          "value" : 3,
                          "description" : "N, total number of documents with field",
                          "details" : [ ]
                        }
                      ]
                    },
                    {
                      "value" : 0.7256004,
                      "description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                      "details" : [
                        {
                          "value" : 2.0,
                          "description" : "freq, occurrences of term within document",
                          "details" : [ ]
                        },
                        {
                          "value" : 1.2,
                          "description" : "k1, term saturation parameter",
                          "details" : [ ]
                        },
                        {
                          "value" : 0.75,
                          "description" : "b, length normalization parameter",
                          "details" : [ ]
                        },
                        {
                          "value" : 12.0,
                          "description" : "dl, length of field",
                          "details" : [ ]
                        },
                        {
                          "value" : 23.666666,
                          "description" : "avgdl, average length of field",
                          "details" : [ ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "value" : 1.2285928,
              "description" : "weight(description:程序员 in 0) [PerFieldSimilarity], result of:",
              "details" : [
                {
                  "value" : 1.2285928,
                  "description" : "score(freq=1.0), product of:",
                  "details" : [
                    {
                      "value" : 2.2,
                      "description" : "boost",
                      "details" : [ ]
                    },
                    {
                      "value" : 0.98082924,
                      "description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                      "details" : [
                        {
                          "value" : 1,
                          "description" : "n, number of documents containing term",
                          "details" : [ ]
                        },
                        {
                          "value" : 3,
                          "description" : "N, total number of documents with field",
                          "details" : [ ]
                        }
                      ]
                    },
                    {
                      "value" : 0.56936646,
                      "description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                      "details" : [
                        {
                          "value" : 1.0,
                          "description" : "freq, occurrences of term within document",
                          "details" : [ ]
                        },
                        {
                          "value" : 1.2,
                          "description" : "k1, term saturation parameter",
                          "details" : [ ]
                        },
                        {
                          "value" : 0.75,
                          "description" : "b, length normalization parameter",
                          "details" : [ ]
                        },
                        {
                          "value" : 12.0,
                          "description" : "dl, length of field",
                          "details" : [ ]
                        },
                        {
                          "value" : 23.666666,
                          "description" : "avgdl, average length of field",
                          "details" : [ ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      },

对于上面的返回结果，我们先看第一部分，首先就是返回的数据

接着就是对评分计算的解释，按照上面给出的4个步骤分析，首先对关键词分词，这里分为了java 程序员两个关键词，先来看看java关键词的解释

可以看到计算java关键词的tf，idf的值，同理在下方也能看到计算程序员关键词的tf，idf的值。

最后将两个关键词合并起来在计算整个doc的总分，即得到最终的_score值，如下所示。

3、document判断是否被匹配

测试判断一个文档能不能被搜索到，适用于生产环境

例如

GET /book/_explain/1
{
  "query": {
    "match": {
      "description": "java程序员"
    }
  }
}

返回

可以看到对于id为1的doc，并不能匹配到该文档，再来试一下id为3的数据

GET /book/_explain/3
{
  "query": {
    "match": {
      "description": "java程序员"
    }
  }
}

返回

可以看到能够被匹配到，并且能够根据内容来分析为什么该文档能够被匹配到。