N-gram 词元生成器 #

N-gram 词元生成器会将文本拆分为指定长度的重叠 n-gram（固定长度为 n 的字符序列）。当你希望实现部分单词匹配或自动补全搜索功能时，这个分词器特别有用，因为它会生成原始输入文本的子字符串（ n-gram 字符串）。

参考样例 #

以下示例请求创建了一个名为 my_index 的新索引，并配置了一个使用 n-gram 词元生成器的分词器。

PUT /my_index
{
  "settings": {
    "analysis": {
      "tokenizer": {
        "my_ngram_tokenizer": {
          "type": "ngram",
          "min_gram": 3,
          "max_gram": 4,
          "token_chars": ["letter", "digit"]
        }
      },
      "analyzer": {
        "my_ngram_analyzer": {
          "type": "custom",
          "tokenizer": "my_ngram_tokenizer"
        }
      }
    }
  }
}

生成的词元 #

使用以下请求来检查使用该分词器生成的词元：

POST /my_index/_analyze
{
  "analyzer": "my_ngram_analyzer",
  "text": "Search"
}

返回内容包含产生的词元

{
  "tokens": [
    {"token": "Sea","start_offset": 0,"end_offset": 3,"type": "word","position": 0},
    {"token": "Sear","start_offset": 0,"end_offset": 4,"type": "word","position": 1},
    {"token": "ear","start_offset": 1,"end_offset": 4,"type": "word","position": 2},
    {"token": "earc","start_offset": 1,"end_offset": 5,"type": "word","position": 3},
    {"token": "arc","start_offset": 2,"end_offset": 5,"type": "word","position": 4},
    {"token": "arch","start_offset": 2,"end_offset": 6,"type": "word","position": 5},
    {"token": "rch","start_offset": 3,"end_offset": 6,"type": "word","position": 6}
  ]
}

参数说明 #

N-gram 词元生成器可以使用以下参数进行配置。

关于 min_gram 与 max_gram 的最大差值 #

min_gram 和 max_gram 之间的最大差值可通过索引级别的 index.max_ngram_diff 设置进行配置，其默认值为 1。

以下示例请求创建了一个带有自定义 index.max_ngram_diff 设置的索引：

PUT /my-index
{
  "settings": {
    "index.max_ngram_diff": 2,
    "analysis": {
      "tokenizer": {
        "my_ngram_tokenizer": {
          "type": "ngram",
          "min_gram": 3,
          "max_gram": 5,
          "token_chars": ["letter", "digit"]
        }
      },
      "analyzer": {
        "my_ngram_analyzer": {
          "type": "custom",
          "tokenizer": "my_ngram_tokenizer"
        }
      }
    }
  }
}