路径词元分词器

路径词元分词器 #

路径(path_hierarchy)词元分词器可以把文件路径(或类似的层次结构)的文本按照各个层级分解为词元。当处理诸如文件路径、网址或其他有分隔符的层次结构数据时,这个词元生成器特别有用。

参考样例 #

以下示例请求创建一个名为 my_index 的新索引,并配置一个使用路径词元生成器的分词器:

PUT /my_index
{
  "settings": {
    "analysis": {
      "tokenizer": {
        "my_path_tokenizer": {
          "type": "path_hierarchy"
        }
      },
      "analyzer": {
        "my_path_analyzer": {
          "type": "custom",
          "tokenizer": "my_path_tokenizer"
        }
      }
    }
  }
}

生成的词元 #

使用以下请求来检查使用该分词器生成的词元:

POST /my_index/_analyze
{
  "analyzer": "my_path_analyzer",
  "text": "/users/john/documents/report.txt"
}

返回内容包含产生的词元

{
  "tokens": [
    {
      "token": "/users",
      "start_offset": 0,
      "end_offset": 6,
      "type": "word",
      "position": 0
    },
    {
      "token": "/users/john",
      "start_offset": 0,
      "end_offset": 11,
      "type": "word",
      "position": 0
    },
    {
      "token": "/users/john/documents",
      "start_offset": 0,
      "end_offset": 21,
      "type": "word",
      "position": 0
    },
    {
      "token": "/users/john/documents/report.txt",
      "start_offset": 0,
      "end_offset": 32,
      "type": "word",
      "position": 0
    }
  ]
}

参数说明 #

路径词元生成器可以使用以下参数进行配置。

参数必需/可选数据类型描述
delimiter可选字符串指定用于分隔路径组件的字符。默认值为 /
replacement可选字符串配置用于替换词元中分隔符的字符。默认值为 /
buffer_size可选整数指定缓冲区大小。默认值为 1024
reverse可选布尔值若为 true,则按逆序生成词元。默认值为 false
skip可选整数指定分词时要跳过的初始词元(层级)数量。默认值为 0

使用分隔符和替换参数 #

以下示例请求配置了自定义的分隔符(delimiter)和替换(replacement)参数:

PUT /my_index
{
  "settings": {
    "analysis": {
      "tokenizer": {
        "my_path_tokenizer": {
          "type": "path_hierarchy",
          "delimiter": "\\",
          "replacement": "/"
        }
      },
      "analyzer": {
        "my_path_analyzer": {
          "type": "custom",
          "tokenizer": "my_path_tokenizer"
        }
      }
    }
  }
}

使用以下请求来检查使用该分词器生成的词元:

POST /my_index/_analyze
{
  "analyzer": "my_path_analyzer",
  "text": "C:\\users\\john\\documents\\report.txt"
}

返回内容包含产生的词元

{
  "tokens": [
    {
      "token": "C:",
      "start_offset": 0,
      "end_offset": 2,
      "type": "word",
      "position": 0
    },
    {
      "token": "C:/users",
      "start_offset": 0,
      "end_offset": 8,
      "type": "word",
      "position": 0
    },
    {
      "token": "C:/users/john",
      "start_offset": 0,
      "end_offset": 13,
      "type": "word",
      "position": 0
    },
    {
      "token": "C:/users/john/documents",
      "start_offset": 0,
      "end_offset": 23,
      "type": "word",
      "position": 0
    },
    {
      "token": "C:/users/john/documents/report.txt",
      "start_offset": 0,
      "end_offset": 34,
      "type": "word",
      "position": 0
    }
  ]
}