Elasticsearch Analyzer 内置分词器( 三 ) _生活百科

转小写默认开启 true.stopwords停顿词过滤默认none未开启 , Defaults to _none_.stopwords_path停顿词文件路径8.3 实验Pattern Analyzer 的实现就是如下
PUT /pattern_example{"settings": {"analysis": {"tokenizer": {"split_on_non_word": {"type":"pattern","pattern":"\\W+"}},"analyzer": {"rebuilt_pattern": {"tokenizer": "split_on_non_word","filter": ["lowercase"]}}}}}9. Language Analyzer提供了如下这么多语言分词器 , 其中 english 也在其中
arabic, armenian, basque, bengali, bulgarian, catalan, czech, dutch, english, finnish, french, galician, german, hindi, hungarian, indonesian, irish, italian, latvian, lithuanian, norwegian, portuguese, romanian, russian, sorani, spanish, swedish, turkish.
GET _analyze{"analyzer": "english","text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."}[ 2, quick, brown, foxes, jumped, over, lazy, dog, bone ]10. Customer Analyzer没啥好说的就是当提供的内置分词器不满足你的需求的时候 ,你可以结合如下3部分

Character Filters : 主要对原文本做处理, 例如去除 html 标签
Tokenizer : 按照规则把文本切分为单词, 也就是分词
Token Filters : 将切分后的单词进行加工处理,小写,删除stopwords 停顿词,增加同义词 , 扩展一些

PUT my_index{"settings": {"analysis": {"analyzer": {"my_custom_analyzer": {"type": "custom","char_filter": ["emoticons"],"tokenizer": "punctuation","filter": ["lowercase","english_stop"]}},"tokenizer": {"punctuation": {"type": "pattern","pattern": "[ .,!?]"}},"char_filter": {"emoticons": {"type": "mapping","mappings": [":) => _happy_",":( => _sad_"]}},"filter": {"english_stop": {"type": "stop","stopwords": "_english_"}}}}}POST my_index/_analyze{"analyzer": "my_custom_analyzer","text":"I'm a :) person, and you?"}[ i'm, _happy_, person, you ]

总结本篇主要介绍了 Elasticsearch 中的一些内置的 Analyzer分词器, 这些内置分词器可能不会常用,但是如果你能好好梳理一下这些内置分词器,一定会对你理解Analyzer 有很大的帮助, 可以帮助你理解 Character Filters , Tokenizer 和 Token Filters 的用处.
有机会再聊聊一些中文分词器如 IKAnalyzer, ICU Analyzer ,Thulac 等等.. 毕竟开发中中文分词器用到更多些