KWAN的解忧杂货铺

# 一.安装 ElasticSearch

# 1.注意事项

注意事项：

  • 内存不能太小，否则会启动失败
  • JDK 版本需要对应，es7 需要 Java 11
  • 不能以 root 用户启动

# 2.安装 java11

#java版本查看
java -version

#下载安装
yum install java-11-openjdk.x86_64 -y

#查看位置
ls -rl $(which java)

#修改环境变量
vim /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.19.0.7-1.0.1.al8.x86_64
export PATH=$PATH:$JAVA_HOME/bin

#使生效
source /etc/profile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

image-20230508233643157

# 3.下载 es

下载地址 (opens new window)

#解压到到/usr/local/目录下
tar -zxvf elasticsearch-7.12.0-linux-x86_64.tar.gz -C /usr/local/

#进入解压后的目录
cd /usr/local/elasticsearch-7.12.0

#创建文件和日志目录(日志文件已经存在)
mkdir data
1
2
3
4
5
6
7
8

修改配置:

vim config/elasticsearch.yml
1
cluster.name: my-application #集群名称
node.name: node-1 #节点名称
path.data: /usr/local/elasticsearch-7.12.0/data
path.logs: /usr/local/elasticsearch-7.12.0/logs
#设置绑定的ip，设置为0.0.0.0以后就可以让任何计算机节点访问到了
network.host: 0.0.0.0
http.port: 9200 #端口
#设置在集群中的所有节点名称，这个节点名称就是之前所修改的，目前是单机，放入一个节点即可
cluster.initial_master_nodes: ["node-1"]
1
2
3
4
5
6
7
8
9

# 4.增加用户

#创建用户
useradd elasticsearch

#用户加密码(elastic-search1)
passwd elasticsearch
1
2
3
4
5

# 5.赋权限

#修改目录权限至新增的elasticsearch用户
chown -R elasticsearch:elasticsearch /usr/local/elasticsearch-7.12.0
1
2

# 6.修改系统配置

  • 修改系统中允许应用最多创建多少文件等的限制权限。Linux 默认来说，一般限制应用最多创建的文件是 65535 个。但是 ES 至少需要 65536 的文件创建权限。
  • 修改系统中允许用户启动的进程开启多少个线程。默认的 Linux 限制 root 用户开启的进程可以开启任意数量的线程，其他用户开启的进程可以开启 1024 个线程。必须修改限制数为 4096+。因为 ES 至少需要 4096 的线程池预备。ES 在 5.x 版本之后，强制要求在 linux 中不能使用 root 用户启动 ES 进程。所以必须使用其他用户启动 ES 进程才可以。
  • Linux 低版本内核为线程分配的内存是 128K。4.x 版本的内核分配的内存更大。如果虚拟机的内存是 1G，最多只能开启 3000+个线程数。至少为虚拟机分配 1.5G 以上的内存。
#修改系统配置
vim /etc/security/limits.conf

# 追加到末尾即可
elasticsearch soft nofile 65536
elasticsearch hard nofile 65536
elasticsearch soft nproc 4096
elasticsearch hard nproc 4096
1
2
3
4
5
6
7
8

# 7.内存权限

vim /etc/sysctl.conf
vm.max_map_count=262144

#使配置生效
/sbin/sysctl -p
1
2
3
4
5

# 8.启动

#切换用户
su elasticsearch

#进入目录
cd /usr/local/elasticsearch-7.12.0

#启动
./bin/elasticsearch
./bin/elasticsearch -d

#启动Kibana
nohup sh /usr/local/kibana/bin/kibana &
1
2
3
4
5
6
7
8
9
10
11
12

# 9.验证

#查询端口信息
netstat -ntlp | grep -E '9200|5601'

#linux访问
curl 127.0.0.1:9200

#浏览器访问
http://47.119.162.180:9200

#查询集群状态
http://47.119.162.180:9200/cluster/health

解释：Status：集群状态。Green所有分片可用。Yellow所有主分片可用。Red主分片不可用，集群不可用。
1
2
3
4
5
6
7
8
9
10
11
12
13

image-20230508234022630

{
  "name": "node-1",
  "cluster_name": "my-application",
  "cluster_uuid": "jFwKZO8cT12BdPtU63m-ew",
  "version": {
    "number": "7.12.0",
    "build_flavor": "default",
    "build_type": "tar",
    "build_hash": "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date": "2021-03-18T06:17:15.410153305Z",
    "build_snapshot": false,
    "lucene_version": "8.8.0",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "You Know, for Search"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# 二.安装 Kibana

# 1.下载 Kibana

下载地址 (opens new window)

选择要下载的版本 尽量和 ES 版本保持一致,版本采用 7.12.0 并选择 linux 版本

# 2.安装

#安装目录
cd  /usr/local

#解压
tar -zxvf kibana-7.12.0-linux-x86_64.tar.gz

#改名
mv kibana-7.12.0-linux-x86_64 kibana
1
2
3
4
5
6
7
8

# 3.修改配置

修改 kibana.yml

cd /usr/local/kibana
vim config/kibana.yml
1
2
server.port: 5601         #kibana端口
server.host: "0.0.0.0"   #所有主机都能访问，或者也可以指定一个ip
elasticsearch.hosts: "http://es服务公网IP:9200"     #配置es的访问地址
kibana.index: ".kibana"
i18n.locale: "zh-CN" #配置项默认是英文,配置成中文的
1
2
3
4
5
#root
chown -R elasticsearch:elasticsearch /usr/local/kibana
1
2

# 4.启动

#切换用户
su elasticsearch

#启动
cd /usr/local/kibana/bin
./kabana

#后台启动
nohup sh /usr/local/kibana/bin/kibana &
1
2
3
4
5
6
7
8
9

# 5.验证

#端口号
netstat -ntlp | grep 5601

#浏览器
http://47.119.162.180:5601/

#控制台
http://47.119.162.180:5601/app/dev_tools#/console
1
2
3
4
5
6
7
8

image-20230509001237739

# 6.测试 SQL

GET _search
{
  "query": {
    "match_all": {}
  }
}

GET /

GET /_cluster/health

GET /_cat/health?v


GET /_cat/indices?v


PUT /demo_index?pretty


DELETE /demo_index?pretty

PUT /book

PUT /book/_doc/1
{
    "id":1,
    "title":"这是一文章",
    "content":"xxxxx",
    "comment":"备注信息",
    "mobile":"13344556677"
}

PUT /book/_doc/2
{
    "id":1,
    "title":"这是一11文章",
    "content":"xxxxx",
    "comment":"备注信息",
    "mobile":"13344556677"
}

GET /book/_doc/1

POST /book/_doc/1/_update
{
  "doc": {
    "title": "这是一333文章"
  }
}


POST /book/_update/1
{
  "doc": {
    "title": "这是一3333333444555文章"
  }
}

DELETE /book/_doc/1



POST /book/_doc/
{
    "id":1,
    "title":"这是一11文章",
    "content":"xxxxx",
    "comment":"备注信息",
    "mobile":"13344556677"
}

GET /book/_doc/1

GET /book/_doc/1?_source_includes=id,title

PUT /read_index/_doc/1/_create
{
    "id":1,
    "title":"这是一11文章",
    "content":"xxxxx",
    "comment":"备注信息",
    "mobile":"13344556677"
}

#插入数据
PUT /test_index/_doc/6
{
  "num": 0
}

#执行脚本
POST /test_index/_doc/6/_update
{
  "script": "ctx._source.num+=1"
}

#查询数据
GET /test_index/_doc/6

GET /test_index/_search
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101

# 7.kibana 控制台

http://47.119.162.180:5601/app/dev_tools#/console
1

image-20230513234111640

# 8.Grok 测试

image-20230513234159116

# 9.格式化

image-20230513234226230

# 10.历史记录

image-20230513234323564

# 11.仪表板

image-20230513234555683

# 三.分词器介绍

# 1.ik 分词器种类

  • standard 分词器
  • ik_max_word 分词器
  • ik_smart 分词器

# 2.standard 分词器

GET /_analyze
{
  "analyzer": "standard",
  "text": "中华人民共和国人民大会堂"
}
1
2
3
4
5
{
  "tokens": [
    {
      "token": "中",
      "start_offset": 0,
      "end_offset": 1,
      "type": "<IDEOGRAPHIC>",
      "position": 0
    },
    {
      "token": "华",
      "start_offset": 1,
      "end_offset": 2,
      "type": "<IDEOGRAPHIC>",
      "position": 1
    },
    {
      "token": "人",
      "start_offset": 2,
      "end_offset": 3,
      "type": "<IDEOGRAPHIC>",
      "position": 2
    },
    {
      "token": "民",
      "start_offset": 3,
      "end_offset": 4,
      "type": "<IDEOGRAPHIC>",
      "position": 3
    },
    {
      "token": "共",
      "start_offset": 4,
      "end_offset": 5,
      "type": "<IDEOGRAPHIC>",
      "position": 4
    },
    {
      "token": "和",
      "start_offset": 5,
      "end_offset": 6,
      "type": "<IDEOGRAPHIC>",
      "position": 5
    },
    {
      "token": "国",
      "start_offset": 6,
      "end_offset": 7,
      "type": "<IDEOGRAPHIC>",
      "position": 6
    },
    {
      "token": "人",
      "start_offset": 7,
      "end_offset": 8,
      "type": "<IDEOGRAPHIC>",
      "position": 7
    },
    {
      "token": "民",
      "start_offset": 8,
      "end_offset": 9,
      "type": "<IDEOGRAPHIC>",
      "position": 8
    },
    {
      "token": "大",
      "start_offset": 9,
      "end_offset": 10,
      "type": "<IDEOGRAPHIC>",
      "position": 9
    },
    {
      "token": "会",
      "start_offset": 10,
      "end_offset": 11,
      "type": "<IDEOGRAPHIC>",
      "position": 10
    },
    {
      "token": "堂",
      "start_offset": 11,
      "end_offset": 12,
      "type": "<IDEOGRAPHIC>",
      "position": 11
    }
  ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88

# 3.ik_max_word

GET /_analyze
{
  "analyzer": "ik_max_word",
  "text": "中华人民共和国人民大会堂"
}
1
2
3
4
5
{
  "tokens": [
    {
      "token": "中华人民共和国",
      "start_offset": 0,
      "end_offset": 7,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "中华人民",
      "start_offset": 0,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 1
    },
    {
      "token": "中华",
      "start_offset": 0,
      "end_offset": 2,
      "type": "CN_WORD",
      "position": 2
    },
    {
      "token": "华人",
      "start_offset": 1,
      "end_offset": 3,
      "type": "CN_WORD",
      "position": 3
    },
    {
      "token": "人民共和国",
      "start_offset": 2,
      "end_offset": 7,
      "type": "CN_WORD",
      "position": 4
    },
    {
      "token": "人民",
      "start_offset": 2,
      "end_offset": 4,
      "type": "CN_WORD",
      "position": 5
    },
    {
      "token": "共和国",
      "start_offset": 4,
      "end_offset": 7,
      "type": "CN_WORD",
      "position": 6
    },
    {
      "token": "共和",
      "start_offset": 4,
      "end_offset": 6,
      "type": "CN_WORD",
      "position": 7
    },
    {
      "token": "国人",
      "start_offset": 6,
      "end_offset": 8,
      "type": "CN_WORD",
      "position": 8
    },
    {
      "token": "人民大会堂",
      "start_offset": 7,
      "end_offset": 12,
      "type": "CN_WORD",
      "position": 9
    },
    {
      "token": "人民大会",
      "start_offset": 7,
      "end_offset": 11,
      "type": "CN_WORD",
      "position": 10
    },
    {
      "token": "人民",
      "start_offset": 7,
      "end_offset": 9,
      "type": "CN_WORD",
      "position": 11
    },
    {
      "token": "大会堂",
      "start_offset": 9,
      "end_offset": 12,
      "type": "CN_WORD",
      "position": 12
    },
    {
      "token": "大会",
      "start_offset": 9,
      "end_offset": 11,
      "type": "CN_WORD",
      "position": 13
    },
    {
      "token": "会堂",
      "start_offset": 10,
      "end_offset": 12,
      "type": "CN_WORD",
      "position": 14
    }
  ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109

# 4.ik_smart

GET /_analyze
{
  "analyzer": "ik_smart",
  "text": "中华人民共和国人民大会堂"
}
1
2
3
4
5
{
  "tokens": [
    {
      "token": "中华人民共和国",
      "start_offset": 0,
      "end_offset": 7,
      "type": "CN_WORD",
      "position": 0
    },
    {
      "token": "人民大会堂",
      "start_offset": 7,
      "end_offset": 12,
      "type": "CN_WORD",
      "position": 1
    }
  ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# 5.mysql 热更新词库

# 1.下载源码

https://github.com/medcl/elasticsearch-analysis-ik/releases

ik 分词器，是个标准的 java maven 工程，直接导入 idea 就可以看到源码

# 2.修改源

  1. org.wltea.analyzer.dic.Dictionary 类，160 行 Dictionary 单例类的初始化方法，在这里需要创建一个我们自定义的线程，并且启动它
  2. org.wltea.analyzer.dic.HotDictReloadThread 类：就是死循环，不断调用 Dictionary.getSingleton().reLoadMainDict()，去重新加载词典
  3. Dictionary 类，399 行：this.loadMySQLExtDict(); 加载 mymsql 字典。
  4. Dictionary 类，609 行：this.loadMySQLStopwordDict();加载 mysql 停用词
  5. config 下 jdbc-reload.properties。mysql 配置文件

# 3.mvn package 打包代码

target\releases\elasticsearch-analysis-ik-7.3.0.zip

# 4.解压缩 ik 压缩包

将 mysql 驱动 jar，放入 ik 的目录下

# 5.修改 jdbc 相关配置

# 6.重启 es

观察日志，日志中就会显示我们打印的那些东西，比如加载了什么配置，加载了什么词语，什么停用词

# 7.在 mysql 中添加词库与停用词

# 8.分词实验，验证热更新生效

GET /_analyze
{
  "analyzer": "ik_smart",
  "text": "喊麦"
}
1
2
3
4
5

# 四.ELK Stack

# 1.何为 ELK?

ELK Stack 是由 Elasticsearch、Logstash 和 Kibana 三个开源工具组成的日志分析平台。它能够高效地处理和分析大量日志数据，帮助用户快速发现、分析和可视化数据。

# 2.Elasticsearch

Elasticsearch 是一个基于 Lucene 的搜索服务器，它提供了一个分布式多用户能力的全文搜索引擎，基于 RESTful web 接口。Elasticsearch 通常用于实现复杂搜索和分析功能，能够处理近实时的搜索请求，并且具有高扩展性和高可靠性。

# 3.Logstash

Logstash 是一个服务器端数据处理管道，能够同时从多个来源采集数据，对数据进行处理（例如过滤、转换等），然后将数据发送到指定的存储库中。Logstash 具有强大的数据收集和处理能力，是 ELK Stack 中的数据传输中枢。

# 4.Kibana

Kibana 是一个开源的分析和可视化平台，它能够让用户在 Elasticsearch 中搜索、查看和交互数据，并通过图表、地图和仪表板等多种形式进行数据可视化。Kibana 的使用非常简单，用户可以通过它快速创建和共享数据视图。

# 5.实现步骤

ELK 的安装通常涉及以下步骤：

  1. 下载并安装 Elasticsearch。
  2. 下载并安装 Logstash，并配置输入、过滤和输出插件。
  3. 下载并安装 Kibana，配置与 Elasticsearch 的连接。

# 1.日志数据收集

在实际应用中，Logstash 可以从多种数据源（如文件、数据库、消息队列等）收集日志数据。通过配置 Logstash 的输入插件，可以轻松地将数据导入到 ELK Stack 中。

# 2.数据处理

收集到的数据可以通过 Logstash 的过滤插件进行处理，如数据清洗、格式转换等。这一步骤确保了数据的质量和一致性。

# 3.数据可视化

在 Kibana 中，用户可以创建多种可视化组件，如柱状图、折线图、饼图等，以直观地展示数据。此外，Kibana 还支持创建仪表板，将多个可视化组件组合在一起，提供全面的数据分析视图。

# 6.日志分析

需求：集中收集分布式服务的日志

  • 逻辑模块程序随时输出日志

  • logstash 收集日志到 es

  • 在 kibana 中查看日志,展示数据

package com.itheima.es;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

import java.util.Random;

/**
 * creste by itheima.itcast
 */
@SpringBootTest
@RunWith(SpringRunner.class)
public class TestLog {
    private static final Logger LOGGER= LoggerFactory.getLogger(TestLog.class);

    @Test
    public void testLog(){
        Random random =new Random();

        while (true){
            int userid=random.nextInt(10);
            LOGGER.info("userId:{},send:{}",userid,"hello world.I am "+userid);
            try {
                Thread.sleep(500);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

# 7.grok 内置类型

写 logstash 配置文件

USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b

POSINT \b(?:[1-9][0-9]*)\b
NONNEGINT \b(?:[0-9]+)\b
WORD \b\w+\b
NOTSPACE \S+
SPACE \s*
DATA .*?
GREEDYDATA .*
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}

# Networking
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})
WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
IPV4 (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])
IP (?:%{IPV6}|%{IPV4})
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
HOST %{HOSTNAME}
IPORHOST (?:%{HOSTNAME}|%{IP})
HOSTPORT %{IPORHOST}:%{POSINT}

# paths
PATH (?:%{UNIXPATH}|%{WINPATH})
UNIXPATH (?>/(?>[\w_%!$@:.,-]+|\\.)*)+
TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))
WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)?
URIHOST %{IPORHOST}(?::%{POSINT:port})?
# uripath comes loosely from RFC1738, but mostly from what Firefox
# doesn't turn into %XX
URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+
#URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]]*
URIPATHPARAM %{URIPATH}(?:%{URIPARAM})?
URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?

# Months: January, Feb, 3, 03, 12, December
MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
MONTHNUM (?:0?[1-9]|1[0-2])
MONTHNUM2 (?:0[1-9]|1[0-2])
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

# Days: Monday, Tue, Thu, etc...
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)

# Years?
YEAR (?>\d\d){1,2}
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])
# '60' is a leap second in most time standards and thus is valid.
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
ISO8601_SECOND (?:%{SECOND}|60)
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
DATE %{DATE_US}|%{DATE_EU}
DATESTAMP %{DATE}[- ]%{TIME}
TZ (?:[PMCE][SD]T|UTC)
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}

# Syslog Dates: Month Day HH:MM:SS
SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
PROG (?:[\w._/%-]+)
SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
SYSLOGHOST %{IPORHOST}
SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>
HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}

# Shortcuts
QS %{QUOTEDSTRING}

# Log formats
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

# Log Levels
LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

# 五.Java API

代码地址 (opens new window)

# 1.document

# 1.pom

<dependencies>
    <!--es客户端-->
    <dependency>
        <groupId>org.elasticsearch.client</groupId>
      	<!-- low:偏向底层。high ：高級封装。足够。-->
        <artifactId>elasticsearch-rest-high-level-client</artifactId>
        <version>7.3.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.elasticsearch</groupId>
                <artifactId>elasticsearch</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>7.3.0</version>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
        <version>2.0.6.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
        <version>2.0.6.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>1.16.10</version>
    </dependency>
</dependencies>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# 2.yaml

spring:
  application:
    name: search-service
kwan:
  elasticsearch:
    hostlist: 47.119.162.180:9200 #多个节点用逗号分隔
1
2
3
4
5
6

# 3.config

@Configuration
public class ElasticsearchConfig {

    @Value("${kwan.elasticsearch.hostlist}")
    private String hostlist;

    @Bean(destroyMethod = "close")
    public RestHighLevelClient restHighLevelClient() {
        String[] split = hostlist.split(",");
        HttpHost[] httpHostsArray = new HttpHost[split.length];
        for (int i = 0; i < split.length; i++) {
            String item = split[i];
            httpHostsArray[i] = new HttpHost(item.split(":")[0], Integer.parseInt(item.split(":")[1]), "http");
        }
        return new RestHighLevelClient(RestClient.builder(httpHostsArray));
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# 4.get

@Slf4j
@SpringBootTest(classes = SearchApplication.class)
@RunWith(SpringRunner.class)
public class TestDocument_01_get {

    @Autowired
    RestHighLevelClient client;

    @Test
    public void testGet() throws IOException {
        //构建请求
        GetRequest getRequest = new GetRequest("test_post", "1");
        //添加可选参数
        String[] includes = new String[]{"id", "comment"};
        String[] excludes = Strings.EMPTY_ARRAY;
        FetchSourceContext fetchSourceContext = new FetchSourceContext(true, includes, excludes);
        getRequest.fetchSourceContext(fetchSourceContext);
        //同步查询
        GetResponse getResponse = client.get(getRequest, RequestOptions.DEFAULT);
        //获取结果
        if (getResponse.isExists()) {
            log.info(getResponse.getId());
            log.info(String.valueOf(getResponse.getVersion()));
            log.info(getResponse.getSourceAsString());//以string获取数据
            log.info(String.valueOf(getResponse.getSourceAsBytes()));////以Bytes获取数据
            log.info(String.valueOf(getResponse.getSourceAsMap()));//以Map获取数据
        } else {
            log.info("数据不存在");
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
@Test
    public void testGet() {
        //构建请求
        GetRequest getRequest = new GetRequest("test_post", "1");
        //添加可选参数
        String[] includes = new String[]{"id", "title"};
        String[] excludes = Strings.EMPTY_ARRAY;
        FetchSourceContext fetchSourceContext = new FetchSourceContext(true, includes, excludes);
        getRequest.fetchSourceContext(fetchSourceContext);
        //设置监听器
        ActionListener<GetResponse> listener = new ActionListener<GetResponse>() {
            //成功时
            public void onResponse(GetResponse getResponse) {
                log.info(getResponse.getId());
                log.info(String.valueOf(getResponse.getVersion()));
                log.info(getResponse.getSourceAsString());
            }

            //失败时
            public void onFailure(Exception e) {
                e.printStackTrace();
                log.info("数据获取异常");
            }
        };
        //异步查询
        client.getAsync(getRequest, RequestOptions.DEFAULT, listener);
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# 5.add

@Test
    public void testAdd() throws IOException {
        //构建请求
        IndexRequest request = new IndexRequest("test_post");
        request.id("5");
        //构建文档数据
        String jsonString = "{\n" +
                "  \"user\":\"tomas\",\n" +
                "  \"postDate\":\"2019-07-18\",\n" +
                "  \"message\":\"trying out es1\"\n" +
                "}";
        request.source(jsonString, XContentType.JSON);
        //同步
        IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);
        //获取结果
        log.info(indexResponse.getIndex());
        log.info(indexResponse.getId());
        log.info(String.valueOf(indexResponse.getResult()));
        if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("CREATE" + result);
        } else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("UPDATED" + result);
        } else {
            log.info("其他操作");
        }
        //获取分片信息
        ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();
        if (shardInfo.getTotal() != shardInfo.getSuccessful()) {
            log.info("处理成功的分片数少于总分片！");
        }
        if (shardInfo.getFailed() > 0) {
            for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {
                String reason = failure.reason();//每一个错误的原因
                log.info(reason);
            }
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
 @Test
    public void testAdd() throws IOException {
        //构建请求
        IndexRequest request = new IndexRequest("test_post");
        request.id("6");
        //构建文档数据
        Map<String, Object> jsonMap = new HashMap<String, Object>();
        jsonMap.put("user", "tomas");
        jsonMap.put("postDate", "2019-07-18");
        jsonMap.put("message", "trying out es1");
        request.source(jsonMap);
        //同步执行
        IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);
        //获取结果
        log.info(indexResponse.getIndex());
        log.info(indexResponse.getId());
        log.info(String.valueOf(indexResponse.getResult()));
        if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("CREATE" + result);
        } else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("UPDATED" + result);
        }
        ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();
        if (shardInfo.getTotal() != shardInfo.getSuccessful()) {
            log.info("处理成功的分片数少于总分片！");
        }
        if (shardInfo.getFailed() > 0) {
            for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {
                String reason = failure.reason();//每一个错误的原因
                log.info(reason);
            }
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
@Test
    public void testAdd() throws IOException {
        //构建请求
        IndexRequest request = new IndexRequest("test_post");
        request.id("7");
        //构建文档数据
        XContentBuilder builder = XContentFactory.jsonBuilder();
        builder.startObject();
        {
            builder.field("user", "tomas");
            builder.field("message", "trying out es1");
            builder.timeField("postDate", "2019-07-18");
        }
        builder.endObject();
        request.source(builder);
        //同步执行
        IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);
        //获取结果
        log.info(indexResponse.getIndex());
        log.info(indexResponse.getId());
        log.info(String.valueOf(indexResponse.getResult()));
        if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("CREATE" + result);
        } else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("UPDATED" + result);
        }
        ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();
        if (shardInfo.getTotal() != shardInfo.getSuccessful()) {
            log.info("处理成功的分片数少于总分片！");
        }
        if (shardInfo.getFailed() > 0) {
            for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {
                String reason = failure.reason();//每一个错误的原因
                log.info(reason);
            }
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
@Test
    public void testAdd() throws IOException {
        //构建请求
        IndexRequest request = new IndexRequest("test_post");
        request.id("9");
        //构建文档数据
        request.source("user", "tomas",
                "message", "trying out es1",
                "postDate", "2019-07-18");
        //同步执行
        IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);
        //获取结果
        log.info(indexResponse.getIndex());
        log.info(indexResponse.getId());
        log.info(String.valueOf(indexResponse.getResult()));
        if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("CREATE" + result);
        } else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("UPDATED" + result);
        }
        ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();
        if (shardInfo.getTotal() != shardInfo.getSuccessful()) {
            log.info("处理成功的分片数少于总分片！");
        }
        if (shardInfo.getFailed() > 0) {
            for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {
                String reason = failure.reason();//每一个错误的原因
                log.info(reason);
            }
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
@Test
    public void testAdd() throws IOException {
        //构建请求
        IndexRequest request = new IndexRequest("test_post");
        request.id("10");
        //构建文档数据
        String jsonString = "{\n" +
                "  \"user\":\"tomas\",\n" +
                "  \"postDate\":\"2019-07-18\",\n" +
                "  \"message\":\"trying out es1\"\n" +
                "}";
        request.source(jsonString, XContentType.JSON);
        //设置超时时间
        request.timeout("1s");
        request.timeout(TimeValue.timeValueSeconds(1));
        //手动维护版本号
        request.version(4);
        request.versionType(VersionType.EXTERNAL);
        //同步执行
        IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);
        //获取结果
        log.info(indexResponse.getIndex());
        log.info(indexResponse.getId());
        log.info(String.valueOf(indexResponse.getResult()));
        if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("CREATE" + result);
        } else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {
            DocWriteResponse.Result result = indexResponse.getResult();
            log.info("UPDATED" + result);
        }
        ReplicationResponse.ShardInfo shardInfo = indexResponse.getShardInfo();
        if (shardInfo.getTotal() != shardInfo.getSuccessful()) {
            log.info("处理成功的分片数少于总分片！");
        }
        if (shardInfo.getFailed() > 0) {
            for (ReplicationResponse.ShardInfo.Failure failure : shardInfo.getFailures()) {
                String reason = failure.reason();//每一个错误的原因
                log.info(reason);
            }
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

# 6.update

@Test
    public void testUpdate() throws IOException {
        //创建请求
        UpdateRequest request = new UpdateRequest("test_post", "5");
        Map<String, Object> jsonMap = new HashMap<>();
        jsonMap.put("user", "tomas Lee");
        request.doc(jsonMap);
        request.timeout("1s");
        request.retryOnConflict(3);//重试次数
        //同步执行
        UpdateResponse updateResponse = client.update(request, RequestOptions.DEFAULT);
        //获取结果
        updateResponse.getId();
        updateResponse.getIndex();
        //判断结果
        if (updateResponse.getResult() == DocWriteResponse.Result.CREATED) {
            DocWriteResponse.Result result = updateResponse.getResult();
            log.info("CREATED：" + result);
        } else if (updateResponse.getResult() == DocWriteResponse.Result.UPDATED) {
            DocWriteResponse.Result result = updateResponse.getResult();
            log.info("UPDATED：" + result);
        } else if (updateResponse.getResult() == DocWriteResponse.Result.DELETED) {
            DocWriteResponse.Result result = updateResponse.getResult();
            log.info("DELETED：" + result);
        } else if (updateResponse.getResult() == DocWriteResponse.Result.NOOP) {
            //没有操作
            DocWriteResponse.Result result = updateResponse.getResult();
            log.info("NOOP：" + result);
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

# 7.delete

@Test
    public void testDelete() throws IOException {
        //创建请求
        DeleteRequest request = new DeleteRequest("test_post", "3");
        //执行
        DeleteResponse deleteResponse = client.delete(request, RequestOptions.DEFAULT);
        //获取结果
        deleteResponse.getId();
        deleteResponse.getIndex();
        DocWriteResponse.Result result = deleteResponse.getResult();
        log.info(result.toString());
    }
1
2
3
4
5
6
7
8
9
10
11
12

# 8.bulk

@Test
    public void testBulk() throws IOException {
        //创建请求
        BulkRequest request = new BulkRequest();
        request.add(new IndexRequest("post").id("1").source(XContentType.JSON, "field", "1"));
        request.add(new IndexRequest("post").id("2").source(XContentType.JSON, "field", "2"));
        request.add(new UpdateRequest("post", "1").doc(XContentType.JSON, "field", "3"));
        request.add(new DeleteRequest("post").id("2"));
        //执行
        BulkResponse bulkResponse = client.bulk(request, RequestOptions.DEFAULT);
        //获取结果
        for (BulkItemResponse itemResponse : bulkResponse) {
            DocWriteResponse response = itemResponse.getResponse();
            switch (itemResponse.getOpType()) {
                case INDEX:
                    IndexResponse indexResponse = (IndexResponse) response;
                    log.info("INDEX:" + indexResponse.getResult());
                    break;
                case CREATE:
                    IndexResponse createResponse = (IndexResponse) response;
                    log.info("CREATE:" + createResponse.getResult());
                    break;
                case UPDATE:
                    UpdateResponse updateResponse = (UpdateResponse) response;
                    log.info("UPDATE:" + updateResponse.getResult());
                    break;
                case DELETE:
                    DeleteResponse deleteResponse = (DeleteResponse) response;
                    log.info("DELETE:" + deleteResponse.getResult());
                    break;
            }
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# 2.index

# 1.创建索引

@SpringBootTest
@RunWith(SpringRunner.class)
public class TestIndex_01_Create {
    @Autowired
    private RestHighLevelClient client;

    //创建索引
    @Test
    public void testCreateIndex() throws IOException {
//        PUT /my_index
//        {
//            "settings": {
//            "number_of_shards": 1,
//                    "number_of_replicas": 1
//        },
//            "mappings": {
//            "properties": {
//                "field1":{
//                    "type": "text"
//                },
//                "field2":{
//                    "type": "text"
//                }
//            }
//        },
//            "aliases": {
//            "default_index": {}
//        }
//        }
        //创建索引对象
        CreateIndexRequest createIndexRequest = new CreateIndexRequest("itheima_book");
        //设置参数
        createIndexRequest.settings(Settings.builder().put("number_of_shards", "1").put("number_of_replicas", "0"));
        //指定映射1
        createIndexRequest.mapping(" {\n" +
                " \t\"properties\": {\n" +
                "            \"name\":{\n" +
                "             \"type\":\"keyword\"\n" +
                "           },\n" +
                "           \"description\": {\n" +
                "              \"type\": \"text\"\n" +
                "           },\n" +
                "            \"price\":{\n" +
                "             \"type\":\"long\"\n" +
                "           },\n" +
                "           \"pic\":{\n" +
                "             \"type\":\"text\",\n" +
                "             \"index\":false\n" +
                "           }\n" +
                " \t}\n" +
                "}", XContentType.JSON);
        //设置别名
        createIndexRequest.alias(new Alias("itheima_index_new"));
        // 额外参数
        //设置超时时间
        createIndexRequest.setTimeout(TimeValue.timeValueMinutes(2));
        //设置主节点超时时间
        createIndexRequest.setMasterTimeout(TimeValue.timeValueMinutes(1));
        //在创建索引API返回响应之前等待的活动分片副本的数量，以int形式表示
        createIndexRequest.waitForActiveShards(ActiveShardCount.from(2));
        createIndexRequest.waitForActiveShards(ActiveShardCount.DEFAULT);
        //操作索引的客户端
        IndicesClient indices = client.indices();
        //执行创建索引库
        CreateIndexResponse createIndexResponse = indices.create(createIndexRequest, RequestOptions.DEFAULT);
        //得到响应
        boolean acknowledged = createIndexResponse.isAcknowledged();
        //得到响应 指示是否在超时前为索引中的每个分片启动了所需数量的碎片副本
        boolean shardsAcknowledged = createIndexResponse.isShardsAcknowledged();
        System.out.println("acknowledged:" + acknowledged);
        System.out.println("shardsAcknowledged:" + shardsAcknowledged);
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
@SpringBootTest
@RunWith(SpringRunner.class)
public class TestIndex_04_Async {
    @Autowired
    private RestHighLevelClient client;

    //创建索引异步方式
    @Test
    public void testCreateIndexAsync() {
        //创建索引对象
        CreateIndexRequest createIndexRequest = new CreateIndexRequest("itheima_book");
        //设置参数
        createIndexRequest.settings(Settings.builder().put("number_of_shards", "1").put("number_of_replicas", "0"));
        //指定映射1
        createIndexRequest.mapping(" {\n" +
                " \t\"properties\": {\n" +
                "            \"name\":{\n" +
                "             \"type\":\"keyword\"\n" +
                "           },\n" +
                "           \"description\": {\n" +
                "              \"type\": \"text\"\n" +
                "           },\n" +
                "            \"price\":{\n" +
                "             \"type\":\"long\"\n" +
                "           },\n" +
                "           \"pic\":{\n" +
                "             \"type\":\"text\",\n" +
                "             \"index\":false\n" +
                "           }\n" +
                " \t}\n" +
                "}", XContentType.JSON);
        //设置别名
        createIndexRequest.alias(new Alias("itheima_index_new"));
        // 额外参数
        //设置超时时间
        createIndexRequest.setTimeout(TimeValue.timeValueMinutes(2));
        //设置主节点超时时间
        createIndexRequest.setMasterTimeout(TimeValue.timeValueMinutes(1));
        //在创建索引API返回响应之前等待的活动分片副本的数量，以int形式表示
        createIndexRequest.waitForActiveShards(ActiveShardCount.from(2));
        createIndexRequest.waitForActiveShards(ActiveShardCount.DEFAULT);
        //操作索引的客户端
        IndicesClient indices = client.indices();
        //执行创建索引库
        ActionListener<CreateIndexResponse> listener = new ActionListener<CreateIndexResponse>() {
            @Override
            public void onResponse(CreateIndexResponse createIndexResponse) {
                //得到响应（全部）
                boolean acknowledged = createIndexResponse.isAcknowledged();
                //得到响应 指示是否在超时前为索引中的每个分片启动了所需数量的碎片副本
                boolean shardsAcknowledged = createIndexResponse.isShardsAcknowledged();
                System.out.println("acknowledged:" + acknowledged);
                System.out.println("shardsAcknowledged:" + shardsAcknowledged);
            }

            @Override
            public void onFailure(Exception e) {
                e.printStackTrace();
            }
        };
        client.indices().createAsync(createIndexRequest, RequestOptions.DEFAULT, listener);
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

# 2.删除索引

@Test
    public void testDeleteIndex() throws IOException {
        //创建删除索引请求
        DeleteIndexRequest deleteIndexRequest = new DeleteIndexRequest("itheima_book");
        //        执行
        AcknowledgedResponse delete = client.indices().delete(deleteIndexRequest, RequestOptions.DEFAULT);
        //得到相应
        boolean acknowledged = delete.isAcknowledged();
        System.out.println("acknowledged:" + acknowledged);

    }
1
2
3
4
5
6
7
8
9
10
11

# 3.是否存在索引

 @Test
    public void testExistIndex() throws IOException {
        GetIndexRequest request = new GetIndexRequest("itheima_book");
        //参数
        request.local(false);//从主节点返回本地索引信息状态
        request.humanReadable(true);//以适合人类的格式返回
        request.includeDefaults(false);//是否返回每个索引的所有默认配置
        boolean exists = client.indices().exists(request, RequestOptions.DEFAULT);
        System.out.println("exists:" + exists);
    }
1
2
3
4
5
6
7
8
9
10

# 4.关闭索引

@Test
    public void testCloseIndex() throws IOException {
        CloseIndexRequest request = new CloseIndexRequest("itheima_book");
        AcknowledgedResponse close = client.indices().close(request, RequestOptions.DEFAULT);
        boolean acknowledged = close.isAcknowledged();
        System.out.println("acknowledged:" + acknowledged);
    }
1
2
3
4
5
6
7

# 5.打开索引

@Test
    public void testOpenIndex() throws IOException {
        OpenIndexRequest request = new OpenIndexRequest("itheima_book");
        OpenIndexResponse open = client.indices().open(request, RequestOptions.DEFAULT);
        boolean acknowledged = open.isAcknowledged();
        System.out.println("acknowledged:" + acknowledged);
    }
1
2
3
4
5
6
7

# 3.search

# 1.all

@SpringBootTest
@RunWith(SpringRunner.class)
public class TestSearch_01_all {

    @Autowired
    RestHighLevelClient client;

    @Test
    public void testSearchAll() throws IOException {
//        GET book/_search
//        {
//            "query": {
//                "match_all": {}
//             }
//        }
        //1构建搜索请求
        SearchRequest searchRequest = new SearchRequest("book");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.matchAllQuery());
        //获取某些字段
        searchSourceBuilder.fetchSource(new String[]{"name"}, new String[]{});
        searchRequest.source(searchSourceBuilder);
        //2执行搜索
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //3获取结果
        SearchHits hits = searchResponse.getHits();
        //数据数据
        SearchHit[] searchHits = hits.getHits();
        System.out.println("--------------------------");
        for (SearchHit hit : searchHits) {
            String id = hit.getId();
            float score = hit.getScore();
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            String name = (String) sourceAsMap.get("name");
            String description = (String) sourceAsMap.get("description");
            Double price = (Double) sourceAsMap.get("price");
            System.out.println("id:" + id);
            System.out.println("score:" + score);
            System.out.println("name:" + name);
            System.out.println("description:" + description);
            System.out.println("price:" + price);
            System.out.println("==========================");
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

# 2.page

@Test
    public void testSearchPage() throws IOException {
//    GET book/_search
//    {
//        "query": {
//          "match_all": {}
//       },
//        "from": 0,
//        "size": 2
//    }
        //1构建搜索请求
        SearchRequest searchRequest = new SearchRequest("book");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.matchAllQuery());
        //第几页
        int page = 1;
        //每页几个
        int size = 2;
        //下标计算
        int from = (page - 1) * size;
        searchSourceBuilder.from(from);
        searchSourceBuilder.size(size);
        searchRequest.source(searchSourceBuilder);
        //2执行搜索
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //3获取结果
        SearchHits hits = searchResponse.getHits();
        //数据数据
        SearchHit[] searchHits = hits.getHits();
        System.out.println("--------------------------");
        for (SearchHit hit : searchHits) {
            String id = hit.getId();
            float score = hit.getScore();
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            String name = (String) sourceAsMap.get("name");
            String description = (String) sourceAsMap.get("description");
            Double price = (Double) sourceAsMap.get("price");
            System.out.println("id:" + id);
            System.out.println("name:" + name);
            System.out.println("description:" + description);
            System.out.println("price:" + price);
            System.out.println("==========================");
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

# 3.ids

@Test
    public void testSearchIds() throws IOException {
//    GET /book/_search
//    {
//        "query": {
//           "ids" : {
//             "values" : ["1", "4", "100"]
//          }
//     }
//    }
        //1构建搜索请求
        SearchRequest searchRequest = new SearchRequest("book");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.idsQuery().addIds("1", "4", "100"));
        searchRequest.source(searchSourceBuilder);
        //2执行搜索
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //3获取结果
        SearchHits hits = searchResponse.getHits();
        //数据数据
        SearchHit[] searchHits = hits.getHits();
        System.out.println("--------------------------");
        for (SearchHit hit : searchHits) {
            String id = hit.getId();
            float score = hit.getScore();
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            String name = (String) sourceAsMap.get("name");
            String description = (String) sourceAsMap.get("description");
            Double price = (Double) sourceAsMap.get("price");
            System.out.println("id:" + id);
            System.out.println("name:" + name);
            System.out.println("description:" + description);
            System.out.println("price:" + price);
            System.out.println("==========================");
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# 4.match

@Test
    public void testSearchMatch() throws IOException {
//    GET /book/_search
//    {
//        "query": {
//           "match": {
//            "description": "java程序员"
//        }
//      }
//    }
        //1构建搜索请求
        SearchRequest searchRequest = new SearchRequest("book");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.matchQuery("description", "java程序员"));
        searchRequest.source(searchSourceBuilder);
        //2执行搜索
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //3获取结果
        SearchHits hits = searchResponse.getHits();
        //数据数据
        SearchHit[] searchHits = hits.getHits();
        System.out.println("--------------------------");
        for (SearchHit hit : searchHits) {
            String id = hit.getId();
            float score = hit.getScore();
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            String name = (String) sourceAsMap.get("name");
            String description = (String) sourceAsMap.get("description");
            Double price = (Double) sourceAsMap.get("price");
            System.out.println("id:" + id);
            System.out.println("name:" + name);
            System.out.println("description:" + description);
            System.out.println("price:" + price);
            System.out.println("==========================");
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# 5.term

@Test
    public void testSearchTerm() throws IOException {
//
//    GET /book/_search
//    {
//        "query": {
//           "term": {
//            "description": "java程序员"
//        }
//      }
//    }
        //1构建搜索请求
        SearchRequest searchRequest = new SearchRequest("book");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.termQuery("description", "java程序员"));
        searchRequest.source(searchSourceBuilder);
        //2执行搜索
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //3获取结果
        SearchHits hits = searchResponse.getHits();
        //数据数据
        SearchHit[] searchHits = hits.getHits();
        System.out.println("--------------------------");
        for (SearchHit hit : searchHits) {
            String id = hit.getId();
            float score = hit.getScore();
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            String name = (String) sourceAsMap.get("name");
            String description = (String) sourceAsMap.get("description");
            Double price = (Double) sourceAsMap.get("price");
            System.out.println("id:" + id);
            System.out.println("name:" + name);
            System.out.println("description:" + description);
            System.out.println("price:" + price);
            System.out.println("==========================");
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# 6.multi_match

@Test
    public void testSearchMultiMatch() throws IOException {
//    GET /book/_search
//    {
//        "query": {
//          "multi_match": {
//            "query": "java程序员",
//            "fields": ["name", "description"]
//        }
//      }
//    }
        //1构建搜索请求
        SearchRequest searchRequest = new SearchRequest("book");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.multiMatchQuery("java程序员", "name", "description"));
        searchRequest.source(searchSourceBuilder);
        //2执行搜索
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //3获取结果
        SearchHits hits = searchResponse.getHits();
        //数据数据
        SearchHit[] searchHits = hits.getHits();
        System.out.println("--------------------------");
        for (SearchHit hit : searchHits) {
            String id = hit.getId();
            float score = hit.getScore();
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            String name = (String) sourceAsMap.get("name");
            String description = (String) sourceAsMap.get("description");
            Double price = (Double) sourceAsMap.get("price");
            System.out.println("id:" + id);
            System.out.println("name:" + name);
            System.out.println("description:" + description);
            System.out.println("price:" + price);
            System.out.println("==========================");
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# 7.bool

 @Test
    public void testSearchBool() throws IOException {
//    GET /book/_search
//    {
//        "query": {
//          "bool": {
//            "must": [
//            {
//                "multi_match": {
//                  "query": "java程序员",
//                  "fields": ["name","description"]
//            }
//            }
//      ],
//            "should": [
//            {
//                "match": {
//                "studymodel": "201001"
//            }
//            }
//      ]
//        }
//    }
//    }
        //1构建搜索请求
        SearchRequest searchRequest = new SearchRequest("book");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        //构建multiMatch请求
        MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("java程序员", "name", "description");
        //构建match请求
        MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("studymodel", "201001");
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        boolQueryBuilder.must(multiMatchQueryBuilder);
        boolQueryBuilder.should(matchQueryBuilder);
        searchSourceBuilder.query(boolQueryBuilder);
        searchRequest.source(searchSourceBuilder);
        //2执行搜索
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //3获取结果
        SearchHits hits = searchResponse.getHits();
        //数据数据
        SearchHit[] searchHits = hits.getHits();
        System.out.println("--------------------------");
        for (SearchHit hit : searchHits) {
            String id = hit.getId();
            float score = hit.getScore();
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            String name = (String) sourceAsMap.get("name");
            String description = (String) sourceAsMap.get("description");
            Double price = (Double) sourceAsMap.get("price");
            System.out.println("id:" + id);
            System.out.println("name:" + name);
            System.out.println("description:" + description);
            System.out.println("price:" + price);
            System.out.println("==========================");
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

# 8.filter

@Test
    public void testSearchFilter() throws IOException {
//    GET /book/_search
//    {
//        "query": {
//          "bool": {
//            "must": [
//            {
//                "multi_match": {
//                "query": "java程序员",
//                        "fields": ["name","description"]
//            }
//            }
//      ],
//            "should": [
//            {
//                "match": {
//                "studymodel": "201001"
//            }
//            }
//          ],
//            "filter": {
//                "range": {
//                    "price": {
//                        "gte": 50,
//                         "lte": 90
//                    }
//                }
//
//            }
//        }
//    }
//    }
        //1构建搜索请求
        SearchRequest searchRequest = new SearchRequest("book");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        //构建multiMatch请求
        MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("java程序员", "name", "description");
        //构建match请求
        MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("studymodel", "201001");
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        boolQueryBuilder.must(multiMatchQueryBuilder);
        boolQueryBuilder.should(matchQueryBuilder);
        boolQueryBuilder.filter(QueryBuilders.rangeQuery("price").gte(50).lte(90));
        searchSourceBuilder.query(boolQueryBuilder);
        searchRequest.source(searchSourceBuilder);
        //2执行搜索
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //3获取结果
        SearchHits hits = searchResponse.getHits();
        //数据数据
        SearchHit[] searchHits = hits.getHits();
        System.out.println("--------------------------");
        for (SearchHit hit : searchHits) {
            String id = hit.getId();
            float score = hit.getScore();
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            String name = (String) sourceAsMap.get("name");
            String description = (String) sourceAsMap.get("description");
            Double price = (Double) sourceAsMap.get("price");
            System.out.println("id:" + id);
            System.out.println("name:" + name);
            System.out.println("description:" + description);
            System.out.println("price:" + price);
            System.out.println("==========================");
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

# 9.sort

@Test
    public void testSearchSort() throws IOException {
//    GET /book/_search
//    {
//        "query": {
//        "bool": {
//            "must": [
//            {
//                "multi_match": {
//                "query": "java程序员",
//                        "fields": ["name","description"]
//            }
//            }
//      ],
//            "should": [
//            {
//                "match": {
//                "studymodel": "201001"
//            }
//            }
//      ],
//            "filter": {
//                "range": {
//                    "price": {
//                        "gte": 50,
//                                "lte": 90
//                    }
//                }
//
//            }
//        }
//    },
//        "sort": [
//        {
//            "price": {
//            "order": "asc"
//        }
//        }
//  ]
//    }
        //1构建搜索请求
        SearchRequest searchRequest = new SearchRequest("book");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        //构建multiMatch请求
        MultiMatchQueryBuilder multiMatchQueryBuilder = QueryBuilders.multiMatchQuery("java程序员", "name", "description");
        //构建match请求
        MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("studymodel", "201001");
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        boolQueryBuilder.must(multiMatchQueryBuilder);
        boolQueryBuilder.should(matchQueryBuilder);
        boolQueryBuilder.filter(QueryBuilders.rangeQuery("price").gte(50).lte(90));
        searchSourceBuilder.query(boolQueryBuilder);
        //按照价格升序
        searchSourceBuilder.sort("price", SortOrder.ASC);
        searchRequest.source(searchSourceBuilder);
        //2执行搜索
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        //3获取结果
        SearchHits hits = searchResponse.getHits();
        //数据数据
        SearchHit[] searchHits = hits.getHits();
        System.out.println("--------------------------");
        for (SearchHit hit : searchHits) {
            String id = hit.getId();
            float score = hit.getScore();
            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
            String name = (String) sourceAsMap.get("name");
            String description = (String) sourceAsMap.get("description");
            Double price = (Double) sourceAsMap.get("price");
            System.out.println("id:" + id);
            System.out.println("name:" + name);
            System.out.println("description:" + description);
            System.out.println("price:" + price);
            System.out.println("==========================");
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

# 4.sql 功能

前提 es 拥有白金版功能:

kibana 中管理-》许可管理 开启白金版试用

导入依赖:

<dependency>
  <groupId>org.elasticsearch.plugin</groupId>
  <artifactId>x-pack-sql-jdbc</artifactId>
  <version>7.3.0</version>
</dependency>

<repositories>
  <repository>
      <id>elastic.co</id>
      <url>https://artifacts.elastic.co/maven</url>
  </repository>
</repositories>
1
2
3
4
5
6
7
8
9
10
11
12

代码:

public class TestJdbc {
    public static void main(String[] args) {
        //1创建连接
        try {
            Connection connection = DriverManager.getConnection("jdbc:es://http://localhost:9200");
            //2创建statement
            Statement statement = connection.createStatement();
            //3执行sql语句
            ResultSet resultSet = statement.executeQuery("SELECT * FROM tvs");
            //4获取结果
            while (resultSet.next()) {
                System.out.println(resultSet.getString(1));
                System.out.println(resultSet.getString(2));
                System.out.println(resultSet.getString(3));
                System.out.println(resultSet.getString(4));
                System.out.println("======================================");
            }
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# 5.学成在线站内搜索模块

# 1.mysql 导入学生数据

/*
 Navicat Premium Data Transfer

 Source Server         : local
 Source Server Type    : MySQL
 Source Server Version : 50721
 Source Host           : localhost:3306
 Source Schema         : xc_course

 Target Server Type    : MySQL
 Target Server Version : 50721
 File Encoding         : 65001

 Date: 10/11/2019 02:50:34
 */
SET
  NAMES utf8mb4;

SET
  FOREIGN_KEY_CHECKS = 0;

-- ----------------------------
-- Table structure for course_pub
-- ----------------------------
DROP TABLE IF EXISTS `course_pub`;

CREATE TABLE `course_pub` (
  `id` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '主键',
  `name` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '课程名称',
  `users` varchar(500) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '适用人群',
  `mt` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '大分类',
  `st` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '小分类',
  `grade` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '课程等级',
  `studymodel` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '学习模式',
  `teachmode` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '教育模式',
  `description` text CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '课程介绍',
  `timestamp` timestamp(0) NOT NULL DEFAULT CURRENT_TIMESTAMP(0) COMMENT '时间戳logstash使用',
  `charge` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '收费规则，对应数据字典',
  `valid` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '有效性，对应数据字典',
  `qq` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '咨询qq',
  `price` float(10, 2) NULL DEFAULT NULL COMMENT '价格',
  `price_old` float(10, 2) NULL DEFAULT NULL COMMENT '原价格',
  `expires` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '过期时间',
  `start_time` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '课程有效期-开始时间',
  `end_time` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '课程有效期-结束时间',
  `pic` varchar(500) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '课程图片',
  `teachplan` text CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '课程计划',
  `pub_time` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '发布时间',
  PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;

-- ----------------------------
-- Records of course_pub
-- ----------------------------
INSERT INTO
  `course_pub`
VALUES
  (
    '297e7c7c62b888f00162b8a7dec20000',
    'test_java基础33',
    'b1',
    '1-3',
    '1-3-3',
    '200002',
    '201002',
    NULL,
    'java 从入门到删库跑路',
    '2019-10-28 11:26:25',
    '203002',
    '204002',
    '32432',
    NULL,
    NULL,
    NULL,
    NULL,
    NULL,
    'group1/M00/00/00/wKgZhV2tIgiAaYVMAAA2T52Dthw246.jpg',
    '{\"children\":[{\"children\":[],\"id\":\"40288f9b6e0c10d8016e0c37f72a0000\",\"pname\":\"1\"},{\"children\":[{\"id\":\"40288581632b593e01632bd53ff10001\",\"mediaFileoriginalname\":\"solr.avi\",\"mediaId\":\"5fbb79a2016c0eb609ecd0cd3dc48016\",\"pname\":\"Hello World\"},{\"id\":\"40288f9b6e106273016e106485f30000\",\"mediaFileoriginalname\":\"lucene.avi\",\"mediaId\":\"c5c75d70f382e6016d2f506d134eee11\",\"pname\":\"java基础\"}],\"id\":\"40288581632b593e01632bd4ec360000\",\"pname\":\"程序入门\"},{\"children\":[{\"id\":\"40288f9b6dce18e3016dcef16d860001\",\"mediaFileoriginalname\":\"solr.avi\",\"mediaId\":\"5fbb79a2016c0eb609ecd0cd3dc48016\",\"pname\":\"三级节点\"}],\"id\":\"40288f9b6dce18e3016dcef12a1d0000\",\"pname\":\"二级节点\"},{\"children\":[{\"id\":\"40288c9a6ca3968e016ca417fa8d0001\",\"mediaFileoriginalname\":\"lucene.avi\",\"mediaId\":\"c5c75d70f382e6016d2f506d134eee11\",\"pname\":\"test04-01\"}],\"id\":\"40288c9a6ca3968e016ca417b4a50000\",\"pname\":\"test04\"},{\"children\":[{\"id\":\"40288581632b593e01632bd5d31f0003\",\"mediaFileoriginalname\":\"solr.avi\",\"mediaId\":\"5fbb79a2016c0eb609ecd0cd3dc48016\",\"pname\":\"表达式\"},{\"id\":\"40288581632b593e01632bd606480004\",\"pname\":\"逻辑运算\"}],\"id\":\"40288581632b593e01632bd597810002\",\"pname\":\"编程基础\"},{\"children\":[{\"id\":\"402881e764034e4301640351f3d70003\",\"pname\":\"一切皆为对象\"}],\"id\":\"402881e764034e430164035091a00002\",\"pname\":\"面向对象\"},{\"children\":[{\"id\":\"402899816ad8457c016ad9282a330001\",\"pname\":\"test06\"}],\"id\":\"402899816ad8457c016ad927ba540000\",\"pname\":\"test05\"}],\"id\":\"4028858162bec7f30162becad8590000\",\"pname\":\"test_java基础33\"}',
    '2019-10-28 11:26:24'
  );

INSERT INTO
  `course_pub`
VALUES
  (
    '297e7c7c62b888f00162b8a965510001',
    'test_java基础node',
    'test_java基础',
    '1-3',
    '1-3-2',
    '200001',
    '201001',
    NULL,
    'test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2test_java基础2',
    '2019-10-24 16:26:34',
    '203001',
    '204001',
    '443242',
    NULL,
    NULL,
    NULL,
    NULL,
    NULL,
    NULL,
    '{\"children\":[{\"children\":[{\"id\":\"402881e66417407b01641744fc650001\",\"pname\":\"入门程序\"}],\"id\":\"402881e66417407b01641744afc30000\",\"pname\":\"基础知识\"},{\"children\":[],\"id\":\"4028858162e5d6e00162e5e0727d0001\",\"pname\":\"java基础语法\"},{\"children\":[{\"id\":\"4028d0866b158241016b502433d60002\",\"pname\":\"第二节\"}],\"id\":\"4028d0866b158241016b5023f51e0001\",\"pname\":\"第二章\"}],\"id\":\"4028858162e5d6e00162e5e0227b0000\",\"pname\":\"test_java基础2\"}',
    '2019-10-24 16:26:33'
  );

SET
  FOREIGN_KEY_CHECKS = 1;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

# 2.创建索引 xc_course

# 3.创建映射

PUT /xc_course
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "description" : {
                "analyzer" : "ik_max_word",
                "search_analyzer": "ik_smart",
               "type" : "text"
            },
            "grade" : {
               "type" : "keyword"
            },
            "id" : {
               "type" : "keyword"
            },
            "mt" : {
               "type" : "keyword"
            },
            "name" : {
                "analyzer" : "ik_max_word",
           "search_analyzer": "ik_smart",
               "type" : "text"
            },
            "users" : {
               "index" : false,
               "type" : "text"
            },
            "charge" : {
               "type" : "keyword"
            },
            "valid" : {
               "type" : "keyword"
            },
            "pic" : {
               "index" : false,
               "type" : "keyword"
            },
            "qq" : {
               "index" : false,
               "type" : "keyword"
            },
            "price" : {
               "type" : "float"
            },
            "price_old" : {
               "type" : "float"
            },
            "st" : {
               "type" : "keyword"
            },
            "status" : {
               "type" : "keyword"
            },
            "studymodel" : {
               "type" : "keyword"
            },
            "teachmode" : {
               "type" : "keyword"
            },
            "teachplan" : {
                "analyzer" : "ik_max_word",
           "search_analyzer": "ik_smart",
               "type" : "text"
            },
           "expires" : {
               "type" : "date",
            "format": "yyyy-MM-dd HH:mm:ss"
            },
            "pub_time" : {
               "type" : "date",
             "format": "yyyy-MM-dd HH:mm:ss"
            },
            "start_time" : {
               "type" : "date",
           "format": "yyyy-MM-dd HH:mm:ss"
            },
          "end_time" : {
                 "type" : "date",
           "format": "yyyy-MM-dd HH:mm:ss"
            }
    }
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

# 4.logstash 创建模板文件

Logstash 的工作是从 MySQL 中读取数据，向 ES 中创建索引，这里需要提前创建 mapping 的模板文件以便 logstash 使用。

在 logstach 的 config 目录创建 xc_course_template.json，内容如下：

{
  "mappings": {
    "doc": {
      "properties": {
        "charge": {
          "type": "keyword"
        },
        "description": {
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_smart",
          "type": "text"
        },
        "end_time": {
          "format": "yyyy-MM-dd HH:mm:ss",
          "type": "date"
        },
        "expires": {
          "format": "yyyy-MM-dd HH:mm:ss",
          "type": "date"
        },
        "grade": {
          "type": "keyword"
        },
        "id": {
          "type": "keyword"
        },
        "mt": {
          "type": "keyword"
        },
        "name": {
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_smart",
          "type": "text"
        },
        "pic": {
          "index": false,
          "type": "keyword"
        },
        "price": {
          "type": "float"
        },
        "price_old": {
          "type": "float"
        },
        "pub_time": {
          "format": "yyyy-MM-dd HH:mm:ss",
          "type": "date"
        },
        "qq": {
          "index": false,
          "type": "keyword"
        },
        "st": {
          "type": "keyword"
        },
        "start_time": {
          "format": "yyyy-MM-dd HH:mm:ss",
          "type": "date"
        },
        "status": {
          "type": "keyword"
        },
        "studymodel": {
          "type": "keyword"
        },
        "teachmode": {
          "type": "keyword"
        },
        "teachplan": {
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_smart",
          "type": "text"
        },
        "users": {
          "index": false,
          "type": "text"
        },
        "valid": {
          "type": "keyword"
        }
      }
    }
  },
  "template": "xc_course"
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85

# 5.logstash 配置 mysql.conf

ES 采用 UTC 时区问题

ES 采用 UTC 时区，比北京时间早 8 小时，所以 ES 读取数据时让最后更新时间加 8 小时

where timestamp > date_add(:sql_last_value,INTERVAL 8 HOUR)

logstash 每个执行完成会在/config/logstash_metadata 记录执行时间下次以此时间为基准进行增量同步数据到索引库。

# 6.启动

.\logstash.bat -f ..\config\mysql.conf
1

# 7.后端代码

代码地址 (opens new window)

# 7.1.Controller
@RestController
@RequestMapping("/search/course")
public class EsCourseController {

    @Autowired
    private EsCourseServiceImpl esCourseServiceImpl;

    @GetMapping(value = "/list/{page}/{size}")
    public QueryResponseResult<CoursePub> list(@PathVariable("page") int page
            , @PathVariable("size") int size, CourseSearchParam courseSearchParam) {
        return esCourseServiceImpl.list(page, size, courseSearchParam);
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
# 7.2.service
@Service
public class EsCourseServiceImpl implements EsCourseService {

    @Value("${kwan.course.source_field}")
    private String source_field;
    @Autowired
    private RestHighLevelClient restHighLevelClient;

    /**
     * 课程搜索
     *
     * @param page
     * @param size
     * @param courseSearchParam
     * @return
     */
    @Override
    public QueryResponseResult<CoursePub> list(int page, int size, CourseSearchParam courseSearchParam) {
        if (courseSearchParam == null) {
            courseSearchParam = new CourseSearchParam();
        }
        //1创建搜索请求对象
        SearchRequest searchRequest = new SearchRequest("xc_course");
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        //过虑源字段
        String[] source_field_array = source_field.split(",");
        searchSourceBuilder.fetchSource(source_field_array, new String[]{});
        //创建布尔查询对象
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        //搜索条件
        //根据关键字搜索
        if (StringUtils.isNotEmpty(courseSearchParam.getKeyword())) {
            MultiMatchQueryBuilder multiMatchQueryBuilder =
                    QueryBuilders
                            .multiMatchQuery(courseSearchParam.getKeyword(), "name", "description", "teachplan")
                            .minimumShouldMatch("70%")
                            .field("name", 10);
            boolQueryBuilder.must(multiMatchQueryBuilder);
        }
        if (StringUtils.isNotEmpty(courseSearchParam.getMt())) {
            //根据一级分类
            boolQueryBuilder.filter(QueryBuilders.termQuery("mt", courseSearchParam.getMt()));
        }
        if (StringUtils.isNotEmpty(courseSearchParam.getSt())) {
            //根据二级分类
            boolQueryBuilder.filter(QueryBuilders.termQuery("st", courseSearchParam.getSt()));
        }
        if (StringUtils.isNotEmpty(courseSearchParam.getGrade())) {
            //根据难度等级
            boolQueryBuilder.filter(QueryBuilders.termQuery("grade", courseSearchParam.getGrade()));
        }
        //设置boolQueryBuilder到searchSourceBuilder
        searchSourceBuilder.query(boolQueryBuilder);
        //设置分页参数
        if (page <= 0) {
            page = 1;
        }
        if (size <= 0) {
            size = 12;
        }
        //起始记录下标
        int from = (page - 1) * size;
        searchSourceBuilder.from(from);
        searchSourceBuilder.size(size);
        //设置高亮
        HighlightBuilder highlightBuilder = new HighlightBuilder();
        highlightBuilder.preTags("<font class='eslight'>");
        highlightBuilder.postTags("</font>");
        //设置高亮字段
        highlightBuilder.fields().add(new HighlightBuilder.Field("name"));
        searchSourceBuilder.highlighter(highlightBuilder);
        searchRequest.source(searchSourceBuilder);
        QueryResult<CoursePub> queryResult = new QueryResult();
        List<CoursePub> list = new ArrayList<>();
        try {
            //2执行搜索
            SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            //3获取响应结果
            SearchHits hits = searchResponse.getHits();
            long totalHits = hits.getTotalHits().value;
            //匹配的总记录数
            queryResult.setTotal(totalHits);
            SearchHit[] searchHits = hits.getHits();
            for (SearchHit hit : searchHits) {
                CoursePub coursePub = new CoursePub();
                //源文档
                Map<String, Object> sourceAsMap = hit.getSourceAsMap();
                //取出id
                String id = (String) sourceAsMap.get("id");
                coursePub.setId(id);
                //取出name
                String name = (String) sourceAsMap.get("name");
                //取出高亮字段name
                Map<String, HighlightField> highlightFields = hit.getHighlightFields();
                if (highlightFields != null) {
                    HighlightField highlightFieldName = highlightFields.get("name");
                    if (highlightFieldName != null) {
                        Text[] fragments = highlightFieldName.fragments();
                        StringBuffer stringBuffer = new StringBuffer();
                        for (Text text : fragments) {
                            stringBuffer.append(text);
                        }
                        name = stringBuffer.toString();
                    }
                }
                coursePub.setName(name);
                //图片
                String pic = (String) sourceAsMap.get("pic");
                coursePub.setPic(pic);
                //价格
                Double price = null;
                try {
                    if (sourceAsMap.get("price") != null) {
                        price = (Double) sourceAsMap.get("price");
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
                coursePub.setPrice(price);
                //旧价格
                Double price_old = null;
                try {
                    if (sourceAsMap.get("price_old") != null) {
                        price_old = (Double) sourceAsMap.get("price_old");
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
                coursePub.setPrice_old(price_old);
                //将coursePub对象放入list
                list.add(coursePub);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        queryResult.setList(list);
        QueryResponseResult<CoursePub> queryResponseResult = new QueryResponseResult<>(CommonCode.SUCCESS, queryResult);
        return queryResponseResult;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
上次更新: 10/29/2024, 10:27:50 AM

← 05-ElasticSearch查询 →