zifuy 的个人笔记

环境准备

说明：

Elasticsearch在 8.0及以后的版本中默认开启 https以及 TLS功能，这样客户端连接必须使用证书才可以建立连接。且UI界面改动比较大，单纯就ELK系统来说显得过于繁重。日志系统也没必要配置https功能，开启节点间的TLS传输即可。7.17.7版本也更方便配置。

1、将主机名映射写入各节点的hosts文件

cat>>/etc/hosts<<EOF
10.13.1.2 es.node.01
10.13.1.10 es.node.02
10.13.3.108 es.node.03
10.13.1.9 es.master.01
10.13.15.13 es.master.02
10.13.15.24 es.master.03
EOF

2、创建数据和日志目录并授权

mkdir -p /data/elasticsearch/{logs,data}

chown -R elasticsearch.elasticsearch /data/elasticsearch

部署服务

3、配置yum源

cat>/etc/yum.repos.d/elasticsearch7.repo<<EOF
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF

4、先卸载历史版本(如有)

systemctl stop elasticsearch

yum remove -y elasticsearch-7.17.7

rm -rf /etc/elasticsearch/
rm -rf /data/elasticsearch/data/*
rm -rf /usr/share/elasticsearch
rm -rf /var/lib/elasticsearch

5、部署

下载后rpm后

rpm --install elasticsearch-7.17.7-x86_64.rpm

或者yum

yum install --enablerepo=elasticsearch elasticsearch -y

6、开启加密节点间加密传输

/usr/share/elasticsearch/bin/elasticsearch-certutil ca

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

将 elastic-certificates.p12 文件拷贝到所有节点的配置目录下

cp /usr/share/elasticsearch/elastic-certificates.p12 /etc/elasticsearch/

授权

chmod 777 /etc/elasticsearch/elastic-certificates.p12

第一个master节点配置文件

第一个节点不需要配置 node.roles 配置项，因为第一个节点同时具备master、data角色，节点启动后集群才属于green状态。如果在第一个节点配置 node.roles 配置项，则需要在有data角色节点成功加入后集群才处于green状态

cat >/etc/elasticsearch/elasticsearch.yml<<EOF
cluster.name: elastic-cluster

#node.roles: [ master ]

node.name: ${HOSTNAME}

network.host: ${HOSTNAME}

path.data: /data/elasticsearch/data

path.logs: /data/elasticsearch/logs

cluster.initial_master_nodes: [ "es.master.01" ]

xpack.security.enabled:  true   
xpack.security.transport.ssl.enabled:  true   
xpack.security.transport.ssl.verification_mode: certificate   
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12   
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

http.host: 0.0.0.0

thread_pool.get.queue_size: 10000
thread_pool.write.queue_size: 10000
EOF

启动

systemctl start elasticsearch

查看日志

tail -100f /data/elasticsearch/logs/elastic-cluster.log

其他master节点配置文件

cat >/etc/elasticsearch/elasticsearch.yml<<EOF
cluster.name: elastic-cluster

node.roles: [ master ]

node.name: ${HOSTNAME}

network.host: ${HOSTNAME}

path.data: /data/elasticsearch/data

path.logs: /data/elasticsearch/logs

discovery.seed_hosts: [ "es.master.01","es.master.02","es.master.03" ]

xpack.security.enabled:  true   
xpack.security.transport.ssl.enabled:  true   
xpack.security.transport.ssl.verification_mode: certificate   
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12   
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

http.host: 0.0.0.0

thread_pool.get.queue_size: 10000
thread_pool.write.queue_size: 10000
EOF

启动

systemctl start elasticsearch

查看日志

tail -100f /data/elasticsearch/logs/elastic-cluster.log

data节点配置文件

cat >/etc/elasticsearch/elasticsearch.yml<<EOF
cluster.name: elastic-cluster

node.roles: [ data,transform,ingest ]

#node.attr.temperature: hot

node.name: ${HOSTNAME}

network.host: ${HOSTNAME}

path.data: /data/elasticsearch/data

path.logs: /data/elasticsearch/logs

discovery.seed_hosts: [ "es.master.01","es.master.02","es.master.03" ]

xpack.security.enabled:  true   
xpack.security.transport.ssl.enabled:  true   
xpack.security.transport.ssl.verification_mode: certificate   
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12   
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

http.host: 0.0.0.0

thread_pool.get.queue_size: 10000
thread_pool.write.queue_size: 10000
EOF

#配置node.attr.temperature参数后，可以在后面的索引模版中指定索引的 routing.allocation.require.temperature 配置项，使索引创建在指定节点，且后面生命周期也可以使用该标签进行迁移。

启动

systemctl start elasticsearch

查看日志

tail -100f /data/elasticsearch/logs/elastic-cluster.log

修改第一个master节点的配置文件

1、将 cluster.initial_master_nodes 配置项替换为 discovery.seed_hosts 配置项

2、增加 node.roles: [ master ] 配置项

3、重启第一个master节点

因为cluster.initial_master_nodes 该配置项只需要在master节点第一次初始化启动时使用到。如不去除替换，再次重启节点后会报错。集群在有其他节点接入后 可以将第一个用于专属master的节点角色修改为只有 master角色

验证集群状态

curl -u elastic:ukppvBjp7Ed 'http://10.13.1.9:9200/_cluster/health?pretty'

curl -u elastic:ukppvBjp7Ed 'http://10.13.1.9:9200/_cat/nodes?v'

配置密码

自动生成密码

/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

手动生成密码

/usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

kibana接入

cat >/etc/yum.repos.d/kibana.repo<<EOF
[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

yum install kibana

rpm方式部署kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.13.4-x86_64.rpm

shasum -a 512 kibana-7.13.4-x86_64.rpm 

rpm --install kibana-7.13.4-x86_64.rpm

配置文件

server.port: 5601
server.host: "10.13.1.2"
elasticsearch.hosts: ["http://127.0.0.1:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "oJUzWagO7ukppvBjp7Ed"
i18n.locale: "zh-CN"
logging.dest: /var/log/kibana/kibana.log

Logstash接入

安装

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

echo >/etc/yum.repos.d/logstash.repo<<EOF
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

yum install logstash

配置文件

/etc/logstash/conf.d/logstash.conf

这里只列出 output 到 es 部分的配置

elasticsearch {
  hosts => ["https://10.13.1.9:9200"]
  index => "%{[kubernetes][container][name]}-perf-%{index_date}"
  user => "elastic"
  password => "fyBrlfU7PHgUOtzu8sIa"
}

logstash.yml 配置

config.reload.automatic: true
config.reload.interval: 3s
pipeline.batch.size: 2000
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: oJUzWagO7ukppvBjp7Ed
xpack.monitoring.elasticsearch.hosts: ["127.0.0.1:9200"]

启动服务

systemctl start logstash

查看日志

tail -100f /var/log/logstash/logstash-plain.log

升级白金版

白金版只有7.13.4及以下的版本可以成功升级，所以这里以7.13.4版本为例 7.13.4版本的ES安装方式同7.17.7相同。

解压 /usr/share/elasticsearch/modules/x-pack-core/x-pack-core-7.13.4.jar 文件

创建 x-pack-core 目录 拷贝x-pack-core-7.13.4.jar文件

mkdir /root/x-pack-core

mv /usr/share/elasticsearch/modules/x-pack-core/x-pack-core-7.13.4.jar /root/x-pack-core

cd /root/x-pack-core

解压

/usr/share/elasticsearch/jdk/bin/jar cvf x-pack-core-7.13.4.jar

替换文件

cp /root/classfile/LicenseVerifier.class org/elasticsearch/license/

cp /root/classfile/XPackBuild.class org/elasticsearch/xpack/core/

重新打包成jar文件

/usr/share/elasticsearch/jdk/bin/jar cvf x-pack-core-7.13.4.jar .

将重新打好的 x-pack-core-7.13.4.jar 文件拷贝到 /usr/share/elasticsearch/modules/x-pack-core/

cp /root/x-pack-core/x-pack-core-7.13.4.jar /usr/share/elasticsearch/modules/x-pack-core/

导入licens

curl -XPUT -u elastic:123456 'http://10.13.1.9:9200/_xpack/license' -H "Content-Type: application/json" -d @license1.json

查看licens

curl -XGET -u elastic:bZHXdGYbwl3xjMoyfreV  http://10.13.1.9:9200/_license