沪ICP备2021032517号-1

Elasticsearch & API

  |   0 评论   |   0 浏览

kibana索引模式获取

curl -s  --fail --user elastic:123456 -X GET "http://kibana.tencentelasticsearch.com:5601/s/app-test/api/saved_objects/_find?type=index-pattern&fields=title&fields=type&per_page=1000" | jq -r '.saved_objects[].attributes.title'

sed&awk配合处理前后多余字符

sed -r 's/^.{3}//'  #删除前三个字符

awk '{sub(/.{3}$/,"")}1' #删除最后三个字符

索引节点分片最大数量限制

PUT {index_name}/_settings
 {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "total_shards_per_node": "3"
          }
        }
      }
    }
 }

开启批量写入定向路由

PUT /*-prod-2023.01.07/_settings?pretty
{

 "index.bulk_routing.enabled": true

}

可动态配置，日志批量写入场景开启该功能。cpu下降20%。写入能力提升25%

索引刷新间隔时间

PUT /_settings
{"index":{"refresh_interval":"20s"}}

PUT /bt-vehicle-coredata-service-prod-2023.01.12/_settings
{
  "refresh_interval":"120s"
}

段合并相关设置

PUT /_settings
{
  "index.merge.scheduler.max_thread_count": 4,
  "merge.scheduler.auto_throttle": "true",
  "merge.scheduler.max_merge_count": "9"
}

热线程相关API

GET /_nodes/1635773435000735832/hot_threads

GET /_nodes/hot_threads

GET /_cat/tasks/?v

GET /_cat/thread_pool/?v&h=id,name,active,rejected,completed,size,type&pretty&s=type

禁用分片复制

curl -X PUT "192.168.200.200:9200/_cluster/settings" -H 'Content-Type: application/json' -d'  
{  
 "persistent": {  
 "cluster.routing.allocation.enable": "none"  
 }  
}

启用分片复制

curl -X PUT "192.168.200.200:9200/_cluster/settings" -H 'Content-Type: application/json' -d'  
{  
 "persistent": {  
 "cluster.routing.allocation.enable": "all"  
 }  
}

动态设置分片复制并发

curl -X PUT -u elastic:123456 "10.20.11.2:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
    "persistent": {
        "cluster.routing.allocation.node_concurrent_recoveries": 64
    }
}

集群最大分片数调整

curl -X PUT -u elastic:123456"10.20.11.2:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
    "persistent" : {
        "cluster.max_shards_per_node" : "5000"
    }
}

磁盘占用比限制

PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.disk.watermark.low": "96%",
"cluster.routing.allocation.disk.watermark.high": "97%",
"cluster.routing.allocation.disk.watermark.flood_stage": "98%"
}
}

达到 low水平会的节点会停止分配新的分片

达到 high水平会调度分片到其他节点

达到 flood_stage水平的节点拒绝新数据写入

磁盘使用量查看

GET /_nodes/stats?human&filter_path=nodes.*.fs

调整单个索引副本数

curl -H "Content-Type: application/json" -XPUT -u elastic:123456 '10.10.4.7:9200/index_name/_settings' -d '{"index": {"number_of_replicas": "0"}}'

curl -XPUT 'node3:9205/my_index/_settings' -d '{
    "index": { 
      "number_of_replicas": "0"  
     }
}'

设置 fielddata.limit

PUT _cluster/settings  
{  
 "persistent" : {  
 "indices.breaker.fielddata.limit" : "40%"   
 }  
}

设置 indices.breaker.request.limit

PUT _cluster/settings  
{  
 "persistent" : {  
  
 "indices.breaker.request.limit" : "60%"   
  
 }  
}

清除所有索引缓存

POST _cache/clear

查看集群状态

GET _cluster/stats

查看集群健康状态

GET _cluster/health

curl -XGET  -u elastic:123456  'http://10.10.11.16:9200/_cluster/health?pretty'

删除单个索引

DELETE /bt-zt-gateway-xiandou-test-2020.02.25

或linux执行命令删除

curl -XDELETE http://localhost:9200/index-name

按日期删除索引

DELETE /*-2020.05.18*

curl -XDELETE -u elastic:123456 'http://10.20.11.2:9200/*-2020.04.09*'

查看索引数据及大小

GET /_cat/indices

查看单个索引分片信息

GET /_cat/shards/srm-status-dispatcher-prod-2020.03.22

查看所有索引分片、UNASSINGNED 分片

GET /_cat/shards/  

GET /_cat/shards/UNASSINGNED?v    #查看 UNASSINGNED 状态分片

curl -XGET 'http://10.10.12.101:9200/_cat/shards' | grep UN

image.png

curl -XGET 'http://10.20.12.101:9200/_cat/allocation?v' | grep co

image.png

打开关闭索引

POST /bt-web-h5-prod-2020.03.22/_open?pretty  

POST /bt-web-h5-prod-2020.03.22/_close?pretty

索引只读问题处理

curl -XPUT -u elastic:123456 -H "Content-Type: application/json" http://10.20.11.2:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

或者

PUT _settings
{
  "index": {
    "blocks": {
      "read_only_allow_delete": "false"
    }
  }
}

或者

PUT /twitter/_settings
{
“index.blocks.read_only_allow_delete”: null
}

curl -XPUT -H 'Content-Type: application/json' http://106.14.46.249:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

master节点查看

GET /_cat/master?v

线程池信息查看

GET /_cat/thread_pool/write?v&h=node_name,name,active,rejected,completed

或者

GET /_cat/thread_pool/write?v

node_name        name  active queue rejected
es-master-01     write      0     0        0
xd-tsp-log-es-04 write      6     0        0
es-master-02     write      0     0        0
es-master-03     write      0     0        0
xd-tsp-log-es-05 write      2     0        0
xd-tsp-log-es-03 write      0     0        0
xd-tsp-log-es-01 write      0     0        0
xd-tsp-log-es-02 write      0     0        0

active 表示活动的处理线程任务

queue 表示线程池中等待处理的任务

rejected 表示ES处理不过来 被拒绝的任务

迁移节点上所有分片

当集群有新节点加入并希望旧节点上的分片全部迁移到其他节点时 可执行如下命令

curl -H "Content-Type: application/json" -XPUT  '10.10.15.12:9200/_cluster/settings' -d  '{"transient" : {"cluster.routing.allocation.exclude._ip" : "10.10.12.10"}}'

PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "10.20.11.16"
  }
}

image.png

此命令会将IP为 10.20.11.16 节点上的分片全部迁移至其他节点，当集群 relocating shares 状态结束后即可将该节点下线。

如节点维护完成后再加入集群可使用一下命令：

PUT /_cluster/settings
{

"transient" : {

"cluster.routing.allocation.exclude._ip" : "null"

}

}

设置索引模板

PUT _template/bt-app-vehicle-api
{
  "order": 1,
  "index_patterns": [
    "bt-app-vehicle-api-prod*"
  ],
  "settings": {
    "index": {
      "number_of_shards": "3",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "_source": {},
    "_meta": {},
    "properties": {}
  }
}

orde 优先使用该项数值大的模板

索引模板修改字段类型

比如修改 args 字段 类型为 text

{"properties":{"args":{"type":"text"}}}

设置 refresh_interval

api

curl -XPUT -u elastic:123456 -H "Content-Type: application/json" 'http://10.10.4.24:9200/*test*'/_settings -d '{ "refresh_interval": "60s" }'

查看节点分片数

GET /_cat/allocation?v

修改字段类型和重建索引

查看mapping字段类型

GET /nginx-access-prod-2021.04.19/_mapping

比如修改 args 字段 类型为 text

新建索引并修改字段类型

PUT nginx-access-args-19/
{
    "mappings" : {
            "properties" : {
                "args" : { "type" : "text" }
            }

    }
}

重建索引。就是把 索引 nginx-access-prod-2021.04.19 内容 写入到 nginx-access-args-19 这个索引中

POST _reindex
{
  "source": {
    "index": "nginx-access-prod-2021.04.19"
  },
  "dest": {
    "index": "nginx-access-args-19"
  }
}

建立新的alias

需要在同时绑定建立的新索引以及解绑旧索引，语句如下：

POST _aliases
{
  "actions": [{"add": {
    "index": "teambition_20180328",
    "alias": "teambition_latest"
  }}, {"remove": {
    "index": "teambition",
    "alias": "teambition_latest"
  }}]
}

索引加别名

PUT /index_name/_alias/index_name_alias

损坏分片删除

当集群索引有损坏分片且没有副本时集群状态为red 此时需要删除损坏的分片，删除分片后会导致删除部分分片上的数据丢失。

POST /_cluster/reroute
{
  "commands": [
    {
      "allocate_empty_primary": {
        "index": "hufront-api-prod-2021.05.08",
        "shard": 3,
        "node": "1609838248000020032",
        "accept_data_loss": true
      }
    }
  ]
}

API 创建生命周期

PUT _ilm/policy/DELETE
{
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "14d",
          "actions" : { 
             "delete" : { }
          }
        }
      }
    }
}

API 批量生命周期调整

curl -XPUT "http://10.10.4.207:9200/app2021.06-*/_settings" -H 'Content-Type: application/json' -u elastic:123456 -d '{"index": {"lifecycle": {"name": "hot_warm_del_生命周期"}}}'

ES索引策略生效时间调整

ES索引策略生效时间默认为10min，可通过修改以下配置：

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "3s" 
  }
}

kibana查询提示分片失败处理

PUT /bt-order-ordermanage-service-prod-*/_settings
{"index" : {"highlight.max_analyzed_offset" : 100000000}}

分片平衡

允许控制多个并发分片重新平衡所允许的集群范围。默认为 2 。

curl -XPUT 127.0.0.1:9200/_cluster/settings -d '{
    "transient": {
        "cluster.routing.allocation.cluster_concurrent_rebalance": 10
    }
}'

允许控制多个并发分片重新平衡所允许的节点范围。默认为 2

cluster.routing.allocation.node_concurrent_recoveries

批量建索引模式、索引脚本

先导出服务名并去重

kubectl get deployment -n app-test | awk '{print $1}' > deployment_name

去重

sort -u deployment_name > deployment_name1

#!/bin/bash

date=`date -d "+1 days"  "+%Y.%m.%d"`

for deployment_name in `cat /root/deployment/deployment_name1`

do

curl -X PUT -u elastic:123456 "10.10.1.107:9200/$deployment_name-prod-$date?pretty"

done

查询过滤索引和去重

curl -XGET  -u elastic:123456 'http://10.19.5.17:9200/_cat/indices?v' | awk '{print $3}' | grep 2021.07.28 | sort

批量建模板

#!/bin/bash

for deployment_name in `cat /root/deployment_name1`

do

#建立模板
curl -XPUT --user elastic:123456 -H "Content-Type: application/json" 'http://10.19.5.3:9200/_template/'$deployment_name'-test' -d '{ "order" : 5,"index_patterns": ["'$deployment_name'-test*"],"settings": {"number_of_shards":1 ,"number_of_replicas": 1,"index.lifecycle.name": "test-sit-pre-perf-ilm","index.lifecycle.rollover_alias": "'$deployment_name'-test"}}'

#建立索引并设置别名
curl -XPUT --user elastic:123456 -H "Content-Type: application/json" 'http://10.19.5.3:9200/%3C'$deployment_name'-test-%7Bnow%2Fd%7D-000001%3E ' -d '{"aliases": {"'$deployment_name'-test":{"is_write_index": true}}}'


done

ALL & * 通配符删除索引

#开启批量删除
PUT /_cluster/settings
{"persistent" : {"action.destructive_requires_name" : "false"}}

API创建角色映射

7.5.1版本

PUT /_security/role_mapping/ad_users
{
    "enabled" : true,
    "roles" : [
      "index_read","reporting_user"
    ],
    "rules" : {
      "any" : [
        {
          "field" : {
            "realm.name" : "ad1"
          }
        }
      ]}
    }

以下

realm.name 对某个域进行角色映射；

user.name 对本地账户进行角色映射。

{
  "ad_users" : {
    "enabled" : true,
    "roles" : [
      "index_read",
      "reporting_user"
    ],
    "rules" : {
      "any" : [
        {
          "field" : {
            "realm.name" : "ad1"
          }
        }
      ]
    },
    "metadata" : { }
  },
  "om" : {
    "enabled" : true,
    "roles" : [
      "om",
      "watcher_admin",
      "monitoring_user",
      "reporting_user"
    ],
    "rules" : {
      "any" : [
        {
          "field" : {
            "username" : [
              "guohui",
              "wangjun",
              "weiyan",
              "zhangdy",
              "zhangxh",
              "huajh",
              "lvnan",
              "chenpf01",
              "xiets",
              "lvnan",
              "xiongzf",
              "sunyt"
            ]
          }
        }
      ]
    },
    "metadata" : { }
  }
}

7.10.0 版本

PUT /_security/role_mapping/ad_users_new
 {
    "enabled" : true,
    "roles" : [
      "reporting_user","index_only_read"
    ],
    "rules" : {
      "all" : [
        {
          "field" : {
            "realm.name" : "*"
          }
        }
      ]}
    }

创建用户

POST /_xpack/security/user/xiaomin?pretty
{

   "password" : "password",

   "full_name" : "xiaomin",

   "roles" : ["index_read"],

   "email" : "xiaomin@163.cn"

 }

自动创建索引模式脚本

对比ES中历史数据创建索引模式

#!/bin/bash

date=`date "+%Y.%m.%d"`

date_1=`date -d "-1 days" "+%Y.%m.%d"`

date_m=`date "+%Y.%m.%d %H:%M:%S"`

#date_2=`date -d "-2 days" "+%Y.%m.%d"`

cd  /root/index_pattern/

curl --connect-timeout 10 -XGET -u admin:1234 'http://10.18.0.105:9200/_cat/indices?v' | awk '{print $3}' | grep $date | grep gwm | sort |awk '{sub(/.{11}$/,"")}1' > /root/index_pattern/new_service_name

cat /root/index_pattern/new_service_name > /root/index_pattern/old_1_service_name

sleep 1

curl --connect-timeout 10 -XGET -u admin:1234 'http://10.18.0.105:9200/_cat/indices?v' | awk '{print $3}' | grep $date_1 | grep gwm | sort |awk '{sub(/.{11}$/,"")}1' >> /root/index_pattern/old_service_name #获取前一天的数据。以防止历史数据被重复创建。

if [[ ! -s /root/index_pattern/new_service_name ]]

then

    echo "=== new_service_name为空，退出脚本 $date_m ===" >> log_auto_create_index-pattern.log
    exit

else

    cat /root/index_pattern/old_service_name >> /root/index_pattern/new_service_name

    cat /root/index_pattern/old_service_name >> /root/index_pattern/new_service_name #第二次执行防止0点new_service_name只有几个索引创建的情况，因为很多服务第一时间不会创建索引。

    sort /root/index_pattern/new_service_name | uniq -u > /root/index_pattern/create_service_name

    cat /root/index_pattern/old_1_service_name > /root/index_pattern/old_service_name 

fi

if [[ ! -s /root/index_pattern/create_service_name ]]

then

   echo "=== create_service_name为空，退出脚本 $date_m ===" >> log_auto_create_index-pattern.log
   exit

fi


for deployment_name in `cat /root/index_pattern/create_service_name`

do

    curl --fail --user admin:1234 --request POST --header 'Content-Type: application/json' --header 'kbn-xsrf: this_is_required_header' 'http://10.18.0.109:5601/api/saved_objects/index-pattern/'$deployment_name'-*?overwrite=true' --data '{"attributes":{"title":"'$deployment_name'-*","timeFieldName":"@timestamp"}}' 'http://10.187.0.109:5601/api/saved_objects/index-pattern/'$deployment_name'-*?overwrite=true' >> log_auto_create_index-pattern.log

    echo $date_m >> log_auto_create_index-pattern.log

done

对比kibana索引模式名字和k8s deployment名字创建索引模式

#!/bin/bash

date_m=`date "+%Y.%m.%d %H:%M:%S"`

#获取kibana上测试环境已创建索引模的式服务
curl --fail --user admin:123456 --header 'Content-Type: application/json' --header 'kbn-xsrf: this_is_required_header' 'http://10.17.0.109:5601/api/saved_objects/_find?type=index-pattern&fields=title&fields=type&per_page=1000&search_fields=title&search='^test'-\*' |python -m json.tool | grep title| awk '{print $2}' |awk '{sub(/.{3}$/,"")}1'|grep '^"test-gwm'| sed -r 's/^.{6}//' > /root/index_pattern/kibana_k8s_service_name

# python -m json.tool      以json方式输出结果
# awk '{sub(/.{3}$/,"")}1'  去掉每行最后三个字符
# sed -r 's/^.{6}//'        去掉每行开头的六个字符

#获取k8s中gwm开头服务名
kubectl get deploy -A | awk '{print $2}' | grep '^gwm-' > /root/index_pattern/k8s_service_name

cat /root/index_pattern/kibana_k8s_service_name > /root/index_pattern/deduplication_name
cat /root/index_pattern/k8s_service_name >> /root/index_pattern/deduplication_name

#筛选出多余的名字
sort /root/index_pattern/deduplication_name | uniq -u > /root/index_pattern/superfluous_name

#去除不需要处理的服务名
sed -i 's/gwm-k8s-event-log//g' /root/index_pattern/superfluous_name

#去掉空白行
sed -i '/^\s*$/d' /root/index_pattern/superfluous_name


if [[ ! -s /root/index_pattern/superfluous_name ]]

then

   echo "=== superfluous_name 为空，退出脚本 $date_m ===" >> /root/index_pattern/log_auto_create_index-pattern.log
   exit

fi

for service_name in `cat /root/index_pattern/superfluous_name`
do

        curl --fail --user admin:123456 --request POST --header 'Content-Type: application/json' --header 'kbn-xsrf: this_is_required_header' 'http://10.17.0.109:5601/api/saved_objects/index-pattern/test-'$service_name'-*?overwrite=true' --data '{"attributes":{"title":"test-'$service_name'-*","timeFieldName":"@timestamp"}}' 'http://10.187.0.109:5601/api/saved_objects/index-pattern/test-'$service_name'-*?overwrite=true' >> //root/index_pattern/log_auto_create_index-pattern.log
        echo `date "+%Y.%m.%d %H:%M:%S"`  >> /root/index_pattern/log_auto_create_index-pattern.log
done

mapping定义不可索引字段

PUT /_template/prod
{
  "order": 3,
  "index_patterns": [
    "*-prod-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "warm_del_生命周期"
      },
      "routing": {
        "allocation": {
          "require": {
            "temperature": "warm"
          }
        }
      },
      "refresh_interval": "30s",
      "number_of_shards": "1",
      "translog": {
        "sync_interval": "30s",
        "durability": "async"
      },
      "max_result_window": "10000",
      "unassigned": {
        "node_left": {
          "delayed_timeout": "5m"
        }
      },
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "properties": {
      "kubernetes.container.name": {
        "eager_global_ordinals": false,
        "index_phrases": false,
        "fielddata": false,
        "norms": true,
        "index": false,
        "store": false,
        "type": "text"
      },
      "index_date": {
        "eager_global_ordinals": false,
        "index_phrases": false,
        "fielddata": false,
        "norms": true,
        "index": false,
        "store": false,
        "type": "text"
      }
    }
  },
  "aliases": {}
}

或者kibana 模板编辑界面映射模块

{
  "properties": {
    "kubernetes.container.name": {
      "eager_global_ordinals": false,
      "index_phrases": false,
      "fielddata": false,
      "norms": true,
      "index": false,
      "store": false,
      "type": "text"
    },
    "index_date": {
      "eager_global_ordinals": false,
      "index_phrases": false,
      "fielddata": false,
      "norms": true,
      "index": false,
      "store": false,
      "type": "text"
    }
  }
}
标题：Elasticsearch & API
作者：zifuy
地址：https://www.zifuy.cn/articles/2020/03/23/1584941268117.html