节点标签属性

在elasticsearch.yml里面，格式：

node.attr.{attribute}: {value}

例如：增加一个node_type属性

node.attr.node_type: hot

索引分配过滤器设置

index.routing.allocation.include.{attribute}
index.routing.allocation.exclude.{attribute}
index.routing.allocation.require.{attribute}

节点角色

7.10版本之后的集群不再通过node.master，node.data来区分是何种角色的节点，而是通过node.roles数组来定义一个节点的角色，并且加入了冷热节点的角色属性

例如：

node.roles: ["data_hot", "data_content", "ingest", "ml", "remote_cluster_client", "transform"]

除了对节点角色的优化，还对索引的allocation做了优化，原先我们是通过include.{attribute}、require.{attribute}、exclude.{attribute}来设置索引的allocation settings；而升级到DataTier模式后，则是通过index.routing.allocation.include._tier_preference 来定义.

如果配置多个角色，当集群中有data_hot节点时，则直接将分片分配在data_hot节点上，当没有data_hot节点时，则检查是否有data_warm节点，如果有，则在data_warm节点上分配，没有则在data_cold节点上分配

7.10及以后版本的生命周期和索引模板

生命周期

无冷热节点数据的迁移的生命周期策略

索引创建3天后删除

PUT _ilm/policy/delete
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

有冷热节点数据的迁移的生命周期策略

策略：

数据创建在热节点；

索引创建1分钟后迁移到温节点；

索引创建2分钟后迁移到冷节点；

索引创建3分钟后删除

PUT _ilm/policy/nignx-test
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {}
      },
      "warm": {
        "min_age": "1m",
        "actions": {
          "allocate": {
            "include": {
              "_tier_preference": "data_warm"
            },
            "exclude": {},
            "require": {}
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "2m",
        "actions": {
          "allocate": {
            "include": {
              "_tier_preference": "data_cold"
            },
            "exclude": {},
            "require": {}
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "3m",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

注意：

使用以上生命周期时，ES节点角色需要设置为 data_hot 或 data_warm 或 data_cold；且不能同时使用 data 这个角色，否则生命周期冷热属性不生效。

节点角色没有 data 属性时,通过 cerebro 0.9.4以前的版本无法看到索引所在的节点。更新cerebor到0.9.4在 overvier界面可以展示索引界面

cerebor 0.9.4

https://github.com/lmenezes/cerebro/releases/tag/v0.9.4

有滚动更新的生命周期策略

策略：

索引达到10000个文档

或

索引数据大小超过30gb

或

索引创建后时长达到3天

只要满足上面任意一个。即触发索引的滚动更新策略。

第4天迁移到温节点、第5天迁移到冷节点、第6天删除索引

PUT _ilm/policy/nginx-test
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_age": "3d",
            "max_primary_shard_size": "30gb",
            "max_docs": 10000
          }
        }
      },
      "warm": {
        "min_age": "4d",
        "actions": {
          "set_priority": {
            "priority": 99
          },
          "allocate": {
            "include": {
              "_tier_preference": "data_warm"
            }
          }
        }
      },
      "cold": {
        "min_age": "5d",
        "actions": {
          "set_priority": {
            "priority": 98
          },
          "allocate": {
            "include": {
              "_tier_preference": "data_cold"
            }
          }
        }
      },
      "delete": {
        "min_age": "6d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

索引模板

无冷热节点标签属性的索引模版

PUT _index_template/template_1
{
  "index_patterns": [
    "*-perf-*"
  ],
  "template" : {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "delete"
        },
        "number_of_shards": 1,
        "number_of_replicas": "1",
        "refresh_interval": "30s",
        "translog": {
          "sync_interval": "30s",
          "durability": "async"
        },
        "query": {
          "default_field": [
            "message"
          ]
        },
        "max_result_window": "10000"
      }
    },
  "mappings": {
    "properties": {}
  }
  },
  "priority": 500
}

上面 index.query.default_field 参数指定默认查询字段为 message 字段。ELK场景下kibana默认搜索所有字段。设置默认字段可以提高查询效率,较少资源消耗。

有冷热节点标签属性的索引模版

PUT _index_template/template_1
{
  "index_patterns": [
    "*-perf-*"
  ],
  "template" : {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "delete"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "number_of_shards": 1,
        "number_of_replicas": "1",
        "refresh_interval": "30s",
        "translog": {
           "sync_interval": "30s",
           "durability": "async"
        },
        "query": {
          "default_field": [
            "message"
          ]
        },
        "max_result_window": "10000"
      }
    },
  "mappings": {
    "properties": {}
  }
  },
  "priority": 500
}

有热滚动更新属性的索引模版

PUT _index_template/nginx-test
{
  "index_patterns": [
    "nginx-*"
  ],
  "template" : {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "nginx-test",
          "rollover_alias": "nginx-test"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "number_of_shards": 1,
        "number_of_replicas": "0",
        "refresh_interval": "30s",
        "translog": {
          "sync_interval": "30s",
          "durability": "async"
        },
        "query": {
          "default_field": [
            "message"
          ]
        },
        "max_result_window": "10000"
      }
    },
  "mappings": {
    "properties": {}
  }
  },
  "priority": 600
}

注意

7.10及以后版本索引或模板在使用 index.routing.allocation.include._tier_preference 参数时 不需要 "index.lifecycle.parse_origination_date": "true" 否则生命周期冷热节点的调度会失效。

创建具有滚动更新属性的初始索引

PUT %3Cnginx-test-%7Bnow%2Fd%7D-000001%3E
{
  "aliases": {
    "nginx_-test": {
      "is_write_index":true
    }
  }
}

7.10以前版本的生命周期和索引模板

生命周期

无冷热节点数据的迁移的生命周期策略

索引创建3天后删除

PUT _ilm/policy/delete
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

有冷热节点数据的迁移的生命周期策略

策略：

数据创建在热节点；

索引创建1分钟后迁移到温节点；

索引创建2分钟后迁移到冷节点；

索引创建3分钟后删除

PUT _ilm/policy/bt-user-task-service
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "1m",
        "actions": {
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "require": {
              "temperature": "warm"
            }
          }
        }
      },
      "cold": {
        "min_age": "2m",
        "actions": {
          "set_priority": {
            "priority": 10
          },
          "allocate": {
            "require": {
              "temperature": "cold"
            }
          }
        }
      },
      "delete": {
        "min_age": "3m",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

以上生命周期定义了 index.routing.allocation.require.temperature 配置项，如果索引要成功使用该生命周期，需满足：

1、ES的node节点 elasticsearch.yml 需先配置有和上面周期匹配的 index.routing.allocation 节点属性。

例如：

node.attr.temperature: hot  热节点使用该配置
node.attr.temperature: warm 温节点使用该配置
node.attr.temperature: cold 冷节点使用该配置

2、索引创建时需要配置索引的 index.routing.allocation ，这个配置在 【索引模板】 【有冷热节点标签属性的索引模板】中有给出

有热滚动更新属性的生命周期策略

策略：

索引达到10000个文档

或

索引数据大小超过30gb

或

索引创建后时长达到3天

只要满足上面任意一个。即触发索引的滚动更新策略。

PUT _ilm/policy/nginx-test
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "3d",
            "max_size": "30gb",
            "max_docs": 10000
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

说明： 热滚动更新和所属节点的冷热属性无关、即使索引模板使索引一开始分配到冷节点，但只要关联的生命周期配置了滚动更新，索引在达到触发条件时就会滚动更新。

索引模板

7.10以前的模板API也可以用到7.10及以后版本。但有少许区别：

1、PUT _template/template_1 需使用新的创建方式 PUT _index_template/template_1

2、优先级字段 order 需要改成 priority，且不能有重复的 priority

无冷热节点标签属性的索引模版

PUT _template/template_1
{
  "index_patterns": [
    "test"
  ],
  "settings": {
      "index": {
         "lifecycle": {
             "name": "delete"
         },
      "number_of_shards": 3,
      "number_of_replicas": "1",
      "refresh_interval": "10s",
      "translog": {
          "sync_interval": "15s",
          "durability": "async"
      },
      "query": {
          "default_field": [
          "message"
        ]
      },
    }
  },
  "mappings": {
    "properties": {}
  },
  "order": 10
}

有冷热节点标签属性的索引模版

PUT _template/template_1
{
  "index_patterns": [
    "*-perf-*"
  ],
    "settings": {
      "index": {
        "lifecycle": {
          "name": "delete"
        },
        "routing": {
          "allocation": {
            "require": {
              "temperature": "hot"
            }
          }
        },
        "number_of_shards": 1,
        "number_of_replicas": "1",
        "refresh_interval": "60s",
        "translog": {
           "sync_interval": "30s",
           "durability": "async"
        },
        "query": {
          "default_field": [
            "message"
          ]
        },
        "max_result_window": "10000"
      }
    },
  "mappings": {
    "properties": {}
  },
  "order": 10
}

有热滚动更新属性的索引模版

PUT _template/nginx-test
{
  "index_patterns": [
    "nginx-*"
  ],
    "settings": {
      "index": {
        "lifecycle": {
          "name": "delete",
          "parse_origination_date": "true",
          "rollover_alias": "nginx-test"
        },
        "number_of_shards": 1,
        "number_of_replicas": "1",
        "refresh_interval": "60s",
        "translog": {
           "sync_interval": "30s",
           "durability": "async"
        },
        "query": {
          "default_field": [
            "message"
          ]
        },
        "max_result_window": "10000"
      }
    },
  "mappings": {
    "properties": {}
  },
  "order": 10
}

热滚动更新初始索引创建

PUT %3Cnginx-log-%7Bnow%2Fd%7D-000001%3E   //基于日期的date Math 格式转换
{
  "aliases": {
    "nginx_weblogs": {        //别名
      "is_write_index":true   //允许索引写
    }
  }
}

curl -XPUT --user elastic:123456 -H "Content-Type: application/json" 'http://10.19.5.3:9200/%3Cnginx-ingress-controller-test-%7Bnow%2Fd%7D-000001%3E ' -d '{"aliases": {"nginx-ingress-controller-test":{"is_write_index": true}}}'

生命周期策略检测时间

索引生命周期管理检查符合策略标准的索引的频率。默认为 10m ，调整为3s。

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "3s" 
  }
}

logstash支持生命周期的配置

elasticsearch {

#index => "%{[kubernetes][container][name]}-3-prod" #支持这种配置
#ilm_rollover_alias => "%{[kubernetes][container][name]}-3-prod" #不支持这种配置
#ilm_policy => "%{[kubernetes][container][name]}-3-prod"  #不支持这种配置

ilm_pattern => "{now/d}-000001"
ilm_enabled => true
ilm_rollover_alias => "nginx-ingress-controller-3-prod"
ilm_policy => "nginx-ingress-controller-3-prod"

hosts => ["10.10.4.207"]
user => "elastic"
password => "123456"
}

说明：ilm_rollover_alias、ilm_policy 不支持变量方式配置，改参数值必须是固定字符串值。

配置完成后索引生成及滚动结果如下图

命令参考：

不使用kibana创建模板

curl -XPUT --user elastic:123456 -H "Content-Type: application/json" 'http://10.19.5.3:9200/_template/nginx-ingress-controller-test' -d '{ "order" : 5,"index_patterns": ["nginx-ingress-controller-test*"],"settings": {"number_of_shards":1 ,"number_of_replicas": 1,"index.lifecycle.name": "test-sit-pre-perf-ilm","index.lifecycle.rollover_alias": "nginx-ingress-controller-test"}}'

OSS版ES生命周期

模板

PUT _template/vehicle_original_message
{
  "order": 0,
  "index_patterns": [
    "vehicle_original_message*"
  ],
  "settings": {
    "index": {
      "codec": "best_compression",
      "opendistro": {
        "index_state_management": {
          "policy_id": "vehicle_original_message_policy",
          "rollover_alias": "vehicle_original_message"
        }
      },
      "refresh_interval": "30s",
      "sort": {
        "field": "gatewayReceiveTime",
        "order": "desc"
      },
      "number_of_shards": "6",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "properties": {
      "deviceType": {
        "type": "long"
      },
      "hex": {
        "index": false,
        "type": "text"
      },
      "vin": {
        "type": "text",
        "fields": {
          "keyword": {
            "ignore_above": 256,
            "type": "keyword"
          }
        }
      },
      "acquisitionTime": {
        "index": false,
        "type": "long"
      },
      "gatewayReceiveTime": {
        "type": "long"
      },
      "imsi": {
        "index": false,
        "type": "text"
      },
      "command": {
        "type": "text",
        "fields": {
          "keyword": {
            "ignore_above": 256,
            "type": "keyword"
          }
        }
      }
    }
  },
  "aliases": {}
}

生命周期

PUT _opendistro/_ism/policies/vehicle_original_message_policy
{
  "policy": {
    "policy_id": "vehicle_original_message_policy",
    "description": "vehicle_original_message",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_index_age": "1d",
              "min_doc_count": 2
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "90d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": [
        "vehicle_original_message*"
      ],
      "priority": 100
    }
  }
}

写索引替换

POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "vehicle_original_message-2023.04.17-000045",
        "alias": "vehicle_original_message",
        "is_write_index": false
      }
    },
    {
      "add": {
        "index": "vehicle_original_message-2023.04.18-0000046",
        "alias": "vehicle_original_message",
        "is_write_index": true
      }
    }
  ]
}